2022-07-20T02:13:49.911 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-20T02:13:49.957 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} --verbose --archive /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288 --name yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/orig.config.yaml 2022-07-20T02:13:50.519 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-20T02:13:50.981 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288 branch: octopus-release description: fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} email: ceph-qa@ceph.io first_in_suite: false job_id: '6939288' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: octopus-release ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over - MDS_CLIENT_LATE_RELEASE - responding to mclientcaps sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: octopus-release sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 sleep_before_teardown: 0 suite: fs suite_branch: octopus-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 targets: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDftE6x4eFChcx3puqfvcQI+jLl8ROUdei5ughSJP9Ajd6ImomCxHmgMwAvEfZUMgJvisVUNyHdf4NNwKFJ8aQCYq8qi+EvuD4doyghkTg0x3S116v9UtB9vv7fNKxB12pe5Uq3GUaiXx/kib2npnL8/qYuwT31oh9fTPXKjVZxx0Pmf2H/9y5RbZlkmEMeTVzr8Lyg2jHpTaugHHzkwVjtYAdbY4RJzF+gZ52LLX2SVnuVsScKK6gUGEyrLbrtfPyrQwfi76X2AsvpjFvILEqBIu0+PBeYp5rpwXQLg3FBUEwWfRVABM8yCOezU9vIP5ROgGXSS6O0H/fcqsYkTD3HYL0IckEQTq1VbB5OA9yfM5gpcKVL8y4XjxMvkMpYHI4COO4SLEgNERWGa5p+kE21mokDR0E8zpdjaFG23spgFw4uZ+g4kq1LL07jBCpCkZWEY2XdpHv5o/gLW6ppqnCH7NyG/bt3d3I74XCXZ4FTtk2nG/QBiFuiiuFI4F7XY10= smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNinRuNL786cCN5o7rl2CdnbHLqNOlyxX4GgQ5s/xhMCBIiu3wr6m9XqkNDoOWXErxcs4CN3S4/QEemNwgaZTRSPPi0HhSQQDCbmoSgdTxEnE6DHbjks9NcI/uB58u9Pt487cLNATS1xotJUptd/9fZtc1wZiTTZwIEp0YXtq6lp11vNm9urGv1V/8EPwPBd3wv4sL1l62OgZwXFbXD8YZ17aqkFm4dzESvk4HMhfIiqUQJ3bn+Nd/WkPMnwwUUYjP05kHdINJ1NAIyGtCa1Bn4mEd01rM4PH1pYyeNQLg3o6hSuj7CKzm/YcbM510V9/e+DJ6i1MUtG6R8pHi6KPIRmGueaO15sIsgyuutpyu5oqvS83a4cCs2X6CZarSOujCn2WucGNHGNbWTjl1Cxb6Zq0PxSNZJLHjwCcbxubmhcfS0HOjDG2mrNCPh8HLy/PCmBfhZNyKxiohFZ2knQgTZAywEVECYWh4T1XrAX6/PVxMEFiUXPMJV64KRV+tP30= smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjZ1hzUsMAkKgInBZwQ4m0nyjKcvPVfXzgGvV63vSDF9oLXPe6g9janjl4noX6heXyWHEQ7VIVIMG6jXJUaI7aXqlcu6WGhQAGwkt6HLzXpOAU90bMnlA8be2v+F2pWys/Yddr+RdeUFUAcZxAtH5ZDn9XrVLygy5deztBHZ+Ky0tgy/3AjHYZOQK9VIQMayS4v8UkpzZ5hZYO+PXRlCekpDGfAv5+jQn0GZbrLHpJ+RNuUdVSykuFyFEvw3jRi1qBPxeRYI4wpb9Mggr9nBsOSVf0J6XDRXZ3/AgNHiN43gGS3Yz0e+/QdlQbZvvXW9oFxk17WsUwrwbY1SrJ9hWhFbMQZr0HiFIij3xPD5VtRN88rToqsSPEQNR5uBHddO9/omvc60YocZbCmJpAu5MXdiAmzQVrztHytKMZN3y/pQF3LtVgPG3613RKt3KN3m6HmksXmV8eEARfgglsGsASclIsPZzfln3Q4CuJcWSalNzxb1AYjrwYKAVykHTU058= smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f9jQtixYdFaa5M+hOPFWolp3Aur96UU75qBTq5FyusQFo5Y6U7inZ09QRFmApcUw+9d7V02teTesIoqEB8QP5ffm6ylE9eTEPDyuH+CqnO/8NhZOl+Wezl//zKW8EXbKN1ogN3GtXEj+twBZG1PV3mzMsY2FmxJ4jIubqaXJRkDcx7sVIeDGJ8ry6e30mZtNnT1An6+ubz7Hx32ZZ1YX9edblbcPMrupracArUUFKbuA84QMCNC1oR3iAO2sskjoukErIqT9xoLkqzvAi0yfopCrAdCup3iz+ETnKiJEACDYdUCzZDPeR9DdeyvrJzmBJJS989PV3nsyxGrhg/qEELoVReDt8od+82X2SEe0oVkiZ89wisLf+iin01uM8doeO+8cbZkdzu3zfJLsMuAgIYylsqrP8cCpNf8ullscgVCAm1Lm85Zecr/Vi4/FQ5z+QQMNNq1aB9ws39eHoDGRGAHtjW7qe0BL8plYb9pbrUBFSv5qp63HDVyokxF7aQk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: main teuthology_sha1: 89d874d555f56b1c1f383df72487aecec0ee6cf9 timestamp: 2022-07-19_23:28:53 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-20T02:13:51.032 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa; will attempt to use it 2022-07-20T02:13:51.033 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks 2022-07-20T02:13:51.079 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-20T02:13:51.105 INFO:teuthology.task.internal:Checking packages... 2022-07-20T02:13:51.432 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '57544da114dbbe1c44f27b2ea18e3233e4e917e0' 2022-07-20T02:13:51.433 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-20T02:13:51.433 INFO:teuthology.packaging:ref: None 2022-07-20T02:13:51.463 INFO:teuthology.packaging:tag: None 2022-07-20T02:13:51.464 INFO:teuthology.packaging:branch: octopus-release 2022-07-20T02:13:51.464 INFO:teuthology.packaging:sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:13:51.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=octopus-release 2022-07-20T02:13:51.738 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-336-g57544da1-1focal 2022-07-20T02:13:51.739 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-20T02:13:51.751 INFO:teuthology.task.internal:no buildpackages task found 2022-07-20T02:13:51.752 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-20T02:13:51.772 INFO:teuthology.task.internal:Saving configuration 2022-07-20T02:13:52.023 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-20T02:13:52.036 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-20T02:13:52.097 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:07:53.980059', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZlFO7JzgLpO4KL9wWUHWm8H8H9c8mi7g9dPn8Hfe3G'} 2022-07-20T02:13:52.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:07:53.978510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHmIJC1N4tgWAPDccm3LnISBC4aI0q9/5Mqh0+WYxLZ7'} 2022-07-20T02:13:52.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:07:53.981224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIARhfSggcq+W6c3mU2twbaYj0pLkfZqJOiQJhFuY5YLb'} 2022-07-20T02:13:52.303 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:07:53.975786', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDS7eKeEK9bCU4mGoVFIxJTGL/c52vsjOnz8pnuv8S6c'} 2022-07-20T02:13:52.304 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-20T02:13:52.319 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-07-20T02:13:52.320 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-07-20T02:13:52.320 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['client.0'] 2022-07-20T02:13:52.321 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['client.1'] 2022-07-20T02:13:52.322 INFO:teuthology.run_tasks:Running task console_log... 2022-07-20T02:13:53.096 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3e336b8d90>, signals=[15]) 2022-07-20T02:13:53.100 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-20T02:13:53.157 INFO:teuthology.task.internal:Opening connections... 2022-07-20T02:13:53.158 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:13:53.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:13:53.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:13:53.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:13:54.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2022-07-20T02:13:54.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:13:55.177 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2022-07-20T02:13:55.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:13:55.689 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-20T02:13:55.765 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2022-07-20T02:13:55.779 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2022-07-20T02:13:55.786 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2022-07-20T02:13:55.921 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Ubuntu" 2022-07-20T02:13:55.922 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:13:55.922 INFO:teuthology.orchestra.run.smithi026.stdout:ID=ubuntu 2022-07-20T02:13:55.922 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE=debian 2022-07-20T02:13:55.922 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:13:55.923 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="20.04" 2022-07-20T02:13:55.923 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:13:55.923 INFO:teuthology.orchestra.run.smithi026.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:13:55.923 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:13:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:13:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_CODENAME=focal 2022-07-20T02:13:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:13:55.971 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2022-07-20T02:13:56.028 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2022-07-20T02:13:56.057 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2022-07-20T02:13:56.091 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2022-07-20T02:13:56.119 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Ubuntu" 2022-07-20T02:13:56.120 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:13:56.120 INFO:teuthology.orchestra.run.smithi120.stdout:ID=ubuntu 2022-07-20T02:13:56.121 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE=debian 2022-07-20T02:13:56.121 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:13:56.121 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="20.04" 2022-07-20T02:13:56.122 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:13:56.122 INFO:teuthology.orchestra.run.smithi120.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:13:56.123 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:13:56.123 INFO:teuthology.orchestra.run.smithi120.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:13:56.124 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_CODENAME=focal 2022-07-20T02:13:56.124 INFO:teuthology.orchestra.run.smithi120.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:13:56.125 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2022-07-20T02:13:56.181 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2022-07-20T02:13:56.240 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2022-07-20T02:13:56.241 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2022-07-20T02:13:56.318 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Ubuntu" 2022-07-20T02:13:56.319 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:13:56.319 INFO:teuthology.orchestra.run.smithi159.stdout:ID=ubuntu 2022-07-20T02:13:56.319 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE=debian 2022-07-20T02:13:56.320 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:13:56.320 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="20.04" 2022-07-20T02:13:56.320 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:13:56.320 INFO:teuthology.orchestra.run.smithi159.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:13:56.321 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:13:56.321 INFO:teuthology.orchestra.run.smithi159.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:13:56.321 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_CODENAME=focal 2022-07-20T02:13:56.321 INFO:teuthology.orchestra.run.smithi159.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:13:56.438 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2022-07-20T02:13:56.527 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2022-07-20T02:13:56.667 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2022-07-20T02:13:56.668 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2022-07-20T02:13:56.716 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2022-07-20T02:13:56.717 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:13:56.717 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2022-07-20T02:13:56.717 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2022-07-20T02:13:56.717 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:13:56.718 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="20.04" 2022-07-20T02:13:56.718 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:13:56.718 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:13:56.718 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:13:56.719 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:13:56.719 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=focal 2022-07-20T02:13:56.719 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:13:56.719 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2022-07-20T02:13:56.753 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-20T02:13:56.781 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-20T02:13:56.874 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-20T02:13:56.875 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:13:57.132 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:13:57.272 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:13:57.327 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:13:57.336 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-20T02:13:57.381 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-20T02:13:57.389 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:13:57.444 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:13:57.523 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:13:57.599 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:13:57.633 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-20T02:13:57.895 INFO:teuthology.run_tasks:Running task kernel... 2022-07-20T02:13:57.925 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-20T02:13:57.955 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-20T02:13:57.961 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:13:57.971 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:13:57.981 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:13:57.982 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:13:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:13:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:13:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:13:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:13:58.105 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2022-07-20T02:13:58.194 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2022-07-20T02:13:58.196 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2022-07-20T02:13:58.197 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2022-07-20T02:13:58.239 INFO:teuthology.orchestra.run.smithi176.stdout:5.4.0-122-generic 2022-07-20T02:13:58.240 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.4.0-122-generic 2022-07-20T02:13:58.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2022-07-20T02:13:58.265 INFO:teuthology.orchestra.run.smithi026.stdout:5.4.0-122-generic 2022-07-20T02:13:58.294 INFO:teuthology.orchestra.run.smithi120.stdout:5.4.0-122-generic 2022-07-20T02:13:58.295 INFO:teuthology.orchestra.run.smithi159.stdout:5.4.0-122-generic 2022-07-20T02:13:58.296 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.4.0-122-generic 2022-07-20T02:13:58.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get clean 2022-07-20T02:13:58.298 INFO:teuthology.task.kernel:Running kernel on smithi120: 5.4.0-122-generic 2022-07-20T02:13:58.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get clean 2022-07-20T02:13:58.301 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.4.0-122-generic 2022-07-20T02:13:58.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo apt-get clean 2022-07-20T02:13:58.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get update 2022-07-20T02:13:58.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2022-07-20T02:13:58.481 DEBUG:teuthology.orchestra.run.smithi159:> sudo apt-get update 2022-07-20T02:13:58.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2022-07-20T02:13:58.735 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:13:59.063 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:13:59.201 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:13:59.235 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:13:59.278 INFO:teuthology.orchestra.run.smithi159.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:13:59.323 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:13:59.348 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:13:59.441 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:13:59.493 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:13:59.559 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:13:59.606 INFO:teuthology.orchestra.run.smithi159.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:13:59.621 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:13:59.679 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:13:59.719 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:13:59.738 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:13:59.755 INFO:teuthology.orchestra.run.smithi159.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:13:59.876 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:13:59.877 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:13:59.877 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:13:59.894 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:13:59.936 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:13:59.946 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:14:00.031 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:14:00.137 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:14:00.140 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:14:00.161 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:14:00.166 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:14:00.208 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:14:00.279 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:14:00.284 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:14:00.290 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:14:00.292 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:14:00.343 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:14:00.372 INFO:teuthology.orchestra.run.smithi120.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:14:00.374 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:14:00.381 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:14:00.425 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:14:00.432 INFO:teuthology.orchestra.run.smithi120.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:14:00.447 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:14:00.539 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:14:00.631 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:14:00.722 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:14:00.723 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:14:00.723 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:14:00.790 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:14:00.971 INFO:teuthology.orchestra.run.smithi159.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:14:00.972 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:14:00.990 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:14:01.046 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:14:01.100 INFO:teuthology.orchestra.run.smithi159.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:14:01.136 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:14:01.375 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:14:01.412 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:14:01.536 INFO:teuthology.orchestra.run.smithi120.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:14:01.568 INFO:teuthology.orchestra.run.smithi159.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:14:01.614 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:14:01.765 INFO:teuthology.orchestra.run.smithi159.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:14:01.788 INFO:teuthology.orchestra.run.smithi159.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:14:02.026 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:14:02.338 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:14:02.895 INFO:teuthology.orchestra.run.smithi120.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:14:03.663 INFO:teuthology.orchestra.run.smithi120.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:14:03.758 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 10.3 MB in 3s (3,767 kB/s) 2022-07-20T02:14:04.356 INFO:teuthology.orchestra.run.smithi120.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:14:04.394 INFO:teuthology.orchestra.run.smithi120.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:14:04.629 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 10.3 MB in 4s (2,702 kB/s) 2022-07-20T02:14:04.956 INFO:teuthology.orchestra.run.smithi120.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:14:05.114 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2022-07-20T02:14:05.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:14:05.282 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2022-07-20T02:14:05.466 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2022-07-20T02:14:05.467 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2022-07-20T02:14:05.699 INFO:teuthology.orchestra.run.smithi026.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:14:05.699 INFO:teuthology.orchestra.run.smithi026.stdout:linux-image-generic set to manually installed. 2022-07-20T02:14:05.700 INFO:teuthology.orchestra.run.smithi026.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:14:05.727 DEBUG:teuthology.orchestra.run.smithi026:> dpkg -s linux-image-generic 2022-07-20T02:14:05.768 INFO:teuthology.orchestra.run.smithi026.stdout:Package: linux-image-generic 2022-07-20T02:14:05.768 INFO:teuthology.orchestra.run.smithi026.stdout:Status: install ok installed 2022-07-20T02:14:05.769 INFO:teuthology.orchestra.run.smithi026.stdout:Priority: optional 2022-07-20T02:14:05.769 INFO:teuthology.orchestra.run.smithi026.stdout:Section: kernel 2022-07-20T02:14:05.769 INFO:teuthology.orchestra.run.smithi026.stdout:Installed-Size: 19 2022-07-20T02:14:05.769 INFO:teuthology.orchestra.run.smithi026.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:14:05.772 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: amd64 2022-07-20T02:14:05.772 INFO:teuthology.orchestra.run.smithi026.stdout:Source: linux-meta 2022-07-20T02:14:05.773 INFO:teuthology.orchestra.run.smithi026.stdout:Version: 5.4.0.122.123 2022-07-20T02:14:05.773 INFO:teuthology.orchestra.run.smithi026.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:14:05.773 INFO:teuthology.orchestra.run.smithi026.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:14:05.773 INFO:teuthology.orchestra.run.smithi026.stdout:Recommends: thermald 2022-07-20T02:14:05.774 INFO:teuthology.orchestra.run.smithi026.stdout:Description: Generic Linux kernel image 2022-07-20T02:14:05.774 INFO:teuthology.orchestra.run.smithi026.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:14:05.774 INFO:teuthology.orchestra.run.smithi026.stdout: available. 2022-07-20T02:14:05.774 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:14:05.775 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-07-20T02:14:05.775 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:14:05.838 INFO:teuthology.orchestra.run.smithi120.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:14:05.900 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2022-07-20T02:14:05.901 DEBUG:teuthology.parallel:result is None 2022-07-20T02:14:05.919 INFO:teuthology.orchestra.run.smithi120.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:14:05.950 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-07-20T02:14:05.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:14:06.022 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-07-20T02:14:06.187 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-07-20T02:14:06.187 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-07-20T02:14:06.377 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:14:06.378 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic set to manually installed. 2022-07-20T02:14:06.414 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:14:06.415 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2022-07-20T02:14:06.473 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2022-07-20T02:14:06.497 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2022-07-20T02:14:06.499 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2022-07-20T02:14:06.499 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2022-07-20T02:14:06.500 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 19 2022-07-20T02:14:06.500 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:14:06.500 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2022-07-20T02:14:06.500 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2022-07-20T02:14:06.501 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.4.0.122.123 2022-07-20T02:14:06.501 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:14:06.501 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:14:06.501 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2022-07-20T02:14:06.501 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2022-07-20T02:14:06.502 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:14:06.515 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2022-07-20T02:14:06.516 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:14:06.518 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2022-07-20T02:14:06.518 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:14:06.565 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2022-07-20T02:14:06.583 DEBUG:teuthology.parallel:result is None 2022-07-20T02:14:08.226 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 10.3 MB in 7s (1,401 kB/s) 2022-07-20T02:14:09.550 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-07-20T02:14:09.572 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:14:09.663 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-07-20T02:14:09.843 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-07-20T02:14:09.844 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-07-20T02:14:10.040 INFO:teuthology.orchestra.run.smithi120.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:14:10.041 INFO:teuthology.orchestra.run.smithi120.stdout:linux-image-generic set to manually installed. 2022-07-20T02:14:10.041 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:14:10.041 DEBUG:teuthology.orchestra.run.smithi120:> dpkg -s linux-image-generic 2022-07-20T02:14:10.068 INFO:teuthology.orchestra.run.smithi120.stdout:Package: linux-image-generic 2022-07-20T02:14:10.069 INFO:teuthology.orchestra.run.smithi120.stdout:Status: install ok installed 2022-07-20T02:14:10.069 INFO:teuthology.orchestra.run.smithi120.stdout:Priority: optional 2022-07-20T02:14:10.069 INFO:teuthology.orchestra.run.smithi120.stdout:Section: kernel 2022-07-20T02:14:10.069 INFO:teuthology.orchestra.run.smithi120.stdout:Installed-Size: 19 2022-07-20T02:14:10.070 INFO:teuthology.orchestra.run.smithi120.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:14:10.070 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: amd64 2022-07-20T02:14:10.078 INFO:teuthology.orchestra.run.smithi120.stdout:Source: linux-meta 2022-07-20T02:14:10.078 INFO:teuthology.orchestra.run.smithi120.stdout:Version: 5.4.0.122.123 2022-07-20T02:14:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:14:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:14:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Recommends: thermald 2022-07-20T02:14:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Description: Generic Linux kernel image 2022-07-20T02:14:10.080 INFO:teuthology.orchestra.run.smithi120.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:14:10.080 INFO:teuthology.orchestra.run.smithi120.stdout: available. 2022-07-20T02:14:10.080 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:14:10.080 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-07-20T02:14:10.081 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:14:10.142 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2022-07-20T02:14:10.180 DEBUG:teuthology.parallel:result is None 2022-07-20T02:14:29.795 INFO:teuthology.orchestra.run.smithi159.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:14:29.815 INFO:teuthology.orchestra.run.smithi159.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:14:29.910 INFO:teuthology.orchestra.run.smithi159.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:14:30.111 INFO:teuthology.orchestra.run.smithi159.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:14:30.339 INFO:teuthology.orchestra.run.smithi159.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:14:30.831 INFO:teuthology.orchestra.run.smithi159.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:14:30.921 INFO:teuthology.orchestra.run.smithi159.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:14:31.258 INFO:teuthology.orchestra.run.smithi159.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:14:31.335 INFO:teuthology.orchestra.run.smithi159.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:14:32.090 INFO:teuthology.orchestra.run.smithi159.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:14:32.455 INFO:teuthology.orchestra.run.smithi159.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:14:32.514 INFO:teuthology.orchestra.run.smithi159.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:14:34.896 INFO:teuthology.orchestra.run.smithi159.stdout:Fetched 10.3 MB in 34s (305 kB/s) 2022-07-20T02:14:36.402 INFO:teuthology.orchestra.run.smithi159.stdout:Reading package lists... 2022-07-20T02:14:36.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:14:36.493 INFO:teuthology.orchestra.run.smithi159.stdout:Reading package lists... 2022-07-20T02:14:36.673 INFO:teuthology.orchestra.run.smithi159.stdout:Building dependency tree... 2022-07-20T02:14:36.674 INFO:teuthology.orchestra.run.smithi159.stdout:Reading state information... 2022-07-20T02:14:36.867 INFO:teuthology.orchestra.run.smithi159.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:14:36.868 INFO:teuthology.orchestra.run.smithi159.stdout:linux-image-generic set to manually installed. 2022-07-20T02:14:36.869 INFO:teuthology.orchestra.run.smithi159.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:14:36.869 DEBUG:teuthology.orchestra.run.smithi159:> dpkg -s linux-image-generic 2022-07-20T02:14:36.960 INFO:teuthology.orchestra.run.smithi159.stdout:Package: linux-image-generic 2022-07-20T02:14:36.960 INFO:teuthology.orchestra.run.smithi159.stdout:Status: install ok installed 2022-07-20T02:14:36.961 INFO:teuthology.orchestra.run.smithi159.stdout:Priority: optional 2022-07-20T02:14:36.961 INFO:teuthology.orchestra.run.smithi159.stdout:Section: kernel 2022-07-20T02:14:36.961 INFO:teuthology.orchestra.run.smithi159.stdout:Installed-Size: 19 2022-07-20T02:14:37.004 INFO:teuthology.orchestra.run.smithi159.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:14:37.004 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: amd64 2022-07-20T02:14:37.005 INFO:teuthology.orchestra.run.smithi159.stdout:Source: linux-meta 2022-07-20T02:14:37.005 INFO:teuthology.orchestra.run.smithi159.stdout:Version: 5.4.0.122.123 2022-07-20T02:14:37.014 INFO:teuthology.orchestra.run.smithi159.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:14:37.014 INFO:teuthology.orchestra.run.smithi159.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:14:37.015 INFO:teuthology.orchestra.run.smithi159.stdout:Recommends: thermald 2022-07-20T02:14:37.015 INFO:teuthology.orchestra.run.smithi159.stdout:Description: Generic Linux kernel image 2022-07-20T02:14:37.015 INFO:teuthology.orchestra.run.smithi159.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:14:37.016 INFO:teuthology.orchestra.run.smithi159.stdout: available. 2022-07-20T02:14:37.021 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:14:37.021 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2022-07-20T02:14:37.022 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:14:37.051 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2022-07-20T02:14:37.076 DEBUG:teuthology.parallel:result is None 2022-07-20T02:14:37.076 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-20T02:14:37.289 INFO:teuthology.task.internal:Creating test directory... 2022-07-20T02:14:37.301 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:14:37.348 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:14:37.388 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:14:37.440 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:14:37.567 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-20T02:14:37.624 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-20T02:14:37.739 INFO:teuthology.task.internal:Creating archive directory... 2022-07-20T02:14:37.741 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:14:37.789 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:14:37.800 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:14:37.816 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:14:37.985 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-20T02:14:38.056 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-20T02:14:38.056 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:14:38.059 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:14:38.079 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:14:38.090 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.091 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:14:38.096 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.099 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.109 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.115 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.120 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.122 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.129 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:14:38.130 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-20T02:14:38.177 INFO:teuthology.task.internal:Configuring sudo... 2022-07-20T02:14:38.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:14:38.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:14:38.256 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:14:38.295 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:14:38.349 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-20T02:14:38.431 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-20T02:14:38.432 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:14:38.440 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:14:38.463 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:14:38.470 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:14:38.506 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:14:38.571 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:14:38.615 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:14:38.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:14:38.674 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:14:38.706 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:14:38.720 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:14:38.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:14:38.812 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:14:38.841 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:14:38.853 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:14:38.853 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:14:38.926 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:14:38.935 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:14:38.941 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:14:38.941 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:14:39.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2022-07-20T02:14:39.014 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2022-07-20T02:14:39.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2022-07-20T02:14:39.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2022-07-20T02:14:39.079 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-20T02:14:39.104 INFO:teuthology.task.internal:Starting timer... 2022-07-20T02:14:39.104 INFO:teuthology.run_tasks:Running task pcp... 2022-07-20T02:14:39.168 INFO:teuthology.run_tasks:Running task selinux... 2022-07-20T02:14:39.232 DEBUG:teuthology.task.selinux:Excluding smithi026: OS 'ubuntu' does not support SELinux 2022-07-20T02:14:39.233 DEBUG:teuthology.task.selinux:Excluding smithi120: OS 'ubuntu' does not support SELinux 2022-07-20T02:14:39.233 DEBUG:teuthology.task.selinux:Excluding smithi159: OS 'ubuntu' does not support SELinux 2022-07-20T02:14:39.234 DEBUG:teuthology.task.selinux:Excluding smithi176: OS 'ubuntu' does not support SELinux 2022-07-20T02:14:39.234 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-20T02:14:39.235 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-07-20T02:14:39.235 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-20T02:14:39.236 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-20T02:14:39.281 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-07-20T02:14:39.324 INFO:teuthology.repo_utils:Fetching main from origin 2022-07-20T02:14:39.819 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-20T02:14:40.097 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-20T02:14:40.099 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi120.front.sepia.ceph.com,smithi159.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-20T02:22:27.812 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2022-07-20T02:22:27.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2022-07-20T02:22:27.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:22:28.023 DEBUG:teuthology.orchestra.run.smithi026:> true 2022-07-20T02:22:28.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-07-20T02:22:28.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:22:28.267 DEBUG:teuthology.orchestra.run.smithi120:> true 2022-07-20T02:22:28.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2022-07-20T02:22:28.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:22:28.555 DEBUG:teuthology.orchestra.run.smithi159:> true 2022-07-20T02:22:28.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-07-20T02:22:28.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:22:28.857 DEBUG:teuthology.orchestra.run.smithi176:> true 2022-07-20T02:22:28.977 INFO:teuthology.run_tasks:Running task clock... 2022-07-20T02:22:29.010 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-20T02:22:29.014 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T02:22:29.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T02:22:29.029 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T02:22:29.029 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T02:22:29.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T02:22:29.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T02:22:29.046 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T02:22:29.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T02:22:29.057 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T02:22:29.058 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Command line: ntpd -gq 2022-07-20T02:22:29.059 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: proto: precision = 0.124 usec (-23) 2022-07-20T02:22:29.059 INFO:teuthology.orchestra.run.smithi026.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.060 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.060 INFO:teuthology.orchestra.run.smithi026.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.061 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.062 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T02:22:29.062 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T02:22:29.062 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T02:22:29.062 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen normally on 3 ens1f0 172.21.15.26:123 2022-07-20T02:22:29.063 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen normally on 4 lo [::1]:123 2022-07-20T02:22:29.063 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:cac0%4]:123 2022-07-20T02:22:29.063 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:29 ntpd[17927]: Listening on routing socket on fd #22 for interface updates 2022-07-20T02:22:29.064 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T02:22:29.064 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Command line: ntpd -gq 2022-07-20T02:22:29.064 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: proto: precision = 0.042 usec (-24) 2022-07-20T02:22:29.065 INFO:teuthology.orchestra.run.smithi120.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.065 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.066 INFO:teuthology.orchestra.run.smithi120.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.067 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.067 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T02:22:29.067 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T02:22:29.068 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T02:22:29.068 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen normally on 3 enp3s0f1 172.21.15.120:123 2022-07-20T02:22:29.068 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen normally on 4 lo [::1]:123 2022-07-20T02:22:29.068 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:478f%5]:123 2022-07-20T02:22:29.068 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:29 ntpd[18155]: Listening on routing socket on fd #22 for interface updates 2022-07-20T02:22:29.080 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T02:22:29.081 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Command line: ntpd -gq 2022-07-20T02:22:29.081 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: proto: precision = 0.104 usec (-23) 2022-07-20T02:22:29.082 INFO:teuthology.orchestra.run.smithi159.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.082 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.083 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.083 INFO:teuthology.orchestra.run.smithi159.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.085 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T02:22:29.086 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T02:22:29.086 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T02:22:29.086 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen normally on 3 enp3s0f1 172.21.15.159:123 2022-07-20T02:22:29.086 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen normally on 4 lo [::1]:123 2022-07-20T02:22:29.087 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:724f%5]:123 2022-07-20T02:22:29.087 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:29 ntpd[18106]: Listening on routing socket on fd #22 for interface updates 2022-07-20T02:22:29.088 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T02:22:29.088 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Command line: ntpd -gq 2022-07-20T02:22:29.088 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: proto: precision = 0.120 usec (-23) 2022-07-20T02:22:29.089 INFO:teuthology.orchestra.run.smithi176.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.090 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T02:22:29.091 INFO:teuthology.orchestra.run.smithi176.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.092 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T02:22:29.092 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T02:22:29.092 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T02:22:29.093 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T02:22:29.093 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen normally on 3 enp3s0f1 172.21.15.176:123 2022-07-20T02:22:29.093 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen normally on 4 lo [::1]:123 2022-07-20T02:22:29.093 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f61%5]:123 2022-07-20T02:22:29.094 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:29 ntpd[18225]: Listening on routing socket on fd #22 for interface updates 2022-07-20T02:22:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:30 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:30.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.072 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.072 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.073 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.073 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:30.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:30 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:30.082 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.083 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.083 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:30.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:30.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:30.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:30 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:31.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.062 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:31.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:31 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:31.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.070 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:31 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:31.079 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:31 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.079 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:31 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.083 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.083 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.084 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:31.084 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:31.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:31.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:31 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:32 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:32.067 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.067 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:32.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:32.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.070 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:32.070 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:32 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:32.084 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:32.085 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:32.086 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:32.086 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:32 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.060 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:33.062 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:33 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:33.063 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.063 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.063 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.064 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.064 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:33.064 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:33 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:33.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:33.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:33 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:33.089 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.090 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.090 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.091 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.091 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.091 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.092 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:33.092 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:33.093 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:33.093 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:33.093 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:33.093 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:33 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:34 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.071 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.071 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.072 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.072 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.073 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:34 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.079 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:34 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.090 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.090 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.090 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.090 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.091 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.091 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.091 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:34.092 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:34.108 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:34.110 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:34.110 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:34 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:35.068 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.069 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.070 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:35.070 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:35 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:35.072 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:35 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:35.086 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:35.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:35 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:35.091 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.091 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.092 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.092 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.100 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.101 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.101 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:35.101 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:35.102 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:35.102 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:35.103 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:35.103 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:35 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.064 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.065 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.066 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.066 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.067 INFO:teuthology.orchestra.run.smithi026.stdout:20 Jul 02:22:36 ntpd[17927]: ntpd: time slew -0.001731 s 2022-07-20T02:22:36.067 INFO:teuthology.orchestra.run.smithi026.stdout:ntpd: time slew -0.001731s 2022-07-20T02:22:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-07-20T02:22:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:20 Jul 02:22:36 ntpd[17927]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T02:22:36.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.074 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.075 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.076 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.076 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.077 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.077 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.077 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.078 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.078 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.079 INFO:teuthology.orchestra.run.smithi120.stdout:20 Jul 02:22:36 ntpd[18155]: ntpd: time slew -0.001532 s 2022-07-20T02:22:36.080 INFO:teuthology.orchestra.run.smithi120.stdout:ntpd: time slew -0.001532s 2022-07-20T02:22:36.081 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-07-20T02:22:36.081 INFO:teuthology.orchestra.run.smithi120.stderr:20 Jul 02:22:36 ntpd[18155]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T02:22:36.084 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.085 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.085 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.085 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.086 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.086 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:36 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.087 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.088 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:36.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:36.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:36.089 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:36.090 INFO:teuthology.orchestra.run.smithi176.stdout:20 Jul 02:22:36 ntpd[18225]: ntpd: time slew -0.002659 s 2022-07-20T02:22:36.091 INFO:teuthology.orchestra.run.smithi176.stdout:ntpd: time slew -0.002659s 2022-07-20T02:22:36.091 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-07-20T02:22:36.092 INFO:teuthology.orchestra.run.smithi176.stderr:20 Jul 02:22:36 ntpd[18225]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T02:22:36.117 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T02:22:36.117 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2022-07-20T02:22:36.118 INFO:teuthology.orchestra.run.smithi026.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.118 INFO:teuthology.orchestra.run.smithi026.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.118 INFO:teuthology.orchestra.run.smithi026.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.118 INFO:teuthology.orchestra.run.smithi026.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.138 INFO:teuthology.orchestra.run.smithi120.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T02:22:36.138 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================== 2022-07-20T02:22:36.139 INFO:teuthology.orchestra.run.smithi120.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.139 INFO:teuthology.orchestra.run.smithi120.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.140 INFO:teuthology.orchestra.run.smithi120.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.140 INFO:teuthology.orchestra.run.smithi120.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.142 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T02:22:36.142 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2022-07-20T02:22:36.142 INFO:teuthology.orchestra.run.smithi176.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.143 INFO:teuthology.orchestra.run.smithi176.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.143 INFO:teuthology.orchestra.run.smithi176.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:36.143 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:37.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:37.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:37.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:37.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:37.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:37.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:37.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:37.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:37.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:37.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:37.083 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:37.095 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:37 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:38.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:38.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:38.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:38.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:38.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:38.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:38 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:39.080 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:39.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:39.081 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:39.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:39.082 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T02:22:39.083 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T02:22:39.083 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T02:22:39.083 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T02:22:39.084 INFO:teuthology.orchestra.run.smithi159.stdout:20 Jul 02:22:39 ntpd[18106]: ntpd: time slew -0.000450 s 2022-07-20T02:22:39.085 INFO:teuthology.orchestra.run.smithi159.stdout:ntpd: time slew -0.000450s 2022-07-20T02:22:39.086 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-07-20T02:22:39.087 INFO:teuthology.orchestra.run.smithi159.stderr:20 Jul 02:22:39 ntpd[18106]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T02:22:39.137 INFO:teuthology.orchestra.run.smithi159.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T02:22:39.138 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================== 2022-07-20T02:22:39.138 INFO:teuthology.orchestra.run.smithi159.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:39.139 INFO:teuthology.orchestra.run.smithi159.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:39.139 INFO:teuthology.orchestra.run.smithi159.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:39.140 INFO:teuthology.orchestra.run.smithi159.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T02:22:39.141 INFO:teuthology.run_tasks:Running task install... 2022-07-20T02:22:39.194 DEBUG:teuthology.task.install:project ceph 2022-07-20T02:22:39.199 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}} 2022-07-20T02:22:39.200 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'} 2022-07-20T02:22:39.201 INFO:teuthology.task.install:Using flavor: default 2022-07-20T02:22:39.215 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-20T02:22:39.216 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2022-07-20T02:22:39.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2022-07-20T02:22:39.268 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2022-07-20T02:22:39.274 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.275 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-07-20T02:22:39.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-key list | grep Ceph 2022-07-20T02:22:39.327 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-07-20T02:22:39.347 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.348 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-20T02:22:39.349 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-key list | grep Ceph 2022-07-20T02:22:39.387 INFO:teuthology.orchestra.run.smithi026.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T02:22:39.409 INFO:teuthology.orchestra.run.smithi159.stdout:Ubuntu 2022-07-20T02:22:39.411 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-07-20T02:22:39.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo apt-key list | grep Ceph 2022-07-20T02:22:39.443 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T02:22:39.443 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T02:22:39.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2022-07-20T02:22:39.465 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-07-20T02:22:39.466 INFO:teuthology.orchestra.run.smithi120.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T02:22:39.468 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-key list | grep Ceph 2022-07-20T02:22:39.510 INFO:teuthology.orchestra.run.smithi120.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T02:22:39.510 INFO:teuthology.orchestra.run.smithi120.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T02:22:39.511 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-07-20T02:22:39.516 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2022-07-20T02:22:39.517 INFO:teuthology.orchestra.run.smithi159.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T02:22:39.522 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.522 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T02:22:39.523 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T02:22:39.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:22:39.582 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T02:22:39.584 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-07-20T02:22:39.584 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.585 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T02:22:39.585 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T02:22:39.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:22:39.592 INFO:teuthology.orchestra.run.smithi159.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T02:22:39.593 INFO:teuthology.orchestra.run.smithi159.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T02:22:39.593 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-20T02:22:39.677 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T02:22:39.678 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T02:22:39.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-07-20T02:22:39.706 INFO:teuthology.orchestra.run.smithi159.stdout:Ubuntu 2022-07-20T02:22:39.707 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.707 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T02:22:39.708 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T02:22:39.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:22:39.800 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-07-20T02:22:39.801 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:22:39.802 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T02:22:39.803 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T02:22:39.803 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:22:39.940 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T02:22:39.943 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T02:22:39.974 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T02:22:39.975 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T02:22:40.034 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T02:22:40.035 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T02:22:40.117 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:22:40.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T02:22:40.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2022-07-20T02:22:40.157 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:22:40.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T02:22:40.159 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T02:22:40.160 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T02:22:40.189 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get update 2022-07-20T02:22:40.213 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:22:40.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T02:22:40.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo apt-get update 2022-07-20T02:22:40.297 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:22:40.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T02:22:40.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2022-07-20T02:22:40.383 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:22:40.384 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-20T02:22:40.398 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-20T02:22:40.435 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T02:22:40.437 INFO:teuthology.orchestra.run.smithi159.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T02:22:40.442 INFO:teuthology.orchestra.run.smithi120.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T02:22:40.458 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T02:22:40.463 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T02:22:40.464 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:22:40.465 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T02:22:40.471 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-20T02:22:40.483 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T02:22:40.489 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-20T02:22:40.489 INFO:teuthology.orchestra.run.smithi120.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T02:22:40.490 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T02:22:40.501 INFO:teuthology.orchestra.run.smithi159.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T02:22:40.504 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:22:40.505 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T02:22:40.506 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T02:22:40.518 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T02:22:40.525 INFO:teuthology.orchestra.run.smithi159.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T02:22:40.530 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T02:22:40.542 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T02:22:40.548 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T02:22:40.550 INFO:teuthology.orchestra.run.smithi159.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T02:22:40.552 INFO:teuthology.orchestra.run.smithi159.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:22:40.553 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T02:22:40.571 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-20T02:22:40.574 INFO:teuthology.orchestra.run.smithi159.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T02:22:40.579 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T02:22:40.607 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T02:22:40.609 INFO:teuthology.orchestra.run.smithi159.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T02:22:40.633 INFO:teuthology.orchestra.run.smithi159.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-20T02:22:40.644 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T02:22:40.663 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-20T02:22:40.719 INFO:teuthology.orchestra.run.smithi159.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-20T02:22:41.048 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 26.6 kB in 1s (40.7 kB/s) 2022-07-20T02:22:41.084 INFO:teuthology.orchestra.run.smithi159.stdout:Fetched 26.6 kB in 1s (42.5 kB/s) 2022-07-20T02:22:41.132 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 26.6 kB in 1s (41.4 kB/s) 2022-07-20T02:22:41.142 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 26.6 kB in 1s (35.7 kB/s) 2022-07-20T02:22:41.967 INFO:teuthology.orchestra.run.smithi159.stdout:Reading package lists... 2022-07-20T02:22:41.968 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2022-07-20T02:22:41.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T02:22:42.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T02:22:42.015 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-07-20T02:22:42.048 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T02:22:42.049 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-07-20T02:22:42.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T02:22:42.079 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2022-07-20T02:22:42.080 INFO:teuthology.orchestra.run.smithi159.stdout:Reading package lists... 2022-07-20T02:22:42.121 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-07-20T02:22:42.122 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-07-20T02:22:42.242 INFO:teuthology.orchestra.run.smithi159.stdout:Building dependency tree... 2022-07-20T02:22:42.243 INFO:teuthology.orchestra.run.smithi159.stdout:Reading state information... 2022-07-20T02:22:42.258 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2022-07-20T02:22:42.259 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2022-07-20T02:22:42.279 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-07-20T02:22:42.286 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-07-20T02:22:42.295 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-07-20T02:22:42.296 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-07-20T02:22:42.384 INFO:teuthology.orchestra.run.smithi159.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:22:42.384 INFO:teuthology.orchestra.run.smithi159.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:22:42.385 INFO:teuthology.orchestra.run.smithi159.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:22:42.385 INFO:teuthology.orchestra.run.smithi159.stdout:The following additional packages will be installed: 2022-07-20T02:22:42.386 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.386 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.387 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T02:22:42.387 INFO:teuthology.orchestra.run.smithi159.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T02:22:42.388 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T02:22:42.389 INFO:teuthology.orchestra.run.smithi159.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T02:22:42.389 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T02:22:42.399 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T02:22:42.399 INFO:teuthology.orchestra.run.smithi159.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T02:22:42.400 INFO:teuthology.orchestra.run.smithi159.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T02:22:42.400 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T02:22:42.401 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T02:22:42.401 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T02:22:42.402 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T02:22:42.403 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T02:22:42.403 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T02:22:42.404 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T02:22:42.404 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T02:22:42.404 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T02:22:42.405 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet 2022-07-20T02:22:42.406 INFO:teuthology.orchestra.run.smithi159.stdout:Suggested packages: 2022-07-20T02:22:42.406 INFO:teuthology.orchestra.run.smithi159.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T02:22:42.406 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T02:22:42.407 INFO:teuthology.orchestra.run.smithi159.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T02:22:42.407 INFO:teuthology.orchestra.run.smithi159.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T02:22:42.408 INFO:teuthology.orchestra.run.smithi159.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T02:22:42.408 INFO:teuthology.orchestra.run.smithi159.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T02:22:42.409 INFO:teuthology.orchestra.run.smithi159.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T02:22:42.409 INFO:teuthology.orchestra.run.smithi159.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T02:22:42.410 INFO:teuthology.orchestra.run.smithi159.stdout:Recommended packages: 2022-07-20T02:22:42.410 INFO:teuthology.orchestra.run.smithi159.stdout: btrfs-tools 2022-07-20T02:22:42.418 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:22:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:22:42.419 INFO:teuthology.orchestra.run.smithi026.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:22:42.419 INFO:teuthology.orchestra.run.smithi026.stdout:The following additional packages will be installed: 2022-07-20T02:22:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T02:22:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T02:22:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T02:22:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T02:22:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T02:22:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T02:22:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T02:22:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T02:22:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T02:22:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T02:22:42.433 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T02:22:42.433 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T02:22:42.434 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T02:22:42.434 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T02:22:42.435 INFO:teuthology.orchestra.run.smithi026.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T02:22:42.435 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T02:22:42.436 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T02:22:42.436 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet 2022-07-20T02:22:42.439 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2022-07-20T02:22:42.439 INFO:teuthology.orchestra.run.smithi026.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T02:22:42.440 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T02:22:42.440 INFO:teuthology.orchestra.run.smithi026.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T02:22:42.440 INFO:teuthology.orchestra.run.smithi026.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T02:22:42.441 INFO:teuthology.orchestra.run.smithi026.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T02:22:42.441 INFO:teuthology.orchestra.run.smithi026.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T02:22:42.442 INFO:teuthology.orchestra.run.smithi026.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T02:22:42.442 INFO:teuthology.orchestra.run.smithi026.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T02:22:42.458 INFO:teuthology.orchestra.run.smithi026.stdout:Recommended packages: 2022-07-20T02:22:42.459 INFO:teuthology.orchestra.run.smithi026.stdout: btrfs-tools 2022-07-20T02:22:42.462 INFO:teuthology.orchestra.run.smithi120.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:22:42.463 INFO:teuthology.orchestra.run.smithi120.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:22:42.463 INFO:teuthology.orchestra.run.smithi120.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:22:42.464 INFO:teuthology.orchestra.run.smithi120.stdout:The following additional packages will be installed: 2022-07-20T02:22:42.464 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.465 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.465 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T02:22:42.466 INFO:teuthology.orchestra.run.smithi120.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T02:22:42.466 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T02:22:42.467 INFO:teuthology.orchestra.run.smithi120.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T02:22:42.467 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T02:22:42.468 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T02:22:42.468 INFO:teuthology.orchestra.run.smithi120.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T02:22:42.469 INFO:teuthology.orchestra.run.smithi120.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T02:22:42.469 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T02:22:42.469 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T02:22:42.470 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T02:22:42.470 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T02:22:42.471 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T02:22:42.471 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T02:22:42.472 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T02:22:42.472 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T02:22:42.473 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T02:22:42.474 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet 2022-07-20T02:22:42.474 INFO:teuthology.orchestra.run.smithi120.stdout:Suggested packages: 2022-07-20T02:22:42.475 INFO:teuthology.orchestra.run.smithi120.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T02:22:42.475 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T02:22:42.475 INFO:teuthology.orchestra.run.smithi120.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T02:22:42.476 INFO:teuthology.orchestra.run.smithi120.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T02:22:42.476 INFO:teuthology.orchestra.run.smithi120.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T02:22:42.477 INFO:teuthology.orchestra.run.smithi120.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T02:22:42.477 INFO:teuthology.orchestra.run.smithi120.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T02:22:42.478 INFO:teuthology.orchestra.run.smithi120.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T02:22:42.478 INFO:teuthology.orchestra.run.smithi120.stdout:Recommended packages: 2022-07-20T02:22:42.479 INFO:teuthology.orchestra.run.smithi120.stdout: btrfs-tools 2022-07-20T02:22:42.479 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:22:42.480 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:22:42.480 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:22:42.481 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2022-07-20T02:22:42.481 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.482 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.482 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T02:22:42.483 INFO:teuthology.orchestra.run.smithi176.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T02:22:42.483 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T02:22:42.483 INFO:teuthology.orchestra.run.smithi176.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T02:22:42.484 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T02:22:42.484 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T02:22:42.485 INFO:teuthology.orchestra.run.smithi176.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T02:22:42.485 INFO:teuthology.orchestra.run.smithi176.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T02:22:42.486 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T02:22:42.486 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T02:22:42.486 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T02:22:42.487 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T02:22:42.487 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T02:22:42.488 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T02:22:42.488 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T02:22:42.489 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T02:22:42.489 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T02:22:42.489 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet 2022-07-20T02:22:42.490 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2022-07-20T02:22:42.490 INFO:teuthology.orchestra.run.smithi176.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T02:22:42.491 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T02:22:42.491 INFO:teuthology.orchestra.run.smithi176.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T02:22:42.492 INFO:teuthology.orchestra.run.smithi176.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T02:22:42.493 INFO:teuthology.orchestra.run.smithi176.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T02:22:42.493 INFO:teuthology.orchestra.run.smithi176.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T02:22:42.494 INFO:teuthology.orchestra.run.smithi176.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T02:22:42.494 INFO:teuthology.orchestra.run.smithi176.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T02:22:42.495 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2022-07-20T02:22:42.495 INFO:teuthology.orchestra.run.smithi176.stdout: btrfs-tools 2022-07-20T02:22:42.498 INFO:teuthology.orchestra.run.smithi159.stdout:The following NEW packages will be installed: 2022-07-20T02:22:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T02:22:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T02:22:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T02:22:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T02:22:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T02:22:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T02:22:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T02:22:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T02:22:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T02:22:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T02:22:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T02:22:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T02:22:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T02:22:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T02:22:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T02:22:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T02:22:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T02:22:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T02:22:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T02:22:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T02:22:42.510 INFO:teuthology.orchestra.run.smithi159.stdout:The following packages will be upgraded: 2022-07-20T02:22:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 librbd1 2022-07-20T02:22:42.529 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-07-20T02:22:42.530 INFO:teuthology.orchestra.run.smithi176.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T02:22:42.530 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.531 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.531 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T02:22:42.531 INFO:teuthology.orchestra.run.smithi176.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T02:22:42.532 INFO:teuthology.orchestra.run.smithi176.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T02:22:42.532 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T02:22:42.533 INFO:teuthology.orchestra.run.smithi176.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T02:22:42.533 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T02:22:42.534 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T02:22:42.534 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T02:22:42.534 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T02:22:42.535 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T02:22:42.535 INFO:teuthology.orchestra.run.smithi176.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T02:22:42.536 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T02:22:42.536 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T02:22:42.537 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T02:22:42.537 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T02:22:42.538 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T02:22:42.538 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T02:22:42.538 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T02:22:42.539 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T02:22:42.540 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2022-07-20T02:22:42.540 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 librbd1 2022-07-20T02:22:42.545 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-07-20T02:22:42.545 INFO:teuthology.orchestra.run.smithi120.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T02:22:42.546 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.546 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.546 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T02:22:42.547 INFO:teuthology.orchestra.run.smithi120.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T02:22:42.547 INFO:teuthology.orchestra.run.smithi120.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T02:22:42.548 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T02:22:42.548 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T02:22:42.549 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T02:22:42.549 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T02:22:42.550 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T02:22:42.550 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T02:22:42.550 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T02:22:42.551 INFO:teuthology.orchestra.run.smithi120.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T02:22:42.551 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T02:22:42.552 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T02:22:42.552 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T02:22:42.553 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T02:22:42.553 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T02:22:42.554 INFO:teuthology.orchestra.run.smithi120.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T02:22:42.554 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T02:22:42.555 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T02:22:42.555 INFO:teuthology.orchestra.run.smithi120.stdout:The following packages will be upgraded: 2022-07-20T02:22:42.555 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 librbd1 2022-07-20T02:22:42.556 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2022-07-20T02:22:42.556 INFO:teuthology.orchestra.run.smithi026.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T02:22:42.557 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T02:22:42.557 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T02:22:42.558 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T02:22:42.558 INFO:teuthology.orchestra.run.smithi026.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T02:22:42.559 INFO:teuthology.orchestra.run.smithi026.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T02:22:42.559 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T02:22:42.560 INFO:teuthology.orchestra.run.smithi026.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T02:22:42.560 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T02:22:42.560 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T02:22:42.561 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T02:22:42.561 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T02:22:42.562 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T02:22:42.562 INFO:teuthology.orchestra.run.smithi026.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T02:22:42.563 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T02:22:42.563 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T02:22:42.564 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T02:22:42.564 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T02:22:42.564 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T02:22:42.565 INFO:teuthology.orchestra.run.smithi026.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T02:22:42.565 INFO:teuthology.orchestra.run.smithi026.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T02:22:42.566 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T02:22:42.566 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages will be upgraded: 2022-07-20T02:22:42.567 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 librbd1 2022-07-20T02:22:42.569 INFO:teuthology.orchestra.run.smithi159.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:22:42.569 INFO:teuthology.orchestra.run.smithi159.stdout:Need to get 153 MB of archives. 2022-07-20T02:22:42.569 INFO:teuthology.orchestra.run.smithi159.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T02:22:42.569 INFO:teuthology.orchestra.run.smithi159.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T02:22:42.570 INFO:teuthology.orchestra.run.smithi159.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T02:22:42.581 INFO:teuthology.orchestra.run.smithi159.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T02:22:42.589 INFO:teuthology.orchestra.run.smithi159.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T02:22:42.611 INFO:teuthology.orchestra.run.smithi026.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:22:42.612 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 153 MB of archives. 2022-07-20T02:22:42.612 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T02:22:42.612 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T02:22:42.616 INFO:teuthology.orchestra.run.smithi159.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T02:22:42.622 INFO:teuthology.orchestra.run.smithi120.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:22:42.623 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 153 MB of archives. 2022-07-20T02:22:42.623 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T02:22:42.623 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T02:22:42.624 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T02:22:42.641 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T02:22:42.644 INFO:teuthology.orchestra.run.smithi159.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T02:22:42.646 INFO:teuthology.orchestra.run.smithi159.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T02:22:42.649 INFO:teuthology.orchestra.run.smithi159.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T02:22:42.650 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T02:22:42.661 INFO:teuthology.orchestra.run.smithi159.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T02:22:42.674 INFO:teuthology.orchestra.run.smithi159.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T02:22:42.679 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T02:22:42.680 INFO:teuthology.orchestra.run.smithi159.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T02:22:42.689 INFO:teuthology.orchestra.run.smithi159.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T02:22:42.692 INFO:teuthology.orchestra.run.smithi159.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T02:22:42.693 INFO:teuthology.orchestra.run.smithi159.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T02:22:42.694 INFO:teuthology.orchestra.run.smithi159.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T02:22:42.695 INFO:teuthology.orchestra.run.smithi159.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T02:22:42.695 INFO:teuthology.orchestra.run.smithi159.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T02:22:42.695 INFO:teuthology.orchestra.run.smithi159.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T02:22:42.697 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T02:22:42.697 INFO:teuthology.orchestra.run.smithi176.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:22:42.698 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 153 MB of archives. 2022-07-20T02:22:42.700 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T02:22:42.700 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T02:22:42.707 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T02:22:42.715 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T02:22:42.716 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T02:22:42.716 INFO:teuthology.orchestra.run.smithi159.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T02:22:42.716 INFO:teuthology.orchestra.run.smithi159.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T02:22:42.717 INFO:teuthology.orchestra.run.smithi159.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T02:22:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T02:22:42.719 INFO:teuthology.orchestra.run.smithi159.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T02:22:42.720 INFO:teuthology.orchestra.run.smithi159.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T02:22:42.720 INFO:teuthology.orchestra.run.smithi159.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T02:22:42.720 INFO:teuthology.orchestra.run.smithi159.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T02:22:42.721 INFO:teuthology.orchestra.run.smithi159.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T02:22:42.725 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T02:22:42.726 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T02:22:42.727 INFO:teuthology.orchestra.run.smithi159.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T02:22:42.727 INFO:teuthology.orchestra.run.smithi159.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T02:22:42.728 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T02:22:42.732 INFO:teuthology.orchestra.run.smithi159.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T02:22:42.749 INFO:teuthology.orchestra.run.smithi159.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T02:22:42.749 INFO:teuthology.orchestra.run.smithi159.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T02:22:42.753 INFO:teuthology.orchestra.run.smithi159.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T02:22:42.753 INFO:teuthology.orchestra.run.smithi159.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T02:22:42.754 INFO:teuthology.orchestra.run.smithi159.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T02:22:42.754 INFO:teuthology.orchestra.run.smithi159.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T02:22:42.755 INFO:teuthology.orchestra.run.smithi159.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T02:22:42.756 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T02:22:42.757 INFO:teuthology.orchestra.run.smithi120.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T02:22:42.757 INFO:teuthology.orchestra.run.smithi120.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T02:22:42.758 INFO:teuthology.orchestra.run.smithi120.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T02:22:42.758 INFO:teuthology.orchestra.run.smithi120.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T02:22:42.759 INFO:teuthology.orchestra.run.smithi120.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T02:22:42.759 INFO:teuthology.orchestra.run.smithi120.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T02:22:42.760 INFO:teuthology.orchestra.run.smithi120.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T02:22:42.760 INFO:teuthology.orchestra.run.smithi120.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T02:22:42.762 INFO:teuthology.orchestra.run.smithi159.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T02:22:42.763 INFO:teuthology.orchestra.run.smithi159.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T02:22:42.763 INFO:teuthology.orchestra.run.smithi120.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T02:22:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T02:22:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T02:22:42.765 INFO:teuthology.orchestra.run.smithi159.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T02:22:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T02:22:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T02:22:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T02:22:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T02:22:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T02:22:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T02:22:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T02:22:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T02:22:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T02:22:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T02:22:42.806 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T02:22:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T02:22:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T02:22:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T02:22:42.810 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T02:22:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T02:22:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T02:22:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T02:22:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T02:22:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T02:22:42.838 INFO:teuthology.orchestra.run.smithi159.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T02:22:42.843 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T02:22:42.843 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T02:22:42.881 INFO:teuthology.orchestra.run.smithi159.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T02:22:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T02:22:42.931 INFO:teuthology.orchestra.run.smithi120.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T02:22:42.932 INFO:teuthology.orchestra.run.smithi159.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T02:22:42.932 INFO:teuthology.orchestra.run.smithi159.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T02:22:42.943 INFO:teuthology.orchestra.run.smithi159.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T02:22:42.944 INFO:teuthology.orchestra.run.smithi159.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T02:22:42.944 INFO:teuthology.orchestra.run.smithi159.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T02:22:42.959 INFO:teuthology.orchestra.run.smithi159.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T02:22:42.960 INFO:teuthology.orchestra.run.smithi159.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T02:22:42.961 INFO:teuthology.orchestra.run.smithi159.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T02:22:42.961 INFO:teuthology.orchestra.run.smithi159.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T02:22:42.962 INFO:teuthology.orchestra.run.smithi159.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T02:22:42.963 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T02:22:42.964 INFO:teuthology.orchestra.run.smithi159.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T02:22:42.976 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T02:22:42.987 INFO:teuthology.orchestra.run.smithi159.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T02:22:42.987 INFO:teuthology.orchestra.run.smithi159.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T02:22:42.987 INFO:teuthology.orchestra.run.smithi159.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T02:22:42.988 INFO:teuthology.orchestra.run.smithi159.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T02:22:42.988 INFO:teuthology.orchestra.run.smithi159.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T02:22:42.988 INFO:teuthology.orchestra.run.smithi159.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T02:22:42.988 INFO:teuthology.orchestra.run.smithi159.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T02:22:42.989 INFO:teuthology.orchestra.run.smithi159.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T02:22:42.989 INFO:teuthology.orchestra.run.smithi159.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T02:22:42.989 INFO:teuthology.orchestra.run.smithi120.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T02:22:42.992 INFO:teuthology.orchestra.run.smithi159.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T02:22:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T02:22:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T02:22:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T02:22:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T02:22:43.006 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T02:22:43.015 INFO:teuthology.orchestra.run.smithi120.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T02:22:43.016 INFO:teuthology.orchestra.run.smithi120.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T02:22:43.016 INFO:teuthology.orchestra.run.smithi120.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T02:22:43.016 INFO:teuthology.orchestra.run.smithi120.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T02:22:43.017 INFO:teuthology.orchestra.run.smithi120.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T02:22:43.017 INFO:teuthology.orchestra.run.smithi120.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T02:22:43.017 INFO:teuthology.orchestra.run.smithi120.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T02:22:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T02:22:43.026 INFO:teuthology.orchestra.run.smithi159.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T02:22:43.027 INFO:teuthology.orchestra.run.smithi159.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T02:22:43.027 INFO:teuthology.orchestra.run.smithi159.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T02:22:43.028 INFO:teuthology.orchestra.run.smithi159.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T02:22:43.028 INFO:teuthology.orchestra.run.smithi159.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T02:22:43.029 INFO:teuthology.orchestra.run.smithi159.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T02:22:43.030 INFO:teuthology.orchestra.run.smithi159.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T02:22:43.030 INFO:teuthology.orchestra.run.smithi159.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T02:22:43.031 INFO:teuthology.orchestra.run.smithi120.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T02:22:43.031 INFO:teuthology.orchestra.run.smithi120.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T02:22:43.031 INFO:teuthology.orchestra.run.smithi120.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T02:22:43.032 INFO:teuthology.orchestra.run.smithi120.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T02:22:43.034 INFO:teuthology.orchestra.run.smithi120.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T02:22:43.035 INFO:teuthology.orchestra.run.smithi120.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T02:22:43.043 INFO:teuthology.orchestra.run.smithi120.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T02:22:43.044 INFO:teuthology.orchestra.run.smithi120.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T02:22:43.044 INFO:teuthology.orchestra.run.smithi120.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T02:22:43.044 INFO:teuthology.orchestra.run.smithi120.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T02:22:43.051 INFO:teuthology.orchestra.run.smithi159.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T02:22:43.059 INFO:teuthology.orchestra.run.smithi120.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T02:22:43.060 INFO:teuthology.orchestra.run.smithi120.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T02:22:43.061 INFO:teuthology.orchestra.run.smithi120.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T02:22:43.061 INFO:teuthology.orchestra.run.smithi120.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T02:22:43.062 INFO:teuthology.orchestra.run.smithi120.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T02:22:43.064 INFO:teuthology.orchestra.run.smithi120.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T02:22:43.077 INFO:teuthology.orchestra.run.smithi120.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T02:22:43.077 INFO:teuthology.orchestra.run.smithi120.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T02:22:43.079 INFO:teuthology.orchestra.run.smithi120.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T02:22:43.079 INFO:teuthology.orchestra.run.smithi120.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T02:22:43.080 INFO:teuthology.orchestra.run.smithi120.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T02:22:43.093 INFO:teuthology.orchestra.run.smithi120.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T02:22:43.094 INFO:teuthology.orchestra.run.smithi120.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T02:22:43.104 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T02:22:43.105 INFO:teuthology.orchestra.run.smithi159.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T02:22:43.105 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T02:22:43.106 INFO:teuthology.orchestra.run.smithi120.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T02:22:43.107 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T02:22:43.113 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T02:22:43.113 INFO:teuthology.orchestra.run.smithi159.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T02:22:43.114 INFO:teuthology.orchestra.run.smithi159.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T02:22:43.128 INFO:teuthology.orchestra.run.smithi159.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T02:22:43.128 INFO:teuthology.orchestra.run.smithi159.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T02:22:43.130 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T02:22:43.131 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T02:22:43.131 INFO:teuthology.orchestra.run.smithi159.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T02:22:43.226 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T02:22:43.228 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T02:22:43.246 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T02:22:43.248 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T02:22:43.252 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T02:22:43.254 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T02:22:43.277 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T02:22:43.286 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T02:22:43.287 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T02:22:43.288 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T02:22:43.288 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T02:22:43.289 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T02:22:43.298 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T02:22:43.299 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T02:22:43.300 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T02:22:43.301 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T02:22:43.303 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T02:22:43.304 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T02:22:43.338 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T02:22:43.340 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T02:22:43.340 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T02:22:43.341 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T02:22:43.343 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T02:22:43.344 INFO:teuthology.orchestra.run.smithi026.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T02:22:43.345 INFO:teuthology.orchestra.run.smithi026.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T02:22:43.451 INFO:teuthology.orchestra.run.smithi159.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T02:22:43.452 INFO:teuthology.orchestra.run.smithi159.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T02:22:43.453 INFO:teuthology.orchestra.run.smithi159.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T02:22:43.459 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T02:22:43.459 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T02:22:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T02:22:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T02:22:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T02:22:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T02:22:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T02:22:43.465 INFO:teuthology.orchestra.run.smithi026.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T02:22:43.465 INFO:teuthology.orchestra.run.smithi026.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T02:22:43.466 INFO:teuthology.orchestra.run.smithi026.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T02:22:43.466 INFO:teuthology.orchestra.run.smithi026.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T02:22:43.467 INFO:teuthology.orchestra.run.smithi026.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T02:22:43.467 INFO:teuthology.orchestra.run.smithi026.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T02:22:43.468 INFO:teuthology.orchestra.run.smithi026.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T02:22:43.468 INFO:teuthology.orchestra.run.smithi026.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T02:22:43.469 INFO:teuthology.orchestra.run.smithi026.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T02:22:43.503 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T02:22:43.503 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T02:22:43.504 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T02:22:43.505 INFO:teuthology.orchestra.run.smithi026.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T02:22:43.505 INFO:teuthology.orchestra.run.smithi026.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T02:22:43.509 INFO:teuthology.orchestra.run.smithi026.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T02:22:43.522 INFO:teuthology.orchestra.run.smithi026.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T02:22:43.523 INFO:teuthology.orchestra.run.smithi026.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T02:22:43.564 INFO:teuthology.orchestra.run.smithi026.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T02:22:43.565 INFO:teuthology.orchestra.run.smithi026.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T02:22:43.572 INFO:teuthology.orchestra.run.smithi120.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T02:22:43.578 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T02:22:43.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T02:22:43.583 INFO:teuthology.orchestra.run.smithi026.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T02:22:43.583 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T02:22:43.586 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T02:22:43.586 INFO:teuthology.orchestra.run.smithi026.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T02:22:43.587 INFO:teuthology.orchestra.run.smithi026.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T02:22:43.589 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T02:22:43.590 INFO:teuthology.orchestra.run.smithi026.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T02:22:43.591 INFO:teuthology.orchestra.run.smithi026.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T02:22:43.592 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T02:22:43.593 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T02:22:43.594 INFO:teuthology.orchestra.run.smithi026.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T02:22:43.597 INFO:teuthology.orchestra.run.smithi026.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T02:22:43.598 INFO:teuthology.orchestra.run.smithi026.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T02:22:43.639 INFO:teuthology.orchestra.run.smithi120.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T02:22:43.658 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T02:22:43.663 INFO:teuthology.orchestra.run.smithi026.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T02:22:43.705 INFO:teuthology.orchestra.run.smithi120.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T02:22:43.706 INFO:teuthology.orchestra.run.smithi120.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T02:22:43.771 INFO:teuthology.orchestra.run.smithi120.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T02:22:43.772 INFO:teuthology.orchestra.run.smithi120.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T02:22:43.797 INFO:teuthology.orchestra.run.smithi120.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T02:22:43.992 INFO:teuthology.orchestra.run.smithi026.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T02:22:43.992 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T02:22:44.057 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T02:22:44.059 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T02:22:44.060 INFO:teuthology.orchestra.run.smithi026.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T02:22:44.061 INFO:teuthology.orchestra.run.smithi026.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T02:22:44.088 INFO:teuthology.orchestra.run.smithi026.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T02:22:44.089 INFO:teuthology.orchestra.run.smithi026.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T02:22:44.099 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T02:22:44.111 INFO:teuthology.orchestra.run.smithi176.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T02:22:44.111 INFO:teuthology.orchestra.run.smithi176.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T02:22:44.112 INFO:teuthology.orchestra.run.smithi176.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T02:22:44.112 INFO:teuthology.orchestra.run.smithi176.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T02:22:44.113 INFO:teuthology.orchestra.run.smithi176.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T02:22:44.113 INFO:teuthology.orchestra.run.smithi176.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T02:22:44.118 INFO:teuthology.orchestra.run.smithi026.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T02:22:44.119 INFO:teuthology.orchestra.run.smithi026.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T02:22:44.119 INFO:teuthology.orchestra.run.smithi026.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T02:22:44.120 INFO:teuthology.orchestra.run.smithi026.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T02:22:44.120 INFO:teuthology.orchestra.run.smithi026.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T02:22:44.120 INFO:teuthology.orchestra.run.smithi026.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T02:22:44.121 INFO:teuthology.orchestra.run.smithi026.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T02:22:44.145 INFO:teuthology.orchestra.run.smithi176.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T02:22:44.146 INFO:teuthology.orchestra.run.smithi176.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T02:22:44.150 INFO:teuthology.orchestra.run.smithi026.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T02:22:44.150 INFO:teuthology.orchestra.run.smithi026.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T02:22:44.164 INFO:teuthology.orchestra.run.smithi026.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T02:22:44.225 INFO:teuthology.orchestra.run.smithi176.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T02:22:44.229 INFO:teuthology.orchestra.run.smithi176.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T02:22:44.230 INFO:teuthology.orchestra.run.smithi026.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T02:22:44.233 INFO:teuthology.orchestra.run.smithi176.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T02:22:44.235 INFO:teuthology.orchestra.run.smithi026.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T02:22:44.238 INFO:teuthology.orchestra.run.smithi176.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T02:22:44.239 INFO:teuthology.orchestra.run.smithi176.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T02:22:44.242 INFO:teuthology.orchestra.run.smithi026.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T02:22:44.244 INFO:teuthology.orchestra.run.smithi176.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T02:22:44.244 INFO:teuthology.orchestra.run.smithi176.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T02:22:44.245 INFO:teuthology.orchestra.run.smithi176.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T02:22:44.249 INFO:teuthology.orchestra.run.smithi026.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T02:22:44.249 INFO:teuthology.orchestra.run.smithi026.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T02:22:44.250 INFO:teuthology.orchestra.run.smithi026.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T02:22:44.250 INFO:teuthology.orchestra.run.smithi026.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T02:22:44.250 INFO:teuthology.orchestra.run.smithi026.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T02:22:44.311 INFO:teuthology.orchestra.run.smithi026.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T02:22:44.312 INFO:teuthology.orchestra.run.smithi176.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T02:22:44.312 INFO:teuthology.orchestra.run.smithi176.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T02:22:44.323 INFO:teuthology.orchestra.run.smithi026.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T02:22:44.376 INFO:teuthology.orchestra.run.smithi120.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T02:22:44.383 INFO:teuthology.orchestra.run.smithi120.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T02:22:44.384 INFO:teuthology.orchestra.run.smithi176.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T02:22:44.385 INFO:teuthology.orchestra.run.smithi176.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T02:22:44.386 INFO:teuthology.orchestra.run.smithi176.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T02:22:44.387 INFO:teuthology.orchestra.run.smithi176.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T02:22:44.391 INFO:teuthology.orchestra.run.smithi026.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T02:22:44.391 INFO:teuthology.orchestra.run.smithi026.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T02:22:44.392 INFO:teuthology.orchestra.run.smithi026.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T02:22:44.393 INFO:teuthology.orchestra.run.smithi026.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T02:22:44.407 INFO:teuthology.orchestra.run.smithi176.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T02:22:44.407 INFO:teuthology.orchestra.run.smithi176.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T02:22:44.408 INFO:teuthology.orchestra.run.smithi176.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T02:22:44.408 INFO:teuthology.orchestra.run.smithi176.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T02:22:44.411 INFO:teuthology.orchestra.run.smithi026.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T02:22:44.412 INFO:teuthology.orchestra.run.smithi026.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T02:22:44.412 INFO:teuthology.orchestra.run.smithi026.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T02:22:44.413 INFO:teuthology.orchestra.run.smithi026.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T02:22:44.455 INFO:teuthology.orchestra.run.smithi120.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T02:22:44.471 INFO:teuthology.orchestra.run.smithi176.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T02:22:44.472 INFO:teuthology.orchestra.run.smithi176.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T02:22:44.473 INFO:teuthology.orchestra.run.smithi026.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T02:22:44.473 INFO:teuthology.orchestra.run.smithi026.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T02:22:44.494 INFO:teuthology.orchestra.run.smithi176.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T02:22:44.544 INFO:teuthology.orchestra.run.smithi176.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T02:22:44.548 INFO:teuthology.orchestra.run.smithi176.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T02:22:44.550 INFO:teuthology.orchestra.run.smithi026.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T02:22:44.553 INFO:teuthology.orchestra.run.smithi026.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T02:22:44.646 INFO:teuthology.orchestra.run.smithi176.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T02:22:44.725 INFO:teuthology.orchestra.run.smithi176.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T02:22:44.726 INFO:teuthology.orchestra.run.smithi176.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T02:22:44.750 INFO:teuthology.orchestra.run.smithi176.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T02:22:44.766 INFO:teuthology.orchestra.run.smithi159.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T02:22:44.793 INFO:teuthology.orchestra.run.smithi176.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T02:22:44.876 INFO:teuthology.orchestra.run.smithi176.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T02:22:45.061 INFO:teuthology.orchestra.run.smithi159.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T02:22:45.065 INFO:teuthology.orchestra.run.smithi159.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T02:22:45.148 INFO:teuthology.orchestra.run.smithi159.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T02:22:45.520 INFO:teuthology.orchestra.run.smithi159.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T02:22:45.663 INFO:teuthology.orchestra.run.smithi176.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T02:22:45.722 INFO:teuthology.orchestra.run.smithi176.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T02:22:45.789 INFO:teuthology.orchestra.run.smithi176.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T02:22:47.066 INFO:teuthology.orchestra.run.smithi159.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T02:22:47.067 INFO:teuthology.orchestra.run.smithi159.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T02:22:47.078 INFO:teuthology.orchestra.run.smithi159.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T02:22:47.114 INFO:teuthology.orchestra.run.smithi159.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T02:22:47.117 INFO:teuthology.orchestra.run.smithi159.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T02:22:47.120 INFO:teuthology.orchestra.run.smithi159.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T02:22:47.301 INFO:teuthology.orchestra.run.smithi159.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T02:22:47.303 INFO:teuthology.orchestra.run.smithi159.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T02:22:47.738 INFO:teuthology.orchestra.run.smithi026.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T02:22:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T02:22:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T02:22:48.686 INFO:teuthology.orchestra.run.smithi026.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T02:22:48.744 INFO:teuthology.orchestra.run.smithi026.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T02:22:49.061 INFO:teuthology.orchestra.run.smithi026.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T02:22:49.574 INFO:teuthology.orchestra.run.smithi120.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T02:22:50.100 INFO:teuthology.orchestra.run.smithi159.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T02:22:50.102 INFO:teuthology.orchestra.run.smithi159.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T02:22:50.103 INFO:teuthology.orchestra.run.smithi159.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T02:22:50.327 INFO:teuthology.orchestra.run.smithi026.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T02:22:50.465 INFO:teuthology.orchestra.run.smithi159.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T02:22:50.670 INFO:teuthology.orchestra.run.smithi176.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T02:22:50.755 INFO:teuthology.orchestra.run.smithi120.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T02:22:50.760 INFO:teuthology.orchestra.run.smithi120.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T02:22:50.936 INFO:teuthology.orchestra.run.smithi120.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T02:22:51.060 INFO:teuthology.orchestra.run.smithi159.stdout:Fetched 153 MB in 8s (19.2 MB/s) 2022-07-20T02:22:51.138 INFO:teuthology.orchestra.run.smithi176.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T02:22:51.140 INFO:teuthology.orchestra.run.smithi176.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T02:22:51.167 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T02:22:51.211 INFO:teuthology.orchestra.run.smithi159.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T02:22:51.214 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T02:22:51.249 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T02:22:51.267 INFO:teuthology.orchestra.run.smithi176.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T02:22:51.528 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T02:22:51.539 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T02:22:51.577 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:22:51.733 INFO:teuthology.orchestra.run.smithi120.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T02:22:51.846 INFO:teuthology.orchestra.run.smithi176.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T02:22:51.913 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T02:22:51.926 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T02:22:51.963 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:22:52.266 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T02:22:52.281 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T02:22:52.375 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:22:52.492 INFO:teuthology.orchestra.run.smithi026.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T02:22:52.493 INFO:teuthology.orchestra.run.smithi026.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T02:22:52.503 INFO:teuthology.orchestra.run.smithi026.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T02:22:52.633 INFO:teuthology.orchestra.run.smithi026.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T02:22:52.637 INFO:teuthology.orchestra.run.smithi026.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T02:22:52.657 INFO:teuthology.orchestra.run.smithi026.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T02:22:52.660 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:52.762 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:22:52.937 INFO:teuthology.orchestra.run.smithi026.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T02:22:52.938 INFO:teuthology.orchestra.run.smithi026.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T02:22:53.454 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:53.608 INFO:teuthology.orchestra.run.smithi026.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T02:22:53.609 INFO:teuthology.orchestra.run.smithi026.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T02:22:53.616 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:22:54.149 INFO:teuthology.orchestra.run.smithi176.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T02:22:54.150 INFO:teuthology.orchestra.run.smithi176.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T02:22:54.160 INFO:teuthology.orchestra.run.smithi176.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T02:22:54.249 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T02:22:54.260 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:54.297 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:54.334 INFO:teuthology.orchestra.run.smithi176.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T02:22:54.336 INFO:teuthology.orchestra.run.smithi176.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T02:22:54.341 INFO:teuthology.orchestra.run.smithi176.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T02:22:54.593 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-rados. 2022-07-20T02:22:54.606 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:54.621 INFO:teuthology.orchestra.run.smithi176.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T02:22:54.622 INFO:teuthology.orchestra.run.smithi176.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T02:22:54.663 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:54.741 INFO:teuthology.orchestra.run.smithi120.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T02:22:54.742 INFO:teuthology.orchestra.run.smithi120.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T02:22:54.749 INFO:teuthology.orchestra.run.smithi120.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T02:22:55.019 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T02:22:55.020 INFO:teuthology.orchestra.run.smithi120.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T02:22:55.021 INFO:teuthology.orchestra.run.smithi120.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T02:22:55.021 INFO:teuthology.orchestra.run.smithi120.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T02:22:55.022 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:22:55.071 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:55.288 INFO:teuthology.orchestra.run.smithi176.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T02:22:55.289 INFO:teuthology.orchestra.run.smithi176.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T02:22:55.372 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T02:22:55.392 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:55.411 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:55.431 INFO:teuthology.orchestra.run.smithi120.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T02:22:55.431 INFO:teuthology.orchestra.run.smithi120.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T02:22:55.690 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T02:22:55.710 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:22:55.747 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:56.096 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T02:22:56.106 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T02:22:56.150 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T02:22:56.436 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T02:22:56.441 INFO:teuthology.orchestra.run.smithi120.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T02:22:56.442 INFO:teuthology.orchestra.run.smithi120.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T02:22:56.449 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:56.494 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:56.847 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T02:22:56.870 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T02:22:56.889 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:22:57.229 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T02:22:57.238 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T02:22:57.281 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:22:57.585 INFO:teuthology.orchestra.run.smithi026.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T02:22:57.586 INFO:teuthology.orchestra.run.smithi026.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T02:22:57.587 INFO:teuthology.orchestra.run.smithi026.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T02:22:57.651 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package librgw2. 2022-07-20T02:22:57.661 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:57.707 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:58.265 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T02:22:58.277 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:58.322 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:58.450 INFO:teuthology.orchestra.run.smithi176.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T02:22:58.451 INFO:teuthology.orchestra.run.smithi176.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T02:22:58.452 INFO:teuthology.orchestra.run.smithi176.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T02:22:58.489 INFO:teuthology.orchestra.run.smithi026.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T02:22:58.583 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T02:22:58.598 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T02:22:58.638 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:22:58.833 INFO:teuthology.orchestra.run.smithi176.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T02:22:59.135 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 153 MB in 16s (9,594 kB/s) 2022-07-20T02:22:59.136 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package liboath0. 2022-07-20T02:22:59.157 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T02:22:59.204 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T02:22:59.273 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T02:22:59.305 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T02:22:59.308 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T02:22:59.350 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T02:22:59.411 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 153 MB in 16s (9,407 kB/s) 2022-07-20T02:22:59.505 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T02:22:59.520 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:59.556 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T02:22:59.563 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:22:59.600 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T02:22:59.601 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T02:22:59.628 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T02:22:59.635 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T02:22:59.642 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T02:22:59.689 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:22:59.943 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-common. 2022-07-20T02:22:59.955 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:22:59.968 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T02:22:59.979 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T02:22:59.979 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T02:23:00.020 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T02:23:00.021 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:00.022 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:23:00.039 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:23:00.231 INFO:teuthology.orchestra.run.smithi120.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T02:23:00.231 INFO:teuthology.orchestra.run.smithi120.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T02:23:00.232 INFO:teuthology.orchestra.run.smithi120.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T02:23:00.304 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T02:23:00.311 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T02:23:00.333 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T02:23:00.340 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:23:00.349 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T02:23:00.442 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:23:00.571 INFO:teuthology.orchestra.run.smithi120.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T02:23:00.647 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T02:23:00.684 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T02:23:00.768 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:23:00.775 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:00.923 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:01.138 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:01.168 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 153 MB in 18s (8,507 kB/s) 2022-07-20T02:23:01.282 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:01.287 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T02:23:01.329 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T02:23:01.332 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T02:23:01.377 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T02:23:01.402 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:01.506 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:01.663 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T02:23:01.676 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T02:23:01.767 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:23:01.816 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:01.918 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:02.021 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-base. 2022-07-20T02:23:02.026 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.056 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T02:23:02.067 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T02:23:02.074 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:02.131 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T02:23:02.133 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:23:02.144 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.187 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:02.415 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T02:23:02.427 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T02:23:02.494 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rados. 2022-07-20T02:23:02.502 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.520 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:23:02.558 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:02.563 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T02:23:02.578 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.629 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:02.829 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.842 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T02:23:02.857 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:02.891 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:02.957 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rados. 2022-07-20T02:23:02.974 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:02.999 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:03.016 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.036 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T02:23:03.051 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T02:23:03.085 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:23:03.198 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T02:23:03.211 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:03.253 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.359 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T02:23:03.372 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:03.406 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.472 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T02:23:03.489 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T02:23:03.530 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T02:23:03.560 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:03.564 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T02:23:03.581 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:03.621 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.643 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T02:23:03.650 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:03.687 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.712 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T02:23:03.750 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T02:23:03.763 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:03.796 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:03.918 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T02:23:03.938 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T02:23:03.975 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T02:23:04.023 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T02:23:04.038 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:04.082 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:04.167 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T02:23:04.177 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T02:23:04.237 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T02:23:04.237 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:04.238 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:04.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:04.366 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T02:23:04.381 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:04.416 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:04.417 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T02:23:04.432 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T02:23:04.467 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T02:23:04.557 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T02:23:04.571 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T02:23:04.610 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:23:04.668 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T02:23:04.681 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T02:23:04.681 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rados. 2022-07-20T02:23:04.691 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:04.726 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:04.727 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T02:23:04.728 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:04.739 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:04.798 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:04.943 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T02:23:04.961 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T02:23:05.002 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:05.017 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T02:23:05.038 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T02:23:05.074 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:23:05.095 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T02:23:05.109 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T02:23:05.113 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T02:23:05.125 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:05.141 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:05.174 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:05.356 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T02:23:05.359 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T02:23:05.414 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T02:23:05.447 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librgw2. 2022-07-20T02:23:05.463 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:05.475 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T02:23:05.494 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:05.506 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:05.506 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T02:23:05.516 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T02:23:05.535 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:05.558 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:23:05.692 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T02:23:05.697 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T02:23:05.733 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:05.850 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T02:23:05.857 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:05.887 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:05.928 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librgw2. 2022-07-20T02:23:05.946 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:05.977 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:06.019 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-mako. 2022-07-20T02:23:06.034 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T02:23:06.058 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T02:23:06.072 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:06.080 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:23:06.113 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:06.164 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T02:23:06.182 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T02:23:06.205 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T02:23:06.426 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T02:23:06.426 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T02:23:06.439 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T02:23:06.446 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T02:23:06.466 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:06.482 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:23:06.509 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T02:23:06.516 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:06.530 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T02:23:06.543 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:06.551 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:06.572 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:06.744 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T02:23:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T02:23:06.818 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:23:06.819 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liboath0. 2022-07-20T02:23:06.846 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T02:23:06.894 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:06.903 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T02:23:06.915 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T02:23:06.959 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:06.963 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T02:23:06.977 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T02:23:07.023 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:07.089 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-webob. 2022-07-20T02:23:07.103 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T02:23:07.146 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T02:23:07.209 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T02:23:07.222 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:07.263 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:07.317 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T02:23:07.318 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liboath0. 2022-07-20T02:23:07.331 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T02:23:07.332 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T02:23:07.366 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:23:07.409 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:07.507 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T02:23:07.522 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T02:23:07.573 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:23:07.658 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-common. 2022-07-20T02:23:07.683 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:07.684 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T02:23:07.685 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:07.722 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:07.737 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:07.742 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librgw2. 2022-07-20T02:23:07.760 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:07.805 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:07.868 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T02:23:07.886 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T02:23:07.952 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T02:23:08.145 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-common. 2022-07-20T02:23:08.158 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:08.195 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:08.255 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T02:23:08.278 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:08.299 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:08.386 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T02:23:08.400 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:08.450 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:08.613 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T02:23:08.622 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T02:23:08.649 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T02:23:08.756 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T02:23:08.769 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T02:23:08.814 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:08.938 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-paste. 2022-07-20T02:23:08.950 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T02:23:08.998 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:23:09.170 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liboath0. 2022-07-20T02:23:09.182 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T02:23:09.226 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:09.305 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T02:23:09.320 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T02:23:09.358 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:23:09.564 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T02:23:09.578 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:09.620 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:09.692 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T02:23:09.707 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T02:23:09.755 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:23:09.946 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-base. 2022-07-20T02:23:09.962 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:10.015 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:10.026 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-common. 2022-07-20T02:23:10.036 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T02:23:10.040 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:10.056 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T02:23:10.074 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:10.094 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T02:23:10.308 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-base. 2022-07-20T02:23:10.320 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:10.377 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:10.405 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T02:23:10.425 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T02:23:10.455 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:23:10.835 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T02:23:10.841 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T02:23:10.885 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:23:10.936 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T02:23:10.947 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T02:23:10.992 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:23:11.247 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T02:23:11.251 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T02:23:11.263 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:11.264 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T02:23:11.296 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:11.309 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:23:11.407 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T02:23:11.418 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T02:23:11.468 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T02:23:11.671 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T02:23:11.684 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T02:23:11.707 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T02:23:11.721 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T02:23:11.722 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:11.765 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:11.792 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T02:23:11.804 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:11.841 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:11.965 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T02:23:11.978 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:12.015 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:12.103 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T02:23:12.116 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T02:23:12.146 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:12.213 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-base. 2022-07-20T02:23:12.228 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:12.281 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:12.411 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T02:23:12.427 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T02:23:12.432 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T02:23:12.443 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T02:23:12.469 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:12.492 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:23:12.786 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T02:23:12.787 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:12.792 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T02:23:12.798 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T02:23:12.805 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T02:23:12.810 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T02:23:12.837 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:12.854 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:23:12.854 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:13.160 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T02:23:13.177 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T02:23:13.187 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T02:23:13.202 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T02:23:13.208 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T02:23:13.213 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:23:13.238 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T02:23:13.239 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T02:23:13.258 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:13.537 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T02:23:13.539 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T02:23:13.549 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T02:23:13.556 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T02:23:13.556 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T02:23:13.571 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T02:23:13.577 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T02:23:13.597 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T02:23:13.598 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:13.839 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T02:23:13.852 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T02:23:13.888 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:13.891 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T02:23:13.905 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:13.942 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:14.225 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-mako. 2022-07-20T02:23:14.240 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T02:23:14.278 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:23:14.296 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T02:23:14.309 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T02:23:14.337 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:14.567 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-mako. 2022-07-20T02:23:14.582 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T02:23:14.595 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T02:23:14.607 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T02:23:14.618 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:23:14.645 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:23:14.659 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T02:23:14.672 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T02:23:14.720 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:23:14.945 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T02:23:14.947 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T02:23:14.959 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T02:23:14.960 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T02:23:14.995 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:23:15.005 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:23:15.292 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webob. 2022-07-20T02:23:15.298 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T02:23:15.305 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T02:23:15.312 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T02:23:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T02:23:15.356 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:23:15.525 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph. 2022-07-20T02:23:15.539 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:15.583 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:15.611 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T02:23:15.625 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T02:23:15.669 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:23:15.676 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webob. 2022-07-20T02:23:15.687 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T02:23:15.698 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T02:23:15.702 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T02:23:15.734 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T02:23:15.744 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:15.862 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T02:23:15.875 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:15.903 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:16.045 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T02:23:16.059 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T02:23:16.093 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T02:23:16.111 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T02:23:16.111 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:23:16.149 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T02:23:16.256 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T02:23:16.271 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T02:23:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T02:23:16.337 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:16.358 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T02:23:16.370 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T02:23:16.377 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:16.431 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T02:23:16.444 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T02:23:16.452 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T02:23:16.480 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:16.613 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T02:23:16.743 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T02:23:16.759 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:16.771 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-mako. 2022-07-20T02:23:16.784 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T02:23:16.804 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:16.821 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:23:16.886 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package cephadm. 2022-07-20T02:23:16.898 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:16.942 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:16.994 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T02:23:17.005 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:17.044 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:17.131 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T02:23:17.145 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T02:23:17.188 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T02:23:17.189 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T02:23:17.191 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T02:23:17.198 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T02:23:17.199 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T02:23:17.228 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:17.233 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:23:17.408 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T02:23:17.409 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T02:23:17.444 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T02:23:17.515 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T02:23:17.523 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-paste. 2022-07-20T02:23:17.527 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:17.538 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T02:23:17.552 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T02:23:17.567 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:17.568 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T02:23:17.583 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:23:17.602 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:23:17.747 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-paste. 2022-07-20T02:23:17.762 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T02:23:17.823 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:23:17.860 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T02:23:17.868 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T02:23:17.872 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webob. 2022-07-20T02:23:17.874 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T02:23:17.882 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T02:23:17.899 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T02:23:17.922 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T02:23:17.944 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:23:17.959 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:18.150 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T02:23:18.166 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T02:23:18.199 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:23:18.229 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T02:23:18.246 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T02:23:18.271 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T02:23:18.280 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T02:23:18.287 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:23:18.288 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T02:23:18.305 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T02:23:18.319 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:23:18.346 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:23:18.544 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T02:23:18.553 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T02:23:18.559 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T02:23:18.565 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T02:23:18.573 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T02:23:18.588 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T02:23:18.594 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T02:23:18.603 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:23:18.624 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:23:18.648 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T02:23:18.663 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T02:23:18.707 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T02:23:18.881 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T02:23:18.884 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-routes. 2022-07-20T02:23:18.899 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T02:23:18.900 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T02:23:18.928 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T02:23:18.930 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T02:23:18.942 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T02:23:18.954 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:18.992 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:19.052 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T02:23:19.066 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T02:23:19.114 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:23:19.149 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T02:23:19.162 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T02:23:19.199 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:23:19.256 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T02:23:19.271 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:19.302 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T02:23:19.303 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:19.316 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T02:23:19.360 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T02:23:19.456 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T02:23:19.470 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T02:23:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:23:19.520 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T02:23:19.538 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T02:23:19.581 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:23:19.697 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-paste. 2022-07-20T02:23:19.709 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T02:23:19.754 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:23:19.907 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T02:23:19.918 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T02:23:19.923 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:19.932 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:19.956 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:19.967 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:20.065 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T02:23:20.082 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T02:23:20.110 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:23:20.309 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T02:23:20.329 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:20.362 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:20.438 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T02:23:20.461 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T02:23:20.461 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T02:23:20.477 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:20.487 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T02:23:20.500 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:23:20.501 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:20.502 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:20.537 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:20.623 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T02:23:20.639 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T02:23:20.673 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:20.784 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T02:23:20.795 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T02:23:20.824 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T02:23:20.968 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T02:23:20.980 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T02:23:21.017 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:23:21.086 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T02:23:21.098 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T02:23:21.137 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:23:21.329 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T02:23:21.345 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T02:23:21.408 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T02:23:21.455 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T02:23:21.474 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:21.502 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T02:23:21.508 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:21.517 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T02:23:21.552 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T02:23:21.553 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:23:21.563 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:21.624 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:21.903 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T02:23:21.918 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:21.953 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:22.449 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T02:23:22.461 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:22.498 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:22.721 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T02:23:22.734 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T02:23:22.779 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:23.275 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T02:23:23.291 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T02:23:23.324 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T02:23:23.539 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T02:23:23.548 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:23.584 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:23.672 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T02:23:23.683 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T02:23:23.753 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:23.876 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph. 2022-07-20T02:23:23.890 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:23.926 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:23.962 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph. 2022-07-20T02:23:23.976 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:24.014 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:24.270 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T02:23:24.291 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T02:23:24.292 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:24.303 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:24.328 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:24.331 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:24.475 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T02:23:24.485 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:24.529 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:24.738 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T02:23:24.740 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T02:23:24.740 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:24.746 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:24.782 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:24.868 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:25.245 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephadm. 2022-07-20T02:23:25.258 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:25.292 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:25.387 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package cephadm. 2022-07-20T02:23:25.402 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:25.440 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:25.590 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T02:23:25.606 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T02:23:25.649 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:25.673 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T02:23:25.693 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T02:23:25.724 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T02:23:25.732 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T02:23:25.778 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:25.999 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T02:23:26.012 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:26.034 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T02:23:26.055 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T02:23:26.056 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:26.086 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:23:26.118 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T02:23:26.135 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:26.173 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph. 2022-07-20T02:23:26.178 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:26.187 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:26.231 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:26.371 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T02:23:26.389 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T02:23:26.401 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T02:23:26.414 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T02:23:26.420 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:23:26.451 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:26.497 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T02:23:26.510 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T02:23:26.551 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T02:23:26.554 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:26.564 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:26.602 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:26.679 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T02:23:26.692 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T02:23:26.700 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T02:23:26.716 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T02:23:26.748 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T02:23:26.782 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T02:23:26.798 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T02:23:26.833 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:23:26.874 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:23:26.980 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T02:23:26.995 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:27.038 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:27.052 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T02:23:27.066 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T02:23:27.108 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T02:23:27.119 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T02:23:27.120 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T02:23:27.151 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:23:27.190 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T02:23:27.202 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T02:23:27.263 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:23:27.429 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-routes. 2022-07-20T02:23:27.452 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T02:23:27.479 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T02:23:27.522 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T02:23:27.538 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T02:23:27.582 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:23:27.593 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-routes. 2022-07-20T02:23:27.606 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T02:23:27.611 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package cephadm. 2022-07-20T02:23:27.625 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:27.656 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T02:23:27.666 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:27.793 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T02:23:27.806 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:27.840 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:27.923 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T02:23:27.933 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T02:23:27.950 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T02:23:27.957 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T02:23:27.963 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T02:23:27.978 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:27.984 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:23:27.986 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:28.012 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:28.273 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T02:23:28.286 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:28.323 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:28.337 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T02:23:28.362 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T02:23:28.394 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:23:28.610 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T02:23:28.624 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T02:23:28.664 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T02:23:28.668 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:28.677 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T02:23:28.708 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:23:28.781 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T02:23:28.796 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:28.831 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:28.937 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T02:23:28.960 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T02:23:28.996 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:23:29.027 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T02:23:29.049 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:29.049 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T02:23:29.059 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T02:23:29.082 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:29.092 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:23:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T02:23:29.105 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T02:23:29.142 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:29.242 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T02:23:29.255 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T02:23:29.306 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:23:29.372 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T02:23:29.385 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T02:23:29.432 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:29.499 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T02:23:29.522 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T02:23:29.554 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:23:29.627 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-routes. 2022-07-20T02:23:29.633 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T02:23:29.641 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T02:23:29.647 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:29.685 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T02:23:29.689 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:29.733 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T02:23:29.748 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T02:23:29.797 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:23:29.865 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T02:23:29.892 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T02:23:29.923 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T02:23:30.009 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T02:23:30.022 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T02:23:30.039 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T02:23:30.052 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:30.099 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:30.100 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:23:30.127 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T02:23:30.141 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T02:23:30.185 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T02:23:30.653 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T02:23:30.669 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:30.707 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:30.979 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package jq. 2022-07-20T02:23:30.985 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:31.032 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:31.091 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T02:23:31.105 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:31.139 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:31.240 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T02:23:31.260 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T02:23:31.291 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:31.318 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package socat. 2022-07-20T02:23:31.339 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T02:23:31.366 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T02:23:31.493 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T02:23:31.505 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T02:23:31.513 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T02:23:31.526 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T02:23:31.534 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:31.562 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:31.811 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T02:23:31.828 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T02:23:31.845 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T02:23:31.855 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T02:23:31.862 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T02:23:31.863 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T02:23:31.881 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T02:23:31.889 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:23:31.922 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:23:32.091 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T02:23:32.107 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T02:23:32.142 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T02:23:32.157 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T02:23:32.174 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T02:23:32.204 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:23:32.258 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T02:23:32.266 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T02:23:32.272 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T02:23:32.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T02:23:32.311 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T02:23:32.331 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:32.529 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package ceph-test. 2022-07-20T02:23:32.541 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T02:23:32.543 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:32.552 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T02:23:32.592 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:32.592 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:32.972 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T02:23:32.987 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:33.031 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:33.305 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T02:23:33.319 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:33.360 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T02:23:33.580 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T02:23:33.611 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:34.142 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T02:23:34.156 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T02:23:34.179 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T02:23:34.192 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:34.194 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T02:23:34.239 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T02:23:34.456 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T02:23:34.472 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T02:23:34.489 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T02:23:34.492 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T02:23:34.540 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:34.552 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:23:34.591 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T02:23:34.607 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T02:23:34.641 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:34.846 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T02:23:34.854 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T02:23:34.866 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T02:23:34.891 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T02:23:34.892 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:23:34.927 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:23:35.211 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T02:23:35.223 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T02:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T02:23:35.247 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T02:23:35.263 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:23:35.292 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T02:23:35.367 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T02:23:35.390 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:35.450 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:35.579 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T02:23:35.597 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T02:23:35.628 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T02:23:35.637 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T02:23:35.642 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T02:23:35.687 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T02:23:35.952 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T02:23:35.965 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T02:23:36.008 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T02:23:36.023 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T02:23:36.030 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T02:23:36.057 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:23:36.166 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T02:23:36.182 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T02:23:36.219 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:36.330 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T02:23:36.342 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T02:23:36.356 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T02:23:36.365 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T02:23:36.384 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:23:36.402 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:23:36.506 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T02:23:36.523 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:36.551 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T02:23:36.552 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T02:23:36.553 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:36.590 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:36.687 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T02:23:36.687 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T02:23:36.713 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:23:36.748 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T02:23:36.766 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T02:23:36.775 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:36.777 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T02:23:36.808 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:23:36.819 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:36.898 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T02:23:36.914 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T02:23:36.960 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:23:36.995 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T02:23:37.011 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T02:23:37.066 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:23:37.117 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T02:23:37.132 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T02:23:37.176 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:23:37.216 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T02:23:37.232 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T02:23:37.295 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:37.315 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T02:23:37.327 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T02:23:37.372 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T02:23:37.382 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T02:23:37.383 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:23:37.519 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T02:23:37.530 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T02:23:37.571 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:23:37.697 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T02:23:37.713 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T02:23:37.722 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T02:23:37.734 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T02:23:37.760 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:37.771 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T02:23:37.780 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T02:23:37.793 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T02:23:37.830 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:23:38.028 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-attr. 2022-07-20T02:23:38.038 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T02:23:38.074 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T02:23:38.083 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T02:23:38.083 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T02:23:38.097 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:38.099 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T02:23:38.125 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T02:23:38.133 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:38.409 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T02:23:38.422 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:38.435 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T02:23:38.444 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T02:23:38.450 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T02:23:38.459 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:38.461 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T02:23:38.470 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T02:23:38.485 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T02:23:38.494 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:38.503 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:23:38.542 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:23:38.738 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T02:23:38.751 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T02:23:38.804 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T02:23:38.813 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T02:23:38.817 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T02:23:38.833 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T02:23:38.850 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:38.856 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T02:23:38.862 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:23:38.879 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T02:23:38.881 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:38.918 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:23:39.126 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T02:23:39.146 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package jq. 2022-07-20T02:23:39.146 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T02:23:39.166 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:39.182 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T02:23:39.183 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:23:39.199 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:39.199 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:39.231 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:39.287 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T02:23:39.301 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T02:23:39.343 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:23:39.458 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package socat. 2022-07-20T02:23:39.466 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T02:23:39.510 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package jq. 2022-07-20T02:23:39.521 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T02:23:39.521 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:39.551 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:39.621 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T02:23:39.637 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T02:23:39.679 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T02:23:39.705 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T02:23:39.719 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T02:23:39.775 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:23:39.847 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package socat. 2022-07-20T02:23:39.860 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T02:23:39.903 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T02:23:40.026 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T02:23:40.028 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T02:23:40.029 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T02:23:40.029 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T02:23:40.058 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:23:40.078 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T02:23:40.091 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T02:23:40.110 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T02:23:40.146 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:23:40.311 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T02:23:40.342 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T02:23:40.343 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T02:23:40.359 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T02:23:40.360 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:23:40.383 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:23:40.422 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T02:23:40.435 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T02:23:40.476 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T02:23:40.697 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T02:23:40.708 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T02:23:40.714 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:40.719 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T02:23:40.734 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-test. 2022-07-20T02:23:40.742 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:40.766 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:23:40.766 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:40.783 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:40.804 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T02:23:40.827 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T02:23:40.864 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:41.041 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T02:23:41.059 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T02:23:41.065 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-test. 2022-07-20T02:23:41.080 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:41.108 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:23:41.124 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:41.222 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-py. 2022-07-20T02:23:41.230 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:41.283 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:41.481 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T02:23:41.497 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:41.544 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:41.610 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T02:23:41.644 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T02:23:41.660 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:41.814 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package jq. 2022-07-20T02:23:41.837 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T02:23:41.890 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:23:41.955 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T02:23:41.973 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T02:23:42.013 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T02:23:42.134 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package socat. 2022-07-20T02:23:42.151 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T02:23:42.183 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T02:23:42.352 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T02:23:42.366 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T02:23:42.400 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:42.667 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T02:23:42.681 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T02:23:42.754 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:23:42.778 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package radosgw. 2022-07-20T02:23:42.793 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:42.874 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:43.021 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T02:23:43.042 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T02:23:43.120 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:23:43.449 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-test. 2022-07-20T02:23:43.475 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:43.539 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:43.940 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T02:23:43.953 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:44.034 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:44.127 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T02:23:44.139 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T02:23:44.168 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:44.337 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T02:23:44.448 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T02:23:44.464 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:44.504 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:44.619 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:44.631 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T02:23:44.666 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T02:23:44.686 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:44.723 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:44.759 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T02:23:44.773 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:44.819 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:44.837 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:44.963 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T02:23:44.982 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:45.016 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:45.034 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:45.195 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T02:23:45.206 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T02:23:45.228 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T02:23:45.229 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:45.256 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:45.272 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:45.627 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T02:23:45.634 INFO:teuthology.orchestra.run.smithi159.stdout:Adding system user cephadm....done 2022-07-20T02:23:45.641 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T02:23:45.683 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T02:23:45.684 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:45.700 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T02:23:45.755 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:45.820 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:45.993 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-attr. 2022-07-20T02:23:46.012 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T02:23:46.067 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T02:23:46.131 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T02:23:46.147 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:46.147 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T02:23:46.181 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:46.387 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T02:23:46.395 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T02:23:46.437 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:46.458 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:46.487 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-attr. 2022-07-20T02:23:46.502 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T02:23:46.542 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T02:23:46.590 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:46.703 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T02:23:46.713 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T02:23:46.758 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T02:23:46.870 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T02:23:46.890 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T02:23:46.931 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T02:23:46.955 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:47.020 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T02:23:47.033 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T02:23:47.095 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T02:23:47.096 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:47.113 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T02:23:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:23:47.189 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T02:23:47.203 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T02:23:47.231 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T02:23:47.303 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T02:23:47.362 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T02:23:47.408 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:47.572 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T02:23:47.591 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T02:23:47.592 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T02:23:47.592 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:47.603 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T02:23:47.635 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T02:23:47.650 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:23:47.679 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T02:23:47.679 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:47.730 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:47.887 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:47.967 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T02:23:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T02:23:48.027 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T02:23:48.120 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T02:23:48.141 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T02:23:48.143 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T02:23:48.160 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T02:23:48.186 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T02:23:48.190 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:48.212 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:48.355 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T02:23:48.366 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T02:23:48.407 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T02:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:48.498 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T02:23:48.512 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T02:23:48.547 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T02:23:48.627 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T02:23:48.634 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T02:23:48.691 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:48.740 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T02:23:48.755 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T02:23:48.755 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:48.789 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:48.852 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T02:23:48.866 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T02:23:48.905 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T02:23:48.970 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-attr. 2022-07-20T02:23:48.976 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T02:23:49.020 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T02:23:49.037 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:49.111 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-py. 2022-07-20T02:23:49.118 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:49.147 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T02:23:49.161 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:49.261 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T02:23:49.267 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T02:23:49.304 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:49.378 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T02:23:49.387 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T02:23:49.431 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:49.441 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:49.503 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T02:23:49.520 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T02:23:49.574 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:49.642 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-py. 2022-07-20T02:23:49.653 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:49.698 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:49.707 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:49.767 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T02:23:49.781 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T02:23:49.825 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T02:23:49.847 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T02:23:49.867 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T02:23:50.016 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:50.033 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T02:23:50.043 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T02:23:50.056 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T02:23:50.101 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:50.185 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T02:23:50.196 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T02:23:50.229 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:23:50.351 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T02:23:50.376 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T02:23:50.393 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:50.401 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:50.425 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T02:23:50.450 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T02:23:50.505 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T02:23:50.672 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:50.716 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T02:23:50.740 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T02:23:50.787 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package radosgw. 2022-07-20T02:23:50.803 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:50.804 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T02:23:50.842 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T02:23:50.845 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:50.855 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T02:23:50.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:51.018 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:51.097 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T02:23:51.110 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T02:23:51.186 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T02:23:51.291 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:51.329 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package radosgw. 2022-07-20T02:23:51.348 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:51.393 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:51.509 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T02:23:51.525 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T02:23:51.597 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T02:23:51.606 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T02:23:51.882 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T02:23:51.883 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T02:23:51.898 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:51.917 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T02:23:51.924 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T02:23:51.932 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:51.992 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:52.167 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:23:52.249 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T02:23:52.294 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:23:52.332 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-py. 2022-07-20T02:23:52.354 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T02:23:52.420 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:52.434 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T02:23:52.445 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:52.480 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:52.542 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:52.574 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:23:52.663 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:52.764 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T02:23:52.765 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T02:23:52.771 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T02:23:52.789 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:52.832 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:52.987 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:23:53.015 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:53.070 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T02:23:53.082 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T02:23:53.092 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:53.121 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T02:23:53.202 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:53.280 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:23:53.312 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:53.489 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T02:23:53.504 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:53.504 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T02:23:53.541 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:53.569 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user cephadm....done 2022-07-20T02:23:53.640 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:23:53.755 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:53.946 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package radosgw. 2022-07-20T02:23:53.946 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:53.976 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:54.009 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T02:23:54.094 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user cephadm....done 2022-07-20T02:23:54.125 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:54.140 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:54.251 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:54.408 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:54.423 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T02:23:54.534 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:54.613 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:54.749 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:23:54.871 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T02:23:54.904 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T02:23:54.912 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:55.030 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:55.047 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T02:23:55.060 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T02:23:55.107 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:55.173 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:23:55.306 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:23:55.332 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:55.364 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T02:23:55.427 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T02:23:55.622 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:55.625 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:23:55.743 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T02:23:55.785 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:55.883 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T02:23:55.911 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T02:23:55.998 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:56.003 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T02:23:56.084 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:56.179 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:23:56.229 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:56.269 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:56.311 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:23:56.419 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:56.581 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T02:23:56.588 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:56.712 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:56.876 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:56.882 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:23:56.914 INFO:teuthology.orchestra.run.smithi120.stdout:Adding system user cephadm....done 2022-07-20T02:23:56.996 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:23:57.185 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:23:57.194 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:57.249 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T02:23:57.275 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:57.330 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T02:23:57.529 INFO:teuthology.orchestra.run.smithi159.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T02:23:57.540 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:23:57.632 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:57.632 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T02:23:57.673 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T02:23:57.802 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:23:57.897 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:57.938 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:23:57.946 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T02:23:58.058 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T02:23:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:58.238 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:23:58.300 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:23:58.393 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T02:23:58.541 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:58.589 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:23:58.593 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:23:58.715 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:23:58.803 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T02:23:58.825 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:58.846 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T02:23:58.915 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:23:59.130 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T02:23:59.179 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:23:59.201 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:23:59.301 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:23:59.423 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T02:23:59.464 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:23:59.503 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T02:23:59.546 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:23:59.605 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T02:23:59.749 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T02:23:59.855 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T02:23:59.864 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:24:00.010 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T02:24:00.123 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T02:24:00.148 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T02:24:00.260 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:00.414 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:24:00.484 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T02:24:00.515 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:24:00.581 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T02:24:00.720 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T02:24:00.781 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T02:24:00.800 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:24:01.013 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T02:24:01.025 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:01.087 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:24:01.227 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:24:01.269 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:24:01.306 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T02:24:01.414 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:01.521 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:24:01.569 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:24:01.618 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T02:24:01.738 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T02:24:01.897 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:24:01.942 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:24:01.978 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T02:24:02.095 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T02:24:02.200 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:24:02.260 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T02:24:02.334 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T02:24:02.372 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:24:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:02.576 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:24:02.681 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T02:24:02.714 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T02:24:02.733 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T02:24:02.943 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T02:24:03.006 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T02:24:03.032 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:03.070 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:03.085 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:24:03.159 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:24:03.211 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T02:24:03.366 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T02:24:03.396 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T02:24:03.466 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:03.534 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:24:03.564 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:24:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T02:24:03.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:24:03.705 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:24:03.864 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T02:24:03.880 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:24:03.910 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T02:24:03.989 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:24:04.036 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T02:24:04.138 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:24:04.253 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T02:24:04.257 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:24:04.352 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T02:24:04.526 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:24:04.609 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:24:04.642 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:24:04.756 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T02:24:04.889 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:04.898 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T02:24:04.918 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T02:24:04.994 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:24:05.048 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T02:24:05.095 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:24:05.205 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:24:05.221 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:24:05.244 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:24:05.347 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:24:05.406 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T02:24:05.541 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:05.546 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:24:05.614 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T02:24:05.666 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T02:24:05.754 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T02:24:05.869 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:05.870 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T02:24:05.903 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:24:05.966 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T02:24:06.153 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:24:06.183 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:24:06.184 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T02:24:06.243 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:06.262 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:24:06.355 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:06.470 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T02:24:06.533 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T02:24:06.559 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:24:06.561 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:06.561 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:06.656 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T02:24:06.755 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:24:06.907 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:24:06.910 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:24:06.953 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T02:24:07.014 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:07.062 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:24:07.069 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T02:24:07.168 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:24:07.187 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T02:24:07.232 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:24:07.382 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T02:24:07.459 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T02:24:07.522 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:24:07.592 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:24:07.698 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:24:07.702 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T02:24:07.770 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:24:07.798 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:07.807 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T02:24:07.905 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T02:24:07.995 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T02:24:08.138 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:08.155 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T02:24:08.177 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:24:08.269 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:08.370 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:08.446 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T02:24:08.473 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T02:24:08.475 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:08.606 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T02:24:08.759 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T02:24:08.857 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T02:24:09.066 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T02:24:09.167 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:09.292 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:09.368 INFO:teuthology.orchestra.run.smithi120.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T02:24:09.423 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:09.532 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:09.602 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:09.631 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T02:24:09.634 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T02:24:09.704 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:24:09.794 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T02:24:09.980 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T02:24:10.091 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:10.132 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T02:24:10.216 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:10.270 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:24:10.347 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T02:24:10.513 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T02:24:10.554 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T02:24:10.612 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:10.639 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T02:24:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T02:24:10.765 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T02:24:10.903 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:11.045 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:24:11.064 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:24:11.081 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T02:24:11.184 INFO:teuthology.orchestra.run.smithi159.stdout:Adding group ceph....done 2022-07-20T02:24:11.326 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:11.346 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T02:24:11.381 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:11.445 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T02:24:11.482 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:24:11.544 INFO:teuthology.orchestra.run.smithi159.stdout:Adding system user ceph....done 2022-07-20T02:24:11.687 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:11.712 INFO:teuthology.orchestra.run.smithi159.stdout:Setting system user ceph properties....done 2022-07-20T02:24:11.728 INFO:teuthology.orchestra.run.smithi159.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T02:24:11.795 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:24:11.800 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:24:11.878 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T02:24:12.108 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:12.203 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T02:24:12.234 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:24:12.592 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:24:12.658 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:12.784 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:12.891 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:12.916 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:12.991 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:24:13.029 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:13.092 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:13.092 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:13.095 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:24:13.125 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T02:24:13.208 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:24:13.494 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T02:24:13.518 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:13.640 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:13.668 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T02:24:13.762 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:13.766 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:24:13.796 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T02:24:13.902 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:24:13.970 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T02:24:14.026 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:24:14.088 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T02:24:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T02:24:14.268 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:14.378 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:14.412 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:14.439 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:14.508 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T02:24:14.570 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T02:24:14.605 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:14.636 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:14.636 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:14.836 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T02:24:14.866 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:14.971 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T02:24:15.049 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:15.123 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:15.174 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:24:15.223 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:15.251 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:15.294 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T02:24:15.449 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:15.459 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:15.463 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:15.464 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:15.467 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T02:24:15.807 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:24:15.896 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:15.916 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.017 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:24:16.059 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.146 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.299 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.301 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.327 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T02:24:16.400 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T02:24:16.415 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:16.535 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T02:24:16.548 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:16.548 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:16.549 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T02:24:16.652 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T02:24:16.670 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:24:16.927 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T02:24:16.970 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T02:24:16.998 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.021 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.110 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T02:24:17.184 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.211 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.220 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.273 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.321 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.364 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.380 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T02:24:17.401 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T02:24:17.597 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.607 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:17.607 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:17.681 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:17.728 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:24:17.788 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T02:24:17.818 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.028 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:18.029 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T02:24:18.062 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.103 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.198 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.241 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.299 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.422 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T02:24:18.442 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.524 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.545 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T02:24:18.654 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:24:18.668 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:18.711 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:18.847 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T02:24:19.162 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T02:24:19.204 INFO:teuthology.orchestra.run.smithi026.stdout:Adding group ceph....done 2022-07-20T02:24:19.539 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:19.559 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user ceph....done 2022-07-20T02:24:19.616 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:19.700 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:19.723 INFO:teuthology.orchestra.run.smithi026.stdout:Setting system user ceph properties....done 2022-07-20T02:24:19.740 INFO:teuthology.orchestra.run.smithi026.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T02:24:19.824 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:19.893 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T02:24:19.933 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T02:24:20.180 INFO:teuthology.orchestra.run.smithi176.stdout:Adding group ceph....done 2022-07-20T02:24:20.204 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T02:24:20.336 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T02:24:20.533 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user ceph....done 2022-07-20T02:24:20.619 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:20.662 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:20.708 INFO:teuthology.orchestra.run.smithi176.stdout:Setting system user ceph properties....done 2022-07-20T02:24:20.722 INFO:teuthology.orchestra.run.smithi176.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T02:24:20.745 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:20.822 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:20.866 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T02:24:21.017 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:21.018 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:21.101 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:21.173 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T02:24:21.257 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T02:24:21.364 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:21.631 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:21.811 INFO:teuthology.orchestra.run.smithi159.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:21.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T02:24:21.895 INFO:teuthology.orchestra.run.smithi159.stdout:Reading package lists... 2022-07-20T02:24:21.955 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:22.018 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T02:24:22.056 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:22.066 INFO:teuthology.orchestra.run.smithi159.stdout:Building dependency tree... 2022-07-20T02:24:22.066 INFO:teuthology.orchestra.run.smithi159.stdout:Reading state information... 2022-07-20T02:24:22.175 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:22.220 INFO:teuthology.orchestra.run.smithi159.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T02:24:22.221 INFO:teuthology.orchestra.run.smithi159.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T02:24:22.222 INFO:teuthology.orchestra.run.smithi159.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T02:24:22.222 INFO:teuthology.orchestra.run.smithi159.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:24:22.224 INFO:teuthology.orchestra.run.smithi159.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:24:22.224 INFO:teuthology.orchestra.run.smithi159.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:24:22.225 INFO:teuthology.orchestra.run.smithi159.stdout:Suggested packages: 2022-07-20T02:24:22.225 INFO:teuthology.orchestra.run.smithi159.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T02:24:22.256 INFO:teuthology.orchestra.run.smithi159.stdout:The following NEW packages will be installed: 2022-07-20T02:24:22.257 INFO:teuthology.orchestra.run.smithi159.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T02:24:22.258 INFO:teuthology.orchestra.run.smithi159.stdout: libssl-dev 2022-07-20T02:24:22.300 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:22.301 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:22.434 INFO:teuthology.orchestra.run.smithi159.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:24:22.434 INFO:teuthology.orchestra.run.smithi159.stdout:Need to get 2,914 kB of archives. 2022-07-20T02:24:22.434 INFO:teuthology.orchestra.run.smithi159.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T02:24:22.435 INFO:teuthology.orchestra.run.smithi159.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T02:24:22.459 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:22.688 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:22.689 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:22.722 INFO:teuthology.orchestra.run.smithi120.stdout:Adding group ceph....done 2022-07-20T02:24:22.919 INFO:teuthology.orchestra.run.smithi159.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T02:24:22.950 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:23.052 INFO:teuthology.orchestra.run.smithi159.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T02:24:23.073 INFO:teuthology.orchestra.run.smithi120.stdout:Adding system user ceph....done 2022-07-20T02:24:23.074 INFO:teuthology.orchestra.run.smithi159.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T02:24:23.078 INFO:teuthology.orchestra.run.smithi159.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T02:24:23.084 INFO:teuthology.orchestra.run.smithi159.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T02:24:23.087 INFO:teuthology.orchestra.run.smithi159.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T02:24:23.092 INFO:teuthology.orchestra.run.smithi159.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T02:24:23.092 INFO:teuthology.orchestra.run.smithi159.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T02:24:23.093 INFO:teuthology.orchestra.run.smithi159.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T02:24:23.261 INFO:teuthology.orchestra.run.smithi120.stdout:Setting system user ceph properties....done 2022-07-20T02:24:23.272 INFO:teuthology.orchestra.run.smithi120.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T02:24:23.273 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:23.339 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T02:24:23.412 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T02:24:23.475 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:23.476 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:23.559 INFO:teuthology.orchestra.run.smithi159.stdout:Fetched 2,914 kB in 1s (3,128 kB/s) 2022-07-20T02:24:23.660 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package flex. 2022-07-20T02:24:23.701 INFO:teuthology.orchestra.run.smithi159.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T02:24:23.707 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:23.747 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T02:24:23.751 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T02:24:23.784 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:23.920 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:24.081 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:24.101 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:24.101 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:24.234 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package bison. 2022-07-20T02:24:24.245 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T02:24:24.295 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:24.486 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:24.542 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:24.590 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:24.591 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:24.612 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:24.738 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T02:24:24.751 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T02:24:24.770 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:24.771 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:24.820 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:24.875 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:24.875 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T02:24:25.000 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.098 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.169 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package dump. 2022-07-20T02:24:25.177 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T02:24:25.203 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.222 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T02:24:25.232 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.324 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.470 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:25.470 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:25.474 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:25.509 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package indent. 2022-07-20T02:24:25.524 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T02:24:25.592 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T02:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:25.837 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T02:24:25.859 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T02:24:25.873 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T02:24:25.887 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:25.893 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.035 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.116 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T02:24:26.129 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T02:24:26.180 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.181 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:26.247 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.301 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.321 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T02:24:26.358 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.528 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T02:24:26.533 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:26.543 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:26.543 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T02:24:26.544 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:26.584 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:26.614 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:26.763 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:26.763 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:26.866 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T02:24:26.878 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:26.920 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:26.973 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.134 INFO:teuthology.orchestra.run.smithi159.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T02:24:27.147 INFO:teuthology.orchestra.run.smithi159.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T02:24:27.191 INFO:teuthology.orchestra.run.smithi159.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:27.192 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:27.192 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T02:24:27.202 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.350 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.487 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.630 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T02:24:27.646 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.766 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T02:24:27.780 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:27.932 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:27.933 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:28.058 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:28.184 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:28.270 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:28.271 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T02:24:28.284 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:28.403 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:28.559 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T02:24:28.654 INFO:teuthology.orchestra.run.smithi159.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T02:24:28.711 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:28.720 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:28.809 INFO:teuthology.orchestra.run.smithi159.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T02:24:28.872 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:28.879 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T02:24:29.010 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:29.029 INFO:teuthology.orchestra.run.smithi159.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:29.148 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:29.236 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:29.237 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T02:24:29.291 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:29.351 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:29.654 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:29.771 INFO:teuthology.orchestra.run.smithi026.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:29.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T02:24:29.791 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:29.854 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2022-07-20T02:24:29.913 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T02:24:30.038 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2022-07-20T02:24:30.038 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2022-07-20T02:24:30.089 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T02:24:30.203 INFO:teuthology.orchestra.run.smithi026.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T02:24:30.203 INFO:teuthology.orchestra.run.smithi026.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T02:24:30.204 INFO:teuthology.orchestra.run.smithi026.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T02:24:30.204 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:24:30.204 INFO:teuthology.orchestra.run.smithi026.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:24:30.204 INFO:teuthology.orchestra.run.smithi026.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:24:30.204 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2022-07-20T02:24:30.205 INFO:teuthology.orchestra.run.smithi026.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T02:24:30.235 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2022-07-20T02:24:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T02:24:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: libssl-dev 2022-07-20T02:24:30.390 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:30.419 INFO:teuthology.orchestra.run.smithi026.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:24:30.419 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 2,914 kB of archives. 2022-07-20T02:24:30.419 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T02:24:30.420 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T02:24:30.608 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:30.897 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T02:24:31.026 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:31.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T02:24:31.038 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T02:24:31.042 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T02:24:31.061 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T02:24:31.068 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T02:24:31.069 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T02:24:31.074 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T02:24:31.074 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T02:24:31.075 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T02:24:31.101 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-07-20T02:24:31.110 INFO:teuthology.orchestra.run.smithi159.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T02:24:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-07-20T02:24:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-07-20T02:24:31.436 INFO:teuthology.orchestra.run.smithi176.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T02:24:31.436 INFO:teuthology.orchestra.run.smithi176.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T02:24:31.437 INFO:teuthology.orchestra.run.smithi176.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T02:24:31.437 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:24:31.438 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:24:31.438 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:24:31.438 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2022-07-20T02:24:31.439 INFO:teuthology.orchestra.run.smithi176.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T02:24:31.471 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-07-20T02:24:31.472 INFO:teuthology.orchestra.run.smithi176.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T02:24:31.473 INFO:teuthology.orchestra.run.smithi176.stdout: libssl-dev 2022-07-20T02:24:31.529 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:24:31.529 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 2,914 kB of archives. 2022-07-20T02:24:31.529 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T02:24:31.530 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T02:24:31.543 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 2,914 kB in 1s (3,128 kB/s) 2022-07-20T02:24:31.633 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T02:24:31.663 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T02:24:31.664 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T02:24:31.668 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T02:24:31.669 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T02:24:31.670 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T02:24:31.671 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T02:24:31.671 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T02:24:31.672 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T02:24:31.680 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package flex. 2022-07-20T02:24:31.728 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T02:24:31.731 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:31.797 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T02:24:31.819 INFO:teuthology.orchestra.run.smithi159.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:31.822 DEBUG:teuthology.parallel:result is None 2022-07-20T02:24:32.043 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 2,914 kB in 0s (13.8 MB/s) 2022-07-20T02:24:32.189 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package flex. 2022-07-20T02:24:32.233 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T02:24:32.236 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:32.281 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T02:24:32.320 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package bison. 2022-07-20T02:24:32.334 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T02:24:32.375 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:32.766 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package bison. 2022-07-20T02:24:32.782 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T02:24:32.806 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T02:24:32.819 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T02:24:32.821 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:32.898 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:33.012 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:33.227 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package dump. 2022-07-20T02:24:33.235 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T02:24:33.263 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T02:24:33.276 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T02:24:33.278 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T02:24:33.366 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:33.475 INFO:teuthology.orchestra.run.smithi120.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:33.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T02:24:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-07-20T02:24:33.600 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package indent. 2022-07-20T02:24:33.612 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T02:24:33.654 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T02:24:33.731 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-07-20T02:24:33.731 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-07-20T02:24:33.732 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package dump. 2022-07-20T02:24:33.748 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T02:24:33.790 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T02:24:33.888 INFO:teuthology.orchestra.run.smithi120.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T02:24:33.889 INFO:teuthology.orchestra.run.smithi120.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T02:24:33.889 INFO:teuthology.orchestra.run.smithi120.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T02:24:33.891 INFO:teuthology.orchestra.run.smithi120.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T02:24:33.891 INFO:teuthology.orchestra.run.smithi120.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T02:24:33.891 INFO:teuthology.orchestra.run.smithi120.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T02:24:33.892 INFO:teuthology.orchestra.run.smithi120.stdout:Suggested packages: 2022-07-20T02:24:33.893 INFO:teuthology.orchestra.run.smithi120.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T02:24:33.921 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-07-20T02:24:33.922 INFO:teuthology.orchestra.run.smithi120.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T02:24:33.922 INFO:teuthology.orchestra.run.smithi120.stdout: libssl-dev 2022-07-20T02:24:33.924 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T02:24:33.938 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T02:24:33.978 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:24:33.979 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 2,914 kB of archives. 2022-07-20T02:24:33.979 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T02:24:33.979 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T02:24:33.982 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:34.058 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package indent. 2022-07-20T02:24:34.067 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T02:24:34.090 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T02:24:34.109 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T02:24:34.110 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T02:24:34.113 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T02:24:34.115 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T02:24:34.115 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T02:24:34.117 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T02:24:34.118 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T02:24:34.119 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T02:24:34.138 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T02:24:34.223 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T02:24:34.240 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T02:24:34.268 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:34.480 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T02:24:34.487 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 2,914 kB in 0s (14.0 MB/s) 2022-07-20T02:24:34.494 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T02:24:34.538 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:34.632 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package flex. 2022-07-20T02:24:34.638 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T02:24:34.651 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:34.677 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T02:24:34.681 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:34.689 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:34.726 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T02:24:34.814 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T02:24:34.824 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T02:24:34.867 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:34.950 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T02:24:34.958 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:34.999 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:35.236 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T02:24:35.242 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package bison. 2022-07-20T02:24:35.249 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:35.270 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T02:24:35.271 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T02:24:35.291 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:35.292 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T02:24:35.307 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:35.320 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:35.538 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T02:24:35.552 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:35.596 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:35.753 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T02:24:35.764 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T02:24:35.768 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T02:24:35.853 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:35.882 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T02:24:35.890 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:35.898 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T02:24:35.956 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:36.008 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:36.142 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:36.175 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package dump. 2022-07-20T02:24:36.188 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T02:24:36.216 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T02:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:36.377 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:36.422 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T02:24:36.503 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T02:24:36.536 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package indent. 2022-07-20T02:24:36.545 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:36.550 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T02:24:36.586 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T02:24:36.602 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T02:24:36.663 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:36.696 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:36.761 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T02:24:36.822 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:36.823 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T02:24:36.897 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T02:24:36.911 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T02:24:36.950 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:36.951 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:36.974 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:37.074 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:37.085 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:37.208 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T02:24:37.225 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T02:24:37.239 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T02:24:37.283 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:37.295 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T02:24:37.311 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:37.351 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:37.442 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T02:24:37.512 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T02:24:37.642 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:37.670 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T02:24:37.683 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:37.730 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:37.789 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:37.942 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:38.025 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T02:24:38.028 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T02:24:38.072 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T02:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T02:24:38.350 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:38.965 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T02:24:39.063 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T02:24:39.207 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T02:24:39.214 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T02:24:39.350 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T02:24:39.466 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T02:24:39.599 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:39.785 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T02:24:39.796 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T02:24:39.849 INFO:teuthology.orchestra.run.smithi120.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T02:24:39.911 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T02:24:39.922 INFO:teuthology.orchestra.run.smithi026.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:39.923 DEBUG:teuthology.parallel:result is None 2022-07-20T02:24:40.009 INFO:teuthology.orchestra.run.smithi120.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T02:24:40.106 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T02:24:40.247 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T02:24:40.340 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T02:24:40.500 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T02:24:40.511 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:40.513 DEBUG:teuthology.parallel:result is None 2022-07-20T02:24:42.347 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T02:24:43.089 INFO:teuthology.orchestra.run.smithi120.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T02:24:43.093 DEBUG:teuthology.parallel:result is None 2022-07-20T02:24:43.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2022-07-20T02:24:43.147 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2022-07-20T02:24:43.152 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:24:43.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:24:43.339 DEBUG:teuthology.orchestra.run.smithi026:> dpkg-query -W -f '${Version}' ceph 2022-07-20T02:24:43.371 INFO:teuthology.orchestra.run.smithi026.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.374 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.375 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T02:24:43.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-07-20T02:24:43.430 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-07-20T02:24:43.436 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:24:43.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:24:43.632 DEBUG:teuthology.orchestra.run.smithi120:> dpkg-query -W -f '${Version}' ceph 2022-07-20T02:24:43.658 INFO:teuthology.orchestra.run.smithi120.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.659 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.659 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T02:24:43.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo lsb_release -is 2022-07-20T02:24:43.712 INFO:teuthology.orchestra.run.smithi159.stdout:Ubuntu 2022-07-20T02:24:43.720 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:24:43.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:24:43.911 DEBUG:teuthology.orchestra.run.smithi159:> dpkg-query -W -f '${Version}' ceph 2022-07-20T02:24:43.941 INFO:teuthology.orchestra.run.smithi159.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.942 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T02:24:43.942 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T02:24:43.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-07-20T02:24:43.997 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-07-20T02:24:44.001 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T02:24:44.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:24:44.192 DEBUG:teuthology.orchestra.run.smithi176:> dpkg-query -W -f '${Version}' ceph 2022-07-20T02:24:44.216 INFO:teuthology.orchestra.run.smithi176.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T02:24:44.216 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T02:24:44.216 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T02:24:44.217 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-20T02:24:44.218 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:44.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T02:24:44.260 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:44.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T02:24:44.289 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:44.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T02:24:44.315 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:44.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T02:24:44.333 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-20T02:24:44.334 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:44.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T02:24:44.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T02:24:44.416 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:44.417 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T02:24:44.443 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T02:24:44.473 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:44.474 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T02:24:44.491 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T02:24:44.551 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:44.552 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T02:24:44.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T02:24:44.586 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-20T02:24:44.587 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:44.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T02:24:44.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T02:24:44.708 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:44.709 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T02:24:44.751 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T02:24:44.776 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:44.777 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T02:24:44.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T02:24:44.882 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:44.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T02:24:44.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T02:24:44.923 INFO:teuthology.run_tasks:Running task ceph... 2022-07-20T02:24:45.037 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-07-20T02:24:45.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2022-07-20T02:24:45.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2022-07-20T02:24:45.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2022-07-20T02:24:45.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2022-07-20T02:24:45.139 INFO:tasks.ceph:Disabling ceph logrotate... 2022-07-20T02:24:45.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T02:24:45.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T02:24:45.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T02:24:45.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T02:24:45.207 INFO:tasks.ceph:Creating extra log directories... 2022-07-20T02:24:45.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T02:24:45.230 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T02:24:45.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T02:24:45.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T02:24:45.339 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-07-20T02:24:45.339 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-07-20T02:24:45.350 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-07-20T02:24:45.351 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:45.351 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T02:24:45.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T02:24:45.452 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:45.453 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T02:24:45.462 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T02:24:45.504 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:45.505 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T02:24:45.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T02:24:45.612 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:45.613 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T02:24:45.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T02:24:45.723 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-07-20T02:24:45.723 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-20T02:24:45.724 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288', 'branch': 'octopus-release', 'description': 'fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6939288', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'octopus-release'}, 'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'octopus-release', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'], ['client.0'], ['client.1']], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'octopus-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'targets': {'smithi026.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDftE6x4eFChcx3puqfvcQI+jLl8ROUdei5ughSJP9Ajd6ImomCxHmgMwAvEfZUMgJvisVUNyHdf4NNwKFJ8aQCYq8qi+EvuD4doyghkTg0x3S116v9UtB9vv7fNKxB12pe5Uq3GUaiXx/kib2npnL8/qYuwT31oh9fTPXKjVZxx0Pmf2H/9y5RbZlkmEMeTVzr8Lyg2jHpTaugHHzkwVjtYAdbY4RJzF+gZ52LLX2SVnuVsScKK6gUGEyrLbrtfPyrQwfi76X2AsvpjFvILEqBIu0+PBeYp5rpwXQLg3FBUEwWfRVABM8yCOezU9vIP5ROgGXSS6O0H/fcqsYkTD3HYL0IckEQTq1VbB5OA9yfM5gpcKVL8y4XjxMvkMpYHI4COO4SLEgNERWGa5p+kE21mokDR0E8zpdjaFG23spgFw4uZ+g4kq1LL07jBCpCkZWEY2XdpHv5o/gLW6ppqnCH7NyG/bt3d3I74XCXZ4FTtk2nG/QBiFuiiuFI4F7XY10=', 'smithi120.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNinRuNL786cCN5o7rl2CdnbHLqNOlyxX4GgQ5s/xhMCBIiu3wr6m9XqkNDoOWXErxcs4CN3S4/QEemNwgaZTRSPPi0HhSQQDCbmoSgdTxEnE6DHbjks9NcI/uB58u9Pt487cLNATS1xotJUptd/9fZtc1wZiTTZwIEp0YXtq6lp11vNm9urGv1V/8EPwPBd3wv4sL1l62OgZwXFbXD8YZ17aqkFm4dzESvk4HMhfIiqUQJ3bn+Nd/WkPMnwwUUYjP05kHdINJ1NAIyGtCa1Bn4mEd01rM4PH1pYyeNQLg3o6hSuj7CKzm/YcbM510V9/e+DJ6i1MUtG6R8pHi6KPIRmGueaO15sIsgyuutpyu5oqvS83a4cCs2X6CZarSOujCn2WucGNHGNbWTjl1Cxb6Zq0PxSNZJLHjwCcbxubmhcfS0HOjDG2mrNCPh8HLy/PCmBfhZNyKxiohFZ2knQgTZAywEVECYWh4T1XrAX6/PVxMEFiUXPMJV64KRV+tP30=', 'smithi159.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjZ1hzUsMAkKgInBZwQ4m0nyjKcvPVfXzgGvV63vSDF9oLXPe6g9janjl4noX6heXyWHEQ7VIVIMG6jXJUaI7aXqlcu6WGhQAGwkt6HLzXpOAU90bMnlA8be2v+F2pWys/Yddr+RdeUFUAcZxAtH5ZDn9XrVLygy5deztBHZ+Ky0tgy/3AjHYZOQK9VIQMayS4v8UkpzZ5hZYO+PXRlCekpDGfAv5+jQn0GZbrLHpJ+RNuUdVSykuFyFEvw3jRi1qBPxeRYI4wpb9Mggr9nBsOSVf0J6XDRXZ3/AgNHiN43gGS3Yz0e+/QdlQbZvvXW9oFxk17WsUwrwbY1SrJ9hWhFbMQZr0HiFIij3xPD5VtRN88rToqsSPEQNR5uBHddO9/omvc60YocZbCmJpAu5MXdiAmzQVrztHytKMZN3y/pQF3LtVgPG3613RKt3KN3m6HmksXmV8eEARfgglsGsASclIsPZzfln3Q4CuJcWSalNzxb1AYjrwYKAVykHTU058=', 'smithi176.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f9jQtixYdFaa5M+hOPFWolp3Aur96UU75qBTq5FyusQFo5Y6U7inZ09QRFmApcUw+9d7V02teTesIoqEB8QP5ffm6ylE9eTEPDyuH+CqnO/8NhZOl+Wezl//zKW8EXbKN1ogN3GtXEj+twBZG1PV3mzMsY2FmxJ4jIubqaXJRkDcx7sVIeDGJ8ry6e30mZtNnT1An6+ubz7Hx32ZZ1YX9edblbcPMrupracArUUFKbuA84QMCNC1oR3iAO2sskjoukErIqT9xoLkqzvAi0yfopCrAdCup3iz+ETnKiJEACDYdUCzZDPeR9DdeyvrJzmBJJS989PV3nsyxGrhg/qEELoVReDt8od+82X2SEe0oVkiZ89wisLf+iin01uM8doeO+8cbZkdzu3zfJLsMuAgIYylsqrP8cCpNf8ullscgVCAm1Lm85Zecr/Vi4/FQ5z+QQMNNq1aB9ws39eHoDGRGAHtjW7qe0BL8plYb9pbrUBFSv5qp63HDVyokxF7aQk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_misc']}}], 'teuthology_branch': 'main', 'teuthology_sha1': '89d874d555f56b1c1f383df72487aecec0ee6cf9', 'timestamp': '2022-07-19_23:28:53', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-20T02:24:45.728 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T02:24:45.737 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T02:24:45.753 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T02:24:45.766 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T02:24:45.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T02:24:45.803 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T02:24:45.810 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T02:24:45.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T02:24:45.847 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:45.848 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2022-07-20T02:24:45.887 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-20T02:24:45.888 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-20T02:24:45.899 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-20T02:24:45.900 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:45.900 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-07-20T02:24:45.901 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:45.902 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2022-07-20 02:21:28.599934259 +0000 2022-07-20T02:24:45.902 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2022-07-20 02:21:28.152023338 +0000 2022-07-20T02:24:45.903 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2022-07-20 02:21:28.152023338 +0000 2022-07-20T02:24:45.903 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2022-07-20T02:24:45.908 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2022-07-20T02:24:45.909 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2022-07-20T02:24:45.909 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000300204 s, 1.7 MB/s 2022-07-20T02:24:45.914 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-20T02:24:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-20T02:24:45.964 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:45.965 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 366 Links: 1 2022-07-20T02:24:45.965 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:45.965 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2022-07-20 02:21:29.335789570 +0000 2022-07-20T02:24:45.966 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2022-07-20 02:21:28.803893696 +0000 2022-07-20T02:24:45.967 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2022-07-20 02:21:28.803893696 +0000 2022-07-20T02:24:45.967 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2022-07-20T02:24:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2022-07-20T02:24:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2022-07-20T02:24:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000293147 s, 1.7 MB/s 2022-07-20T02:24:45.978 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-20T02:24:45.998 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-20T02:24:45.999 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:45.999 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 372 Links: 1 2022-07-20T02:24:45.999 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:45.999 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2022-07-20 02:21:30.139634401 +0000 2022-07-20T02:24:45.999 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2022-07-20 02:21:29.547748477 +0000 2022-07-20T02:24:46.000 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2022-07-20 02:21:29.547748477 +0000 2022-07-20T02:24:46.000 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2022-07-20T02:24:46.003 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2022-07-20T02:24:46.003 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2022-07-20T02:24:46.003 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000293884 s, 1.7 MB/s 2022-07-20T02:24:46.010 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-20T02:24:46.065 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-20T02:24:46.066 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:46.066 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 380 Links: 1 2022-07-20T02:24:46.066 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:46.067 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2022-07-20 02:21:30.863497683 +0000 2022-07-20T02:24:46.067 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2022-07-20 02:21:30.339596634 +0000 2022-07-20T02:24:46.067 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2022-07-20 02:21:30.339596634 +0000 2022-07-20T02:24:46.068 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2022-07-20T02:24:46.075 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2022-07-20T02:24:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2022-07-20T02:24:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00029187 s, 1.8 MB/s 2022-07-20T02:24:46.088 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:46.089 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:46.089 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2022-07-20T02:24:46.099 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-20T02:24:46.100 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-20T02:24:46.109 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-20T02:24:46.109 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:46.110 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-07-20T02:24:46.110 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:46.111 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-07-20 02:21:32.729180145 +0000 2022-07-20T02:24:46.111 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-07-20 02:21:32.217180144 +0000 2022-07-20T02:24:46.112 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-07-20 02:21:32.217180144 +0000 2022-07-20T02:24:46.113 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-07-20T02:24:46.125 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-07-20T02:24:46.126 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-07-20T02:24:46.126 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000410044 s, 1.2 MB/s 2022-07-20T02:24:46.130 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-20T02:24:46.183 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-20T02:24:46.183 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:46.183 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 368 Links: 1 2022-07-20T02:24:46.184 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:46.184 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-07-20 02:21:32.965180145 +0000 2022-07-20T02:24:46.184 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-07-20 02:21:32.961180145 +0000 2022-07-20T02:24:46.184 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-07-20 02:21:32.961180145 +0000 2022-07-20T02:24:46.185 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-07-20T02:24:46.193 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-07-20T02:24:46.194 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-07-20T02:24:46.194 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000401176 s, 1.3 MB/s 2022-07-20T02:24:46.198 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-20T02:24:46.217 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-20T02:24:46.218 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:46.218 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 374 Links: 1 2022-07-20T02:24:46.218 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:46.219 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-07-20 02:21:34.189180145 +0000 2022-07-20T02:24:46.219 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-07-20 02:21:33.657180145 +0000 2022-07-20T02:24:46.219 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-07-20 02:21:33.657180145 +0000 2022-07-20T02:24:46.219 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-07-20T02:24:46.221 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-07-20T02:24:46.221 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-07-20T02:24:46.221 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000384784 s, 1.3 MB/s 2022-07-20T02:24:46.222 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-20T02:24:46.231 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-20T02:24:46.231 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T02:24:46.232 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 380 Links: 1 2022-07-20T02:24:46.232 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T02:24:46.232 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-07-20 02:21:34.833180146 +0000 2022-07-20T02:24:46.232 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-07-20 02:21:34.437180146 +0000 2022-07-20T02:24:46.232 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-07-20 02:21:34.437180146 +0000 2022-07-20T02:24:46.233 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-07-20T02:24:46.237 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-07-20T02:24:46.237 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-07-20T02:24:46.238 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000403913 s, 1.3 MB/s 2022-07-20T02:24:46.244 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:46.251 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-20T02:24:46.252 INFO:tasks.ceph:Generating config... 2022-07-20T02:24:46.256 INFO:tasks.ceph:[client] client mount timeout = 600 2022-07-20T02:24:46.256 INFO:tasks.ceph:[client] debug client = 20 2022-07-20T02:24:46.257 INFO:tasks.ceph:[client] debug ms = 1 2022-07-20T02:24:46.257 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2022-07-20T02:24:46.258 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2022-07-20T02:24:46.259 INFO:tasks.ceph:[mds] debug mds = 20 2022-07-20T02:24:46.259 INFO:tasks.ceph:[mds] debug ms = 1 2022-07-20T02:24:46.260 INFO:tasks.ceph:[mds] mds bal frag = True 2022-07-20T02:24:46.260 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-07-20T02:24:46.261 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-07-20T02:24:46.261 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-07-20T02:24:46.263 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-07-20T02:24:46.263 INFO:tasks.ceph:[mds] mds debug frag = True 2022-07-20T02:24:46.264 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-07-20T02:24:46.264 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-07-20T02:24:46.265 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-07-20T02:24:46.266 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-07-20T02:24:46.266 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2022-07-20T02:24:46.267 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2022-07-20T02:24:46.267 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-20T02:24:46.268 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-20T02:24:46.269 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-20T02:24:46.269 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-20T02:24:46.281 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-20T02:24:46.281 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-07-20T02:24:46.282 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-07-20T02:24:46.282 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2022-07-20T02:24:46.282 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-07-20T02:24:46.283 INFO:tasks.ceph:[osd] debug bluefs = 20 2022-07-20T02:24:46.283 INFO:tasks.ceph:[osd] debug bluestore = 20 2022-07-20T02:24:46.284 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-20T02:24:46.284 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-20T02:24:46.284 INFO:tasks.ceph:[osd] debug rocksdb = 10 2022-07-20T02:24:46.285 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-07-20T02:24:46.285 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-07-20T02:24:46.286 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-07-20T02:24:46.286 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-07-20T02:24:46.286 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-07-20T02:24:46.287 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-07-20T02:24:46.287 INFO:tasks.ceph:Setting up mon.a... 2022-07-20T02:24:46.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-07-20T02:24:46.337 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2022-07-20T02:24:46.339 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-07-20T02:24:46.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T02:24:46.440 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.26'), ('mon.b', '172.21.15.120'), ('mon.c', '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]')] 2022-07-20T02:24:46.441 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.26 --add b 172.21.15.120 --addv c '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:46.480 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:46.481 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.481 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2022-07-20T02:24:46.482 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.482 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2022-07-20T02:24:46.477995+0000 2022-07-20T02:24:46.482 INFO:teuthology.orchestra.run.smithi026.stdout:created 2022-07-20T02:24:46.477995+0000 2022-07-20T02:24:46.483 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 0 (unknown) 2022-07-20T02:24:46.483 INFO:teuthology.orchestra.run.smithi026.stdout:0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2022-07-20T02:24:46.484 INFO:teuthology.orchestra.run.smithi026.stdout:1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2022-07-20T02:24:46.484 INFO:teuthology.orchestra.run.smithi026.stdout:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2022-07-20T02:24:46.485 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-07-20T02:24:46.486 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0... 2022-07-20T02:24:46.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:24:46.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:24:46.528 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:24:46.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:24:46.559 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2022-07-20T02:24:46.560 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2022-07-20T02:24:46.561 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:24:46.561 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2022-07-20T02:24:46.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.562 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2022-07-20T02:24:46.563 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.563 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2022-07-20T02:24:46.564 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.564 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2022-07-20T02:24:46.564 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2022-07-20T02:24:46.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.565 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2022-07-20T02:24:46.566 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2022-07-20T02:24:46.566 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.567 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:24:46.570 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2022-07-20T02:24:46.570 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.570 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:24:46.571 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:24:46.571 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.571 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2022-07-20T02:24:46.572 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.572 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2022-07-20T02:24:46.572 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2022-07-20T02:24:46.572 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:24:46.572 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:24:46.573 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2022-07-20T02:24:46.573 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:24:46.573 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2022-07-20T02:24:46.573 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:24:46.574 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:24:46.574 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.574 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:24:46.574 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.575 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2022-07-20T02:24:46.575 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.575 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2022-07-20T02:24:46.575 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.576 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2022-07-20T02:24:46.576 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2022-07-20T02:24:46.576 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2022-07-20T02:24:46.576 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:24:46.577 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.577 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.577 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2022-07-20T02:24:46.578 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2022-07-20T02:24:46.578 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.578 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2022-07-20T02:24:46.578 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2022-07-20T02:24:46.579 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.579 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2022-07-20T02:24:46.579 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2022-07-20T02:24:46.579 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.580 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2022-07-20T02:24:46.580 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2022-07-20T02:24:46.580 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2022-07-20T02:24:46.580 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.580 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2022-07-20T02:24:46.581 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2022-07-20T02:24:46.581 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.581 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:24:46.581 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.582 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2022-07-20T02:24:46.582 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.582 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2022-07-20T02:24:46.582 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.582 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:24:46.583 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2022-07-20T02:24:46.583 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.584 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2022-07-20T02:24:46.584 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2022-07-20T02:24:46.587 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2022-07-20T02:24:46.587 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore compression mode = aggressive 2022-07-20T02:24:46.588 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2022-07-20T02:24:46.588 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 20 2022-07-20T02:24:46.588 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 20 2022-07-20T02:24:46.588 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:24:46.589 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2022-07-20T02:24:46.589 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2022-07-20T02:24:46.589 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:24:46.589 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2022-07-20T02:24:46.592 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:24:46.593 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:24:46.593 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2022-07-20T02:24:46.593 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:24:46.593 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.594 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2022-07-20T02:24:46.594 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:24:46.594 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2022-07-20T02:24:46.594 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:24:46.594 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:24:46.595 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.595 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.595 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2022-07-20T02:24:46.595 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.596 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2022-07-20T02:24:46.596 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:24:46.596 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:24:46.596 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2022-07-20T02:24:46.596 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:24:46.597 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2022-07-20T02:24:46.597 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2022-07-20T02:24:46.597 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.597 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2022-07-20T02:24:46.598 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2022-07-20T02:24:46.598 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.598 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.598 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:24:46.598 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:24:46.599 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:24:46.599 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.599 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:24:46.599 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:24:46.600 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:24:46.600 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2022-07-20T02:24:46.600 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:24:46.601 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2022-07-20T02:24:46.601 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2022-07-20T02:24:46.601 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2022-07-20T02:24:46.601 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2022-07-20T02:24:46.601 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:24:46.602 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:24:46.602 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2022-07-20T02:24:46.602 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2022-07-20T02:24:46.602 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:24:46.603 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.603 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.603 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2022-07-20T02:24:46.607 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2022-07-20T02:24:46.607 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2022-07-20T02:24:46.608 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2022-07-20T02:24:46.608 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2022-07-20T02:24:46.608 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2022-07-20T02:24:46.608 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2022-07-20T02:24:46.609 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2022-07-20T02:24:46.609 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:24:46.609 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal frag = True 2022-07-20T02:24:46.609 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal fragment size max = 10000 2022-07-20T02:24:46.610 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal merge size = 5 2022-07-20T02:24:46.610 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split bits = 3 2022-07-20T02:24:46.610 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split size = 100 2022-07-20T02:24:46.610 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2022-07-20T02:24:46.611 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2022-07-20T02:24:46.611 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2022-07-20T02:24:46.611 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2022-07-20T02:24:46.611 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:24:46.611 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.612 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.615 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2022-07-20T02:24:46.616 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2022-07-20T02:24:46.616 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:24:46.616 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2022-07-20T02:24:46.616 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.617 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2022-07-20T02:24:46.617 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.617 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2022-07-20T02:24:46.617 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.618 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2022-07-20T02:24:46.618 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2022-07-20T02:24:46.619 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.619 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2022-07-20T02:24:46.619 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2022-07-20T02:24:46.620 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.620 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:24:46.620 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2022-07-20T02:24:46.620 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.621 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:24:46.621 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:24:46.621 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.621 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2022-07-20T02:24:46.622 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.622 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2022-07-20T02:24:46.622 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2022-07-20T02:24:46.623 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:24:46.623 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:24:46.623 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2022-07-20T02:24:46.623 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:24:46.624 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2022-07-20T02:24:46.624 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:24:46.624 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:24:46.624 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.624 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:24:46.625 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.625 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2022-07-20T02:24:46.625 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.625 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2022-07-20T02:24:46.626 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.626 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2022-07-20T02:24:46.626 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2022-07-20T02:24:46.626 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2022-07-20T02:24:46.627 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:24:46.627 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.627 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.627 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2022-07-20T02:24:46.627 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2022-07-20T02:24:46.628 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.628 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2022-07-20T02:24:46.629 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2022-07-20T02:24:46.629 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.629 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2022-07-20T02:24:46.629 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2022-07-20T02:24:46.630 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.630 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2022-07-20T02:24:46.630 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2022-07-20T02:24:46.630 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2022-07-20T02:24:46.631 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.631 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2022-07-20T02:24:46.631 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2022-07-20T02:24:46.631 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.632 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:24:46.632 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.632 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2022-07-20T02:24:46.632 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.633 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2022-07-20T02:24:46.633 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.633 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:24:46.633 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2022-07-20T02:24:46.634 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.634 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2022-07-20T02:24:46.634 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2022-07-20T02:24:46.634 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2022-07-20T02:24:46.634 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2022-07-20T02:24:46.635 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2022-07-20T02:24:46.636 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2022-07-20T02:24:46.636 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2022-07-20T02:24:46.636 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:24:46.636 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2022-07-20T02:24:46.637 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2022-07-20T02:24:46.637 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:24:46.637 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2022-07-20T02:24:46.637 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:24:46.638 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:24:46.638 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2022-07-20T02:24:46.638 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:24:46.638 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.639 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2022-07-20T02:24:46.639 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:24:46.639 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2022-07-20T02:24:46.639 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:24:46.639 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:24:46.640 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.640 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.640 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2022-07-20T02:24:46.640 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.641 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2022-07-20T02:24:46.641 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:24:46.641 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:24:46.641 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2022-07-20T02:24:46.642 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:24:46.642 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2022-07-20T02:24:46.642 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2022-07-20T02:24:46.642 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.642 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2022-07-20T02:24:46.643 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2022-07-20T02:24:46.643 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.643 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.643 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:24:46.644 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:24:46.644 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:24:46.644 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.644 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:24:46.644 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:24:46.645 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:24:46.645 INFO:teuthology.orchestra.run.smithi176.stdout: mon op complaint time = 120 2022-07-20T02:24:46.645 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:24:46.645 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2022-07-20T02:24:46.646 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2022-07-20T02:24:46.646 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2022-07-20T02:24:46.646 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2022-07-20T02:24:46.646 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:24:46.647 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:24:46.647 INFO:teuthology.orchestra.run.smithi176.stdout: client mount timeout = 600 2022-07-20T02:24:46.647 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2022-07-20T02:24:46.647 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:24:46.648 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.648 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.648 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2022-07-20T02:24:46.648 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2022-07-20T02:24:46.648 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2022-07-20T02:24:46.649 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2022-07-20T02:24:46.649 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2022-07-20T02:24:46.649 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2022-07-20T02:24:46.649 INFO:teuthology.orchestra.run.smithi176.stdout:[mds] 2022-07-20T02:24:46.649 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds = 20 2022-07-20T02:24:46.650 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:24:46.650 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal frag = True 2022-07-20T02:24:46.650 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal fragment size max = 10000 2022-07-20T02:24:46.650 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal merge size = 5 2022-07-20T02:24:46.651 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split bits = 3 2022-07-20T02:24:46.651 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split size = 100 2022-07-20T02:24:46.651 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug frag = True 2022-07-20T02:24:46.651 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug scatterstat = True 2022-07-20T02:24:46.652 INFO:teuthology.orchestra.run.smithi176.stdout: mds op complaint time = 180 2022-07-20T02:24:46.652 INFO:teuthology.orchestra.run.smithi176.stdout: mds verify scatter = True 2022-07-20T02:24:46.652 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:24:46.652 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.653 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.653 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2022-07-20T02:24:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2022-07-20T02:24:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:24:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2022-07-20T02:24:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2022-07-20T02:24:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2022-07-20T02:24:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2022-07-20T02:24:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2022-07-20T02:24:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2022-07-20T02:24:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2022-07-20T02:24:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:24:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2022-07-20T02:24:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:24:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:24:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2022-07-20T02:24:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2022-07-20T02:24:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2022-07-20T02:24:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:24:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:24:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2022-07-20T02:24:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:24:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2022-07-20T02:24:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:24:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:24:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:24:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2022-07-20T02:24:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2022-07-20T02:24:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2022-07-20T02:24:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2022-07-20T02:24:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2022-07-20T02:24:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:24:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.663 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2022-07-20T02:24:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2022-07-20T02:24:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2022-07-20T02:24:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2022-07-20T02:24:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2022-07-20T02:24:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2022-07-20T02:24:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2022-07-20T02:24:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2022-07-20T02:24:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2022-07-20T02:24:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2022-07-20T02:24:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2022-07-20T02:24:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:24:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2022-07-20T02:24:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2022-07-20T02:24:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:24:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2022-07-20T02:24:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2022-07-20T02:24:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2022-07-20T02:24:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2022-07-20T02:24:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression mode = aggressive 2022-07-20T02:24:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2022-07-20T02:24:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2022-07-20T02:24:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2022-07-20T02:24:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:24:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2022-07-20T02:24:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2022-07-20T02:24:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:24:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2022-07-20T02:24:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:24:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:24:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2022-07-20T02:24:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:24:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.673 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2022-07-20T02:24:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:24:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2022-07-20T02:24:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:24:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:24:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2022-07-20T02:24:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.675 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2022-07-20T02:24:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:24:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:24:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2022-07-20T02:24:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:24:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2022-07-20T02:24:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2022-07-20T02:24:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2022-07-20T02:24:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2022-07-20T02:24:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:24:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:24:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:24:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:24:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:24:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:24:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2022-07-20T02:24:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:24:46.680 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2022-07-20T02:24:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2022-07-20T02:24:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2022-07-20T02:24:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2022-07-20T02:24:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:24:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:24:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2022-07-20T02:24:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2022-07-20T02:24:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:24:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.683 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2022-07-20T02:24:46.683 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2022-07-20T02:24:46.683 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2022-07-20T02:24:46.684 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2022-07-20T02:24:46.684 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2022-07-20T02:24:46.684 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2022-07-20T02:24:46.684 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2022-07-20T02:24:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2022-07-20T02:24:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:24:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal frag = True 2022-07-20T02:24:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal fragment size max = 10000 2022-07-20T02:24:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal merge size = 5 2022-07-20T02:24:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split bits = 3 2022-07-20T02:24:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split size = 100 2022-07-20T02:24:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2022-07-20T02:24:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2022-07-20T02:24:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2022-07-20T02:24:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2022-07-20T02:24:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:24:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.688 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2022-07-20T02:24:46.688 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2022-07-20T02:24:46.688 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:24:46.689 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2022-07-20T02:24:46.689 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.689 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2022-07-20T02:24:46.689 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.690 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2022-07-20T02:24:46.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.690 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2022-07-20T02:24:46.690 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2022-07-20T02:24:46.691 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.691 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2022-07-20T02:24:46.691 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2022-07-20T02:24:46.691 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.691 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:24:46.692 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2022-07-20T02:24:46.692 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.692 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:24:46.692 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:24:46.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.693 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2022-07-20T02:24:46.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.693 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2022-07-20T02:24:46.694 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2022-07-20T02:24:46.694 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:24:46.694 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:24:46.694 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2022-07-20T02:24:46.694 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:24:46.695 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2022-07-20T02:24:46.695 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:24:46.695 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:24:46.695 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.696 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:24:46.696 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.696 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2022-07-20T02:24:46.696 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.696 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2022-07-20T02:24:46.697 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.697 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2022-07-20T02:24:46.697 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2022-07-20T02:24:46.697 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2022-07-20T02:24:46.698 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:24:46.698 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:24:46.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.698 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2022-07-20T02:24:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2022-07-20T02:24:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2022-07-20T02:24:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2022-07-20T02:24:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2022-07-20T02:24:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2022-07-20T02:24:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2022-07-20T02:24:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2022-07-20T02:24:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2022-07-20T02:24:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2022-07-20T02:24:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2022-07-20T02:24:46.702 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.702 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:24:46.702 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.702 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2022-07-20T02:24:46.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.703 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2022-07-20T02:24:46.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.703 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:24:46.703 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2022-07-20T02:24:46.704 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.704 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2022-07-20T02:24:46.704 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2022-07-20T02:24:46.704 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2022-07-20T02:24:46.705 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2022-07-20T02:24:46.705 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2022-07-20T02:24:46.705 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2022-07-20T02:24:46.705 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2022-07-20T02:24:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:24:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2022-07-20T02:24:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2022-07-20T02:24:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:24:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2022-07-20T02:24:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:24:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:24:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2022-07-20T02:24:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:24:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.708 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2022-07-20T02:24:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:24:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2022-07-20T02:24:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:24:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:24:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2022-07-20T02:24:46.710 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.710 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2022-07-20T02:24:46.710 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:24:46.710 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:24:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2022-07-20T02:24:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:24:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2022-07-20T02:24:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2022-07-20T02:24:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:24:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2022-07-20T02:24:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2022-07-20T02:24:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:24:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:24:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:24:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:24:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:24:46.714 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:24:46.714 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:24:46.714 INFO:teuthology.orchestra.run.smithi026.stdout: mon op complaint time = 120 2022-07-20T02:24:46.714 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:24:46.715 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2022-07-20T02:24:46.715 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2022-07-20T02:24:46.715 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2022-07-20T02:24:46.715 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2022-07-20T02:24:46.715 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:24:46.716 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:24:46.716 INFO:teuthology.orchestra.run.smithi026.stdout: client mount timeout = 600 2022-07-20T02:24:46.716 INFO:teuthology.orchestra.run.smithi026.stdout: debug client = 20 2022-07-20T02:24:46.716 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:24:46.717 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.717 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.717 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2022-07-20T02:24:46.717 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2022-07-20T02:24:46.718 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2022-07-20T02:24:46.718 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.a] 2022-07-20T02:24:46.718 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.c] 2022-07-20T02:24:46.718 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.b] 2022-07-20T02:24:46.718 INFO:teuthology.orchestra.run.smithi026.stdout:[mds] 2022-07-20T02:24:46.719 INFO:teuthology.orchestra.run.smithi026.stdout: debug mds = 20 2022-07-20T02:24:46.719 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:24:46.719 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal frag = True 2022-07-20T02:24:46.719 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal fragment size max = 10000 2022-07-20T02:24:46.720 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal merge size = 5 2022-07-20T02:24:46.720 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split bits = 3 2022-07-20T02:24:46.720 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split size = 100 2022-07-20T02:24:46.720 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug frag = True 2022-07-20T02:24:46.720 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug scatterstat = True 2022-07-20T02:24:46.721 INFO:teuthology.orchestra.run.smithi026.stdout: mds op complaint time = 180 2022-07-20T02:24:46.721 INFO:teuthology.orchestra.run.smithi026.stdout: mds verify scatter = True 2022-07-20T02:24:46.721 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:24:46.721 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:24:46.722 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:24:46.722 INFO:tasks.ceph:Creating admin key on mon.a... 2022-07-20T02:24:46.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-07-20T02:24:46.772 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-20T02:24:46.793 DEBUG:teuthology.orchestra.remote:smithi026:/etc/ceph/ceph.keyring is 206B 2022-07-20T02:24:46.832 DEBUG:teuthology.orchestra.remote:smithi026:/home/ubuntu/cephtest/ceph.monmap is 379B 2022-07-20T02:24:46.891 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:24:46.891 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:46.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T02:24:46.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T02:24:46.924 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2022-07-20T02:24:46.932 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:46.983 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:24:46.984 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:46.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T02:24:46.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T02:24:47.052 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-07-20T02:24:47.052 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:47.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2022-07-20T02:24:47.107 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:47.107 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T02:24:47.108 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T02:24:47.141 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2022-07-20T02:24:47.142 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:47.209 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2022-07-20T02:24:47.210 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:47.210 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T02:24:47.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T02:24:47.252 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-07-20T02:24:47.253 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:47.320 INFO:tasks.ceph:Setting up mon nodes... 2022-07-20T02:24:47.321 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-20T02:24:47.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-07-20T02:24:47.367 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-07-20T02:24:47.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-07-20T02:24:47.434 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-07-20T02:24:47.437 INFO:tasks.ceph:Setting up mds nodes... 2022-07-20T02:24:47.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-07-20T02:24:47.486 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-07-20T02:24:47.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-07-20T02:24:47.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2022-07-20T02:24:47.589 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2022-07-20T02:24:47.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2022-07-20T02:24:47.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-07-20T02:24:47.648 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-07-20T02:24:47.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-07-20T02:24:47.664 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-20T02:24:47.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-07-20T02:24:47.712 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-07-20T02:24:47.725 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-07-20T02:24:47.774 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-07-20T02:24:47.787 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-20T02:24:47.788 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-20T02:24:47.789 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-07-20T02:24:47.809 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:47.810 INFO:tasks.ceph:role: osd.0 2022-07-20T02:24:47.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:24:47.811 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-20T02:24:47.897 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:24:47.897 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:24:47.898 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:24:47.899 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 2022-07-20T02:24:47.899 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:24:47.899 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:24:47.899 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:24:47.900 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:24:47.900 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:24:47.900 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:24:47.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2022-07-20T02:24:47.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-07-20T02:24:47.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-07-20T02:24:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:24:48.024 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:24:48.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-07-20T02:24:48.092 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:48.093 INFO:tasks.ceph:role: osd.1 2022-07-20T02:24:48.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:24:48.095 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-20T02:24:48.153 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:24:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:24:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:24:48.173 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 2022-07-20T02:24:48.174 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:24:48.174 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:24:48.174 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:24:48.175 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:24:48.175 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:24:48.176 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:24:48.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2022-07-20T02:24:48.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-07-20T02:24:48.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-07-20T02:24:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:24:48.248 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:24:48.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-07-20T02:24:48.322 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:48.323 INFO:tasks.ceph:role: osd.2 2022-07-20T02:24:48.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:24:48.324 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-20T02:24:48.371 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:24:48.371 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:24:48.373 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:24:48.373 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 2022-07-20T02:24:48.373 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:24:48.373 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:24:48.374 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:24:48.374 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:24:48.374 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:24:48.374 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:24:48.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2022-07-20T02:24:48.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-07-20T02:24:48.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-07-20T02:24:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:24:48.467 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:24:48.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-07-20T02:24:48.484 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:48.485 INFO:tasks.ceph:role: osd.3 2022-07-20T02:24:48.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T02:24:48.487 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-20T02:24:48.572 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:24:48.573 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:24:48.573 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:24:48.574 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 2022-07-20T02:24:48.574 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:24:48.575 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:24:48.575 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:24:48.576 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:24:48.576 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:24:48.577 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:24:48.579 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2022-07-20T02:24:48.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-07-20T02:24:48.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-07-20T02:24:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:24:48.653 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:24:48.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:48.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-07-20T02:24:48.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-07-20T02:24:48.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-20T02:24:48.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-20T02:24:48.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-20T02:24:48.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.715+0000 7fd070611d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-07-20T02:24:48.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:48.731+0000 7fd070611d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:24:51.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-07-20T02:24:51.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-07-20T02:24:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-07-20T02:24:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-20T02:24:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-20T02:24:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-20T02:24:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.247+0000 7fd87c396d80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-07-20T02:24:51.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:51.263+0000 7fd87c396d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:24:53.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-07-20T02:24:53.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:53.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.803+0000 7fc6655c8d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-07-20T02:24:53.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.803+0000 7fc6655c8d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-07-20T02:24:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.803+0000 7fc6655c8d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-20T02:24:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.803+0000 7fc6655c8d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-20T02:24:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.803+0000 7fc6655c8d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-20T02:24:53.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.807+0000 7fc6655c8d80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-07-20T02:24:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:53.819+0000 7fc6655c8d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:24:56.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-07-20T02:24:56.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:24:56.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-07-20T02:24:56.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-07-20T02:24:56.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-20T02:24:56.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-20T02:24:56.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-20T02:24:56.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.411+0000 7f8b58c82d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2022-07-20T02:24:56.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:24:56.431+0000 7f8b58c82d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:24:58.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-07-20T02:24:58.905 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-07-20T02:24:58.927 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:58.928 INFO:tasks.ceph:role: osd.4 2022-07-20T02:24:58.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:24:58.929 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-20T02:24:59.504 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:24:59.504 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:24:59.505 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:24:59.505 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2022-07-20T02:24:59.520 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:24:59.521 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:24:59.521 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:24:59.522 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:24:59.522 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:24:59.525 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:24:59.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2022-07-20T02:24:59.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-07-20T02:24:59.651 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-07-20T02:24:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:24:59.663 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:24:59.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-07-20T02:24:59.684 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T02:24:59.684 INFO:tasks.ceph:role: osd.5 2022-07-20T02:24:59.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:24:59.685 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-20T02:25:00.205 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:25:00.205 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:25:00.206 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:25:00.206 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2022-07-20T02:25:00.207 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:25:00.207 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:25:00.208 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:25:00.208 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:25:00.209 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:25:00.209 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:25:00.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2022-07-20T02:25:00.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-07-20T02:25:00.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-07-20T02:25:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:25:00.298 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:25:00.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-07-20T02:25:00.313 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T02:25:00.314 INFO:tasks.ceph:role: osd.6 2022-07-20T02:25:00.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:25:00.315 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-20T02:25:00.858 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:25:00.859 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:25:00.859 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:25:00.859 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2022-07-20T02:25:00.860 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:25:00.860 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:25:00.860 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:25:00.860 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:25:00.861 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:25:00.861 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:25:00.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2022-07-20T02:25:00.862 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-07-20T02:25:00.915 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-07-20T02:25:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:25:00.928 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:25:00.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-07-20T02:25:00.963 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T02:25:00.963 INFO:tasks.ceph:role: osd.7 2022-07-20T02:25:00.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T02:25:00.964 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-20T02:25:01.541 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T02:25:01.542 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T02:25:01.542 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T02:25:01.543 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 2022-07-20T02:25:01.554 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T02:25:01.555 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2022-07-20T02:25:01.555 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T02:25:01.555 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T02:25:01.556 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T02:25:01.556 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T02:25:01.557 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2022-07-20T02:25:01.558 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-07-20T02:25:01.656 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-07-20T02:25:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:25:01.675 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T02:25:01.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.785+0000 7f56f5b93d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-07-20T02:25:01.785 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.785+0000 7f56f5b93d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-07-20T02:25:01.785 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.785+0000 7f56f5b93d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-20T02:25:01.785 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.785+0000 7f56f5b93d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-20T02:25:01.795 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.785+0000 7f56f5b93d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-20T02:25:01.796 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.793+0000 7f56f5b93d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-07-20T02:25:01.823 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:01.825+0000 7f56f5b93d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:25:04.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-07-20T02:25:04.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:04.350 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-07-20T02:25:04.351 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-07-20T02:25:04.352 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-20T02:25:04.352 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-20T02:25:04.353 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-20T02:25:04.353 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.353+0000 7f4006e9bd80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2022-07-20T02:25:04.369 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:04.373+0000 7f4006e9bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:25:06.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-07-20T02:25:06.859 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:06.944 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-07-20T02:25:06.945 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-07-20T02:25:06.946 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-20T02:25:06.946 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-20T02:25:06.947 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-20T02:25:06.947 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.945+0000 7fada0f9cd80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2022-07-20T02:25:06.964 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:06.965+0000 7fada0f9cd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:25:09.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-07-20T02:25:09.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:09.442 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-07-20T02:25:09.442 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-07-20T02:25:09.443 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-20T02:25:09.443 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-20T02:25:09.444 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-20T02:25:09.444 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.441+0000 7fa2a3d03d80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2022-07-20T02:25:09.458 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:09.461+0000 7fa2a3d03d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-20T02:25:11.880 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-07-20T02:25:11.938 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-20T02:25:11.945 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:11.955 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.gTrfu62i5M 2022-07-20T02:25:11.956 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.gTrfu62i5M 2022-07-20T02:25:12.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.gTrfu62i5M 2022-07-20T02:25:12.076 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.gTrfu62i5M is 56B 2022-07-20T02:25:12.095 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.gTrfu62i5M 2022-07-20T02:25:12.103 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:12.151 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.FPC7eFhVjH 2022-07-20T02:25:12.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.FPC7eFhVjH 2022-07-20T02:25:12.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.FPC7eFhVjH 2022-07-20T02:25:12.245 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.FPC7eFhVjH is 56B 2022-07-20T02:25:12.311 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.FPC7eFhVjH 2022-07-20T02:25:12.318 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:12.366 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.2FNzGW111n 2022-07-20T02:25:12.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/mds/ceph-c/keyring /tmp/tmp.2FNzGW111n 2022-07-20T02:25:12.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.2FNzGW111n 2022-07-20T02:25:12.463 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.2FNzGW111n is 56B 2022-07-20T02:25:12.486 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.2FNzGW111n 2022-07-20T02:25:12.500 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:12.567 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.kGV82jHqPT 2022-07-20T02:25:12.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.kGV82jHqPT 2022-07-20T02:25:12.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.kGV82jHqPT 2022-07-20T02:25:12.655 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.kGV82jHqPT is 56B 2022-07-20T02:25:12.725 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.kGV82jHqPT 2022-07-20T02:25:12.741 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:12.748 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.M90wT2KE6W 2022-07-20T02:25:12.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.M90wT2KE6W 2022-07-20T02:25:12.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.M90wT2KE6W 2022-07-20T02:25:12.901 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.M90wT2KE6W is 56B 2022-07-20T02:25:12.977 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.M90wT2KE6W 2022-07-20T02:25:12.988 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:13.040 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.It1E8VCjG1 2022-07-20T02:25:13.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.It1E8VCjG1 2022-07-20T02:25:13.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.It1E8VCjG1 2022-07-20T02:25:13.197 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.It1E8VCjG1 is 56B 2022-07-20T02:25:13.221 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.It1E8VCjG1 2022-07-20T02:25:13.230 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:13.280 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.sdub6MtZiS 2022-07-20T02:25:13.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.sdub6MtZiS 2022-07-20T02:25:13.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.sdub6MtZiS 2022-07-20T02:25:13.373 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.sdub6MtZiS is 56B 2022-07-20T02:25:13.400 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.sdub6MtZiS 2022-07-20T02:25:13.427 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:13.435 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.UfiB8VSenS 2022-07-20T02:25:13.435 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.UfiB8VSenS 2022-07-20T02:25:13.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.UfiB8VSenS 2022-07-20T02:25:13.523 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.UfiB8VSenS is 56B 2022-07-20T02:25:13.576 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.UfiB8VSenS 2022-07-20T02:25:13.585 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:13.634 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.BsCajY1Y8G 2022-07-20T02:25:13.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.BsCajY1Y8G 2022-07-20T02:25:13.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.BsCajY1Y8G 2022-07-20T02:25:13.724 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.BsCajY1Y8G is 56B 2022-07-20T02:25:13.778 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.BsCajY1Y8G 2022-07-20T02:25:13.786 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:13.800 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.3uUR22ZZFZ 2022-07-20T02:25:13.801 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.3uUR22ZZFZ 2022-07-20T02:25:13.860 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.3uUR22ZZFZ 2022-07-20T02:25:13.887 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.3uUR22ZZFZ is 56B 2022-07-20T02:25:13.904 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.3uUR22ZZFZ 2022-07-20T02:25:13.913 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:13.964 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.DfIyPmygaF 2022-07-20T02:25:13.965 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.DfIyPmygaF 2022-07-20T02:25:13.982 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.DfIyPmygaF 2022-07-20T02:25:14.052 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.DfIyPmygaF is 56B 2022-07-20T02:25:14.111 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.DfIyPmygaF 2022-07-20T02:25:14.119 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:14.168 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.Kd6EKvybzx 2022-07-20T02:25:14.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.Kd6EKvybzx 2022-07-20T02:25:14.185 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.Kd6EKvybzx 2022-07-20T02:25:14.255 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.Kd6EKvybzx is 56B 2022-07-20T02:25:14.331 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.Kd6EKvybzx 2022-07-20T02:25:14.341 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:14.391 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.HWtLBPdsF1 2022-07-20T02:25:14.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.HWtLBPdsF1 2022-07-20T02:25:14.408 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.HWtLBPdsF1 2022-07-20T02:25:14.473 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.HWtLBPdsF1 is 56B 2022-07-20T02:25:14.490 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.HWtLBPdsF1 2022-07-20T02:25:14.510 DEBUG:teuthology.orchestra.remote:smithi159:/etc/ceph/ceph.client.0.keyring is 59B 2022-07-20T02:25:14.590 DEBUG:teuthology.orchestra.remote:smithi176:/etc/ceph/ceph.client.1.keyring is 59B 2022-07-20T02:25:14.651 INFO:tasks.ceph:Adding keys to all mons... 2022-07-20T02:25:14.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-20T02:25:14.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-20T02:25:14.668 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.y] 2022-07-20T02:25:14.668 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg== 2022-07-20T02:25:14.669 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.a] 2022-07-20T02:25:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ== 2022-07-20T02:25:14.672 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.c] 2022-07-20T02:25:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg== 2022-07-20T02:25:14.673 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2022-07-20T02:25:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw== 2022-07-20T02:25:14.674 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2022-07-20T02:25:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA== 2022-07-20T02:25:14.674 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2022-07-20T02:25:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g== 2022-07-20T02:25:14.675 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2022-07-20T02:25:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA== 2022-07-20T02:25:14.675 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2022-07-20T02:25:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ== 2022-07-20T02:25:14.676 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.b] 2022-07-20T02:25:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww== 2022-07-20T02:25:14.676 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2022-07-20T02:25:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg== 2022-07-20T02:25:14.677 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2022-07-20T02:25:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ== 2022-07-20T02:25:14.677 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.6] 2022-07-20T02:25:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A== 2022-07-20T02:25:14.678 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.7] 2022-07-20T02:25:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ== 2022-07-20T02:25:14.678 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2022-07-20T02:25:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA== 2022-07-20T02:25:14.679 INFO:teuthology.orchestra.run.smithi026.stdout:[client.1] 2022-07-20T02:25:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg== 2022-07-20T02:25:14.681 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2022-07-20T02:25:14.682 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg== 2022-07-20T02:25:14.690 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2022-07-20T02:25:14.690 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ== 2022-07-20T02:25:14.690 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2022-07-20T02:25:14.690 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg== 2022-07-20T02:25:14.691 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2022-07-20T02:25:14.691 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw== 2022-07-20T02:25:14.691 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2022-07-20T02:25:14.691 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA== 2022-07-20T02:25:14.692 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2022-07-20T02:25:14.692 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g== 2022-07-20T02:25:14.692 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2022-07-20T02:25:14.692 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA== 2022-07-20T02:25:14.693 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2022-07-20T02:25:14.693 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ== 2022-07-20T02:25:14.693 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2022-07-20T02:25:14.693 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww== 2022-07-20T02:25:14.694 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2022-07-20T02:25:14.694 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg== 2022-07-20T02:25:14.694 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2022-07-20T02:25:14.694 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ== 2022-07-20T02:25:14.695 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2022-07-20T02:25:14.695 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A== 2022-07-20T02:25:14.695 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2022-07-20T02:25:14.695 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ== 2022-07-20T02:25:14.695 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2022-07-20T02:25:14.696 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA== 2022-07-20T02:25:14.696 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2022-07-20T02:25:14.696 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg== 2022-07-20T02:25:14.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T02:25:14.701 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T02:25:14.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:14.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:14.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:14.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:14.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:14.978 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.111 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.158 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T02:25:15.250 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T02:25:15.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:15.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T02:25:15.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.494 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.636 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T02:25:15.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T02:25:15.684 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T02:25:15.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:15.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T02:25:15.776 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:15.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T02:25:15.781 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:15.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:15.857 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-20T02:25:15.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-07-20T02:25:15.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T02:25:15.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-07-20T02:25:15.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-07-20T02:25:16.011 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T02:25:16.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-07-20T02:25:16.080 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-07-20T02:25:16.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T02:25:16.147 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-07-20T02:25:16.164 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:16.170 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-20T02:25:16.176 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-07-20T02:25:16.177 INFO:tasks.ceph.mon.a:Restarting daemon 2022-07-20T02:25:16.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-07-20T02:25:16.182 INFO:tasks.ceph.mon.a:Started 2022-07-20T02:25:16.182 INFO:tasks.ceph.mon.b:Restarting daemon 2022-07-20T02:25:16.183 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-07-20T02:25:16.222 INFO:tasks.ceph.mon.b:Started 2022-07-20T02:25:16.223 INFO:tasks.ceph.mon.c:Restarting daemon 2022-07-20T02:25:16.223 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-07-20T02:25:16.226 INFO:tasks.ceph.mon.c:Started 2022-07-20T02:25:16.226 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-07-20T02:25:16.227 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-07-20T02:25:16.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-07-20T02:25:16.230 INFO:tasks.ceph.mgr.y:Started 2022-07-20T02:25:16.230 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-07-20T02:25:16.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-07-20T02:25:16.233 INFO:tasks.ceph.mgr.x:Started 2022-07-20T02:25:16.234 INFO:tasks.ceph:Setting crush tunables to default 2022-07-20T02:25:16.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2022-07-20T02:25:16.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 Processor -- start 2022-07-20T02:25:16.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 -- start start 2022-07-20T02:25:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f405ff70 con 0x7fd0f40b3260 2022-07-20T02:25:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f40606e0 con 0x7fd0f40980d0 2022-07-20T02:25:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f9bcb700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3640 con 0x7fd0f4061860 2022-07-20T02:25:16.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f93ca700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd0f40980d0 0x7fd0f4142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:47832/0 (socket says 172.21.15.26:47832) 2022-07-20T02:25:16.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2994341985 learned_addr learned my addr 172.21.15.26:0/2994341985 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:16.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f3fff700 1 -- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 msgr2=0x7fd0f4067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-20T02:25:16.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f3fff700 1 -- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 msgr2=0x7fd0f4067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-20T02:25:16.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-20T02:25:16.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.331+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-20T02:25:16.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.535+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.535+0000 7fd0f3fff700 1 -- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 msgr2=0x7fd0f4067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-20T02:25:16.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.535+0000 7fd0f3fff700 1 -- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 msgr2=0x7fd0f4067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-20T02:25:16.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.535+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-20T02:25:16.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.535+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-20T02:25:16.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256545340 0 0) 0x7fd0f40606e0 con 0x7fd0f40980d0 2022-07-20T02:25:16.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d8003260 con 0x7fd0f40980d0 2022-07-20T02:25:16.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702262476 0 0) 0x7fd0f405ff70 con 0x7fd0f40b3260 2022-07-20T02:25:16.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f40606e0 con 0x7fd0f40b3260 2022-07-20T02:25:16.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958714444 0 0) 0x7fd0f40606e0 con 0x7fd0f40b3260 2022-07-20T02:25:16.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 msgr2=0x7fd0f4067900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-20T02:25:16.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 --2- 172.21.15.26:0/2994341985 >> v2:172.21.15.120:3300/0 conn(0x7fd0f4061860 0x7fd0f4067900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 >> v1:172.21.15.120:6789/0 conn(0x7fd0f40980d0 legacy=0x7fd0f4142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40601a0 con 0x7fd0f40b3260 2022-07-20T02:25:16.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fd0e00015e0 con 0x7fd0f40b3260 2022-07-20T02:25:16.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.547+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 >> v1:172.21.15.26:6789/0 conn(0x7fd0f40b3260 legacy=0x7fd0f40979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 0x7fd0d8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d8004500 0x7fd0d8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d8004ec0 0x7fd0d800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3640 con 0x7fd0d80038e0 2022-07-20T02:25:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800bd80 con 0x7fd0d8004500 2022-07-20T02:25:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800bf40 con 0x7fd0d8004ec0 2022-07-20T02:25:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 0x7fd0d8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d8004500 0x7fd0d8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f93ca700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d8004ec0 0x7fd0d800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 -- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d8004ec0 msgr2=0x7fd0d800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d8004ec0 0x7fd0d800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 -- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d8004500 msgr2=0x7fd0d8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d8004500 0x7fd0d8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2994341985 reap_dead start 2022-07-20T02:25:16.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 -- 172.21.15.26:0/2994341985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40606e0 con 0x7fd0d80038e0 2022-07-20T02:25:16.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f8bc9700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 0x7fd0d8003ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd0e00013b0 tx=0x7fd0f4060330).ready entity=mon.0 client_cookie=b2695dbefd709a68 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:16.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0e00018f0 con 0x7fd0d80038e0 2022-07-20T02:25:16.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e0001a50 con 0x7fd0d80038e0 2022-07-20T02:25:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.551+0000 7fd0f37fe700 1 -- 172.21.15.26:0/2994341985 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0e0001db0 con 0x7fd0d80038e0 2022-07-20T02:25:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 msgr2=0x7fd0d8003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 0x7fd0d8003ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd0e00013b0 tx=0x7fd0f4060330).stop 2022-07-20T02:25:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2994341985 shutdown_connections 2022-07-20T02:25:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2994341985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0d80038e0 0x7fd0d8003ce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2994341985 >> 172.21.15.26:0/2994341985 conn(0x7fd0f40b67f0 msgr2=0x7fd0f40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2994341985 shutdown_connections 2022-07-20T02:25:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2994341985 wait complete. 2022-07-20T02:25:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 Processor -- start 2022-07-20T02:25:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.555+0000 7fd0f9bcb700 1 -- start start 2022-07-20T02:25:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d80038e0 0x7fd0f4151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f40b56f0 0x7fd0f4152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b5420 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f414ef00 con 0x7fd0f40b56f0 2022-07-20T02:25:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f414f1a0 con 0x7fd0d80038e0 2022-07-20T02:25:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34370/0 (socket says 172.21.15.26:34370) 2022-07-20T02:25:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f8bc9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d80038e0 0x7fd0f4151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2728971729 learned_addr learned my addr 172.21.15.26:0/2728971729 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f3fff700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f40b56f0 0x7fd0f4152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d80038e0 msgr2=0x7fd0f4151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d80038e0 0x7fd0f4151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f40b56f0 msgr2=0x7fd0f4152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f40b56f0 0x7fd0f4152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 -- 172.21.15.26:0/2728971729 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40aff50 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f93ca700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd0ec000b60 tx=0x7fd0ec007e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0ec0079b0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f4156ca0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f4156fd0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec0116b0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0ec011a10 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fd0ec01b030 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.559+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd0ec01e7b0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.563+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc003410 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.567+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fd0ec019070 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.627+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd0cc0036c0 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f1ffb700 1 -- 172.21.15.26:0/2728971729 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd0ec01e970 con 0x7fd0f40b4aa0 2022-07-20T02:25:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 msgr2=0x7fd0f40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd0ec000b60 tx=0x7fd0ec007e30).stop 2022-07-20T02:25:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 shutdown_connections 2022-07-20T02:25:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0f40b4aa0 0x7fd0f40b4f30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f40b56f0 0x7fd0f4152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 --2- 172.21.15.26:0/2728971729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0d80038e0 0x7fd0f4151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 >> 172.21.15.26:0/2728971729 conn(0x7fd0f40b67f0 msgr2=0x7fd0f4143410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 shutdown_connections 2022-07-20T02:25:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:16.635+0000 7fd0f9bcb700 1 -- 172.21.15.26:0/2728971729 wait complete. 2022-07-20T02:25:16.649 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2022-07-20T02:25:16.651 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-07-20T02:25:16.651 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:16.657 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.L3j72mEkN5 2022-07-20T02:25:16.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.L3j72mEkN5 2022-07-20T02:25:16.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.L3j72mEkN5 2022-07-20T02:25:16.757 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.L3j72mEkN5 is 37B 2022-07-20T02:25:16.775 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.L3j72mEkN5 2022-07-20T02:25:16.783 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:16.830 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.mH1MMWzv96 2022-07-20T02:25:16.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.mH1MMWzv96 2022-07-20T02:25:16.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.mH1MMWzv96 2022-07-20T02:25:16.916 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.mH1MMWzv96 is 37B 2022-07-20T02:25:16.947 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.mH1MMWzv96 2022-07-20T02:25:16.955 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:17.002 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.nJHIPw40hr 2022-07-20T02:25:17.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.nJHIPw40hr 2022-07-20T02:25:17.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.nJHIPw40hr 2022-07-20T02:25:17.084 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.nJHIPw40hr is 37B 2022-07-20T02:25:17.142 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.nJHIPw40hr 2022-07-20T02:25:17.153 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T02:25:17.206 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.GP2FShxTjM 2022-07-20T02:25:17.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.GP2FShxTjM 2022-07-20T02:25:17.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.GP2FShxTjM 2022-07-20T02:25:17.306 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.GP2FShxTjM is 37B 2022-07-20T02:25:17.327 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.GP2FShxTjM 2022-07-20T02:25:17.341 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:17.351 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.yQPvT5ACfU 2022-07-20T02:25:17.355 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.yQPvT5ACfU 2022-07-20T02:25:17.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.yQPvT5ACfU 2022-07-20T02:25:17.489 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.yQPvT5ACfU is 37B 2022-07-20T02:25:17.559 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.yQPvT5ACfU 2022-07-20T02:25:17.569 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:17.618 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.E5RhRDhGBm 2022-07-20T02:25:17.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.E5RhRDhGBm 2022-07-20T02:25:17.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.E5RhRDhGBm 2022-07-20T02:25:17.718 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.E5RhRDhGBm is 37B 2022-07-20T02:25:17.745 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.E5RhRDhGBm 2022-07-20T02:25:17.753 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:17.799 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.deXisN2g0Y 2022-07-20T02:25:17.800 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.deXisN2g0Y 2022-07-20T02:25:17.820 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.deXisN2g0Y 2022-07-20T02:25:17.884 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.deXisN2g0Y is 37B 2022-07-20T02:25:17.944 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.deXisN2g0Y 2022-07-20T02:25:17.953 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T02:25:17.999 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.ouJbBIPABO 2022-07-20T02:25:17.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.ouJbBIPABO 2022-07-20T02:25:18.015 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.ouJbBIPABO 2022-07-20T02:25:18.083 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.ouJbBIPABO is 37B 2022-07-20T02:25:18.108 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.ouJbBIPABO 2022-07-20T02:25:18.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 9b99f86d-740c-4e86-aa14-2c202b408772 0 2022-07-20T02:25:18.264 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.265+0000 7ff525b27700 1 Processor -- start 2022-07-20T02:25:18.265 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.265+0000 7ff525b27700 1 -- start start 2022-07-20T02:25:18.277 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.277 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5200b0f90 0x7ff5201436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.277 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff520094fc0 con 0x7ff520061750 2022-07-20T02:25:18.278 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5200602b0 con 0x7ff5200b0f90 2022-07-20T02:25:18.278 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200b2120 con 0x7ff52005f9e0 2022-07-20T02:25:18.279 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.279 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54032/0 (socket says 172.21.15.120:54032) 2022-07-20T02:25:18.280 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5200b0f90 0x7ff5201436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.280 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 -- 172.21.15.120:0/1093916392 learned_addr learned my addr 172.21.15.120:0/1093916392 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.280 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 -- 172.21.15.120:0/1093916392 >> v2:172.21.15.120:3300/0 conn(0x7ff5200b0f90 msgr2=0x7ff5201436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.281 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 --2- 172.21.15.120:0/1093916392 >> v2:172.21.15.120:3300/0 conn(0x7ff5200b0f90 0x7ff5201436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.281 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 -- 172.21.15.120:0/1093916392 >> v1:172.21.15.120:6789/0 conn(0x7ff520061750 legacy=0x7ff5200b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.282 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 -- 172.21.15.120:0/1093916392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5200b1e80 con 0x7ff52005f9e0 2022-07-20T02:25:18.282 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51f7fe700 1 --2- 172.21.15.120:0/1093916392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff510002f70 tx=0x7ff510006eb0).ready entity=mon.2 client_cookie=9c067ae6eac6275c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.282 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51e7fc700 1 -- 172.21.15.120:0/1093916392 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff51000d570 con 0x7ff52005f9e0 2022-07-20T02:25:18.282 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51e7fc700 1 -- 172.21.15.120:0/1093916392 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff510000e90 con 0x7ff52005f9e0 2022-07-20T02:25:18.283 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff51e7fc700 1 -- 172.21.15.120:0/1093916392 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff51000d900 con 0x7ff52005f9e0 2022-07-20T02:25:18.283 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- 172.21.15.120:0/1093916392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 msgr2=0x7ff52005fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.283 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- 172.21.15.120:0/1093916392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff510002f70 tx=0x7ff510006eb0).stop 2022-07-20T02:25:18.283 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- 172.21.15.120:0/1093916392 shutdown_connections 2022-07-20T02:25:18.284 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- 172.21.15.120:0/1093916392 >> v2:172.21.15.120:3300/0 conn(0x7ff5200b0f90 0x7ff5201436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.284 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- 172.21.15.120:0/1093916392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff52005f9e0 0x7ff52005fdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.284 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- 172.21.15.120:0/1093916392 >> 172.21.15.120:0/1093916392 conn(0x7ff5200b5e70 msgr2=0x7ff5200b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.284 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- 172.21.15.120:0/1093916392 shutdown_connections 2022-07-20T02:25:18.285 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- 172.21.15.120:0/1093916392 wait complete. 2022-07-20T02:25:18.285 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 Processor -- start 2022-07-20T02:25:18.286 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- start start 2022-07-20T02:25:18.286 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.286 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 0x7ff520063e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.286 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.287 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52009af70 con 0x7ff520061750 2022-07-20T02:25:18.287 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200668f0 con 0x7ff52005f9e0 2022-07-20T02:25:18.287 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.269+0000 7ff525b27700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff520065f10 con 0x7ff5200b0f90 2022-07-20T02:25:18.287 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.288 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.288 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54042/0 (socket says 172.21.15.120:54042) 2022-07-20T02:25:18.288 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44048/0 (socket says 172.21.15.120:44048) 2022-07-20T02:25:18.288 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51ffff700 1 -- 172.21.15.120:0/639856367 learned_addr learned my addr 172.21.15.120:0/639856367 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.288 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51effd700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 0x7ff520063e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.289 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 -- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 msgr2=0x7ff520064300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.289 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.289 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 -- 172.21.15.120:0/639856367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 msgr2=0x7ff520063e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.289 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 0x7ff520063e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.290 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 -- 172.21.15.120:0/639856367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51000b1e0 con 0x7ff52005f9e0 2022-07-20T02:25:18.290 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51effd700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 0x7ff520063e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:18.290 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51ffff700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:18.291 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51f7fe700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff520065d50 tx=0x7ff510015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.291 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff510011860 con 0x7ff52005f9e0 2022-07-20T02:25:18.291 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5100119c0 con 0x7ff52005f9e0 2022-07-20T02:25:18.291 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5200661b0 con 0x7ff52005f9e0 2022-07-20T02:25:18.291 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff510011d20 con 0x7ff52005f9e0 2022-07-20T02:25:18.292 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5200647f0 con 0x7ff52005f9e0 2022-07-20T02:25:18.292 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7ff510025030 con 0x7ff52005f9e0 2022-07-20T02:25:18.292 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff510028440 con 0x7ff52005f9e0 2022-07-20T02:25:18.292 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.273+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520095220 con 0x7ff52005f9e0 2022-07-20T02:25:18.292 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.277+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7ff510023070 con 0x7ff52005f9e0 2022-07-20T02:25:18.340 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.341+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", "id": 0} v 0) v1 -- 0x7ff5200574e0 con 0x7ff52005f9e0 2022-07-20T02:25:18.348 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff51cff9700 1 -- 172.21.15.120:0/639856367 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff510028650 con 0x7ff52005f9e0 2022-07-20T02:25:18.349 INFO:teuthology.orchestra.run.smithi120.stdout:0 2022-07-20T02:25:18.352 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 msgr2=0x7ff5200971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.352 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff520065d50 tx=0x7ff510015000).stop 2022-07-20T02:25:18.353 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 shutdown_connections 2022-07-20T02:25:18.353 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff520061750 0x7ff520063e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.354 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5200b0f90 0x7ff520064300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.354 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 --2- 172.21.15.120:0/639856367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52005f9e0 0x7ff5200971f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.355 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 >> 172.21.15.120:0/639856367 conn(0x7ff5200b5e70 msgr2=0x7ff520097c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.355 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 shutdown_connections 2022-07-20T02:25:18.356 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.349+0000 7ff525b27700 1 -- 172.21.15.120:0/639856367 wait complete. 2022-07-20T02:25:18.360 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 0256d8ae-ca5e-445d-8c9e-25203aeea92b 1 2022-07-20T02:25:18.470 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 Processor -- start 2022-07-20T02:25:18.470 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 -- start start 2022-07-20T02:25:18.470 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc80005fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.471 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8000aeec0 con 0x7fc800067060 2022-07-20T02:25:18.471 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc800067510 con 0x7fc8000aeae0 2022-07-20T02:25:18.471 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc804ee3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000b2ba0 con 0x7fc8000602c0 2022-07-20T02:25:18.471 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fdd9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc80005fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.472 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fe59c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc8000aeae0 0x7fc800066950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:57358/0 (socket says 172.21.15.120:57358) 2022-07-20T02:25:18.472 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fe59c700 1 -- 172.21.15.120:0/2840647679 learned_addr learned my addr 172.21.15.120:0/2840647679 (peer_addr_for_me v1:172.21.15.120:0/0) 2022-07-20T02:25:18.472 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fdd9b700 1 -- 172.21.15.120:0/2840647679 >> v1:172.21.15.120:6789/0 conn(0x7fc8000aeae0 legacy=0x7fc800066950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.472 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fdd9b700 1 -- 172.21.15.120:0/2840647679 >> v1:172.21.15.26:6789/0 conn(0x7fc800067060 legacy=0x7fc8000af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.473 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fdd9b700 1 -- 172.21.15.120:0/2840647679 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000af210 con 0x7fc8000602c0 2022-07-20T02:25:18.473 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fdd9b700 1 --2- 172.21.15.120:0/2840647679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc80005fa50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc8000b4580 tx=0x7fc7e8006df0).ready entity=mon.2 client_cookie=baff8b8552467218 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.473 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fd59a700 1 -- 172.21.15.120:0/2840647679 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7e800d570 con 0x7fc8000602c0 2022-07-20T02:25:18.473 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fd59a700 1 -- 172.21.15.120:0/2840647679 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8000e90 con 0x7fc8000602c0 2022-07-20T02:25:18.474 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.469+0000 7fc7fd59a700 1 -- 172.21.15.120:0/2840647679 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7e800d8c0 con 0x7fc8000602c0 2022-07-20T02:25:18.474 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- 172.21.15.120:0/2840647679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 msgr2=0x7fc80005fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.474 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 --2- 172.21.15.120:0/2840647679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc80005fa50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc8000b4580 tx=0x7fc7e8006df0).stop 2022-07-20T02:25:18.474 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- 172.21.15.120:0/2840647679 shutdown_connections 2022-07-20T02:25:18.474 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 --2- 172.21.15.120:0/2840647679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc80005fa50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.475 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- 172.21.15.120:0/2840647679 >> 172.21.15.120:0/2840647679 conn(0x7fc8000b66e0 msgr2=0x7fc8000b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.475 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- 172.21.15.120:0/2840647679 shutdown_connections 2022-07-20T02:25:18.475 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- 172.21.15.120:0/2840647679 wait complete. 2022-07-20T02:25:18.479 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 Processor -- start 2022-07-20T02:25:18.479 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- start start 2022-07-20T02:25:18.479 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.480 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc800067060 0x7fc8000ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.480 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8000aeae0 0x7fc8000ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.480 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800064f80 con 0x7fc800067060 2022-07-20T02:25:18.480 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800065790 con 0x7fc8000aeae0 2022-07-20T02:25:18.481 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc804ee3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000667a0 con 0x7fc8000602c0 2022-07-20T02:25:18.481 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc7fe59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.482 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc7fe59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54050/0 (socket says 172.21.15.120:54050) 2022-07-20T02:25:18.482 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc7fe59c700 1 -- 172.21.15.120:0/477079981 learned_addr learned my addr 172.21.15.120:0/477079981 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.482 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc7fed9d700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8000aeae0 0x7fc8000ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.482 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.473+0000 7fc7fdd9b700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc800067060 0x7fc8000ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.483 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 -- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8000aeae0 msgr2=0x7fc8000ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.483 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8000aeae0 0x7fc8000ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.487 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 -- 172.21.15.120:0/477079981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc800067060 msgr2=0x7fc8000ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.487 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc800067060 0x7fc8000ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.488 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 -- 172.21.15.120:0/477079981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e800b180 con 0x7fc8000602c0 2022-07-20T02:25:18.488 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7fe59c700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc8000b2d70 tx=0x7fc7f00014a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.488 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f0005ba0 con 0x7fc8000602c0 2022-07-20T02:25:18.488 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f0005d00 con 0x7fc8000602c0 2022-07-20T02:25:18.488 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc800066a40 con 0x7fc8000602c0 2022-07-20T02:25:18.489 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f00072c0 con 0x7fc8000602c0 2022-07-20T02:25:18.489 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc800066470 con 0x7fc8000602c0 2022-07-20T02:25:18.489 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fc7f0010070 con 0x7fc8000602c0 2022-07-20T02:25:18.489 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.477+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fc7f0013450 con 0x7fc8000602c0 2022-07-20T02:25:18.490 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.481+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc800061630 con 0x7fc8000602c0 2022-07-20T02:25:18.490 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.485+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fc7f0031870 con 0x7fc8000602c0 2022-07-20T02:25:18.551 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.553+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", "id": 1} v 0) v1 -- 0x7fc8000574e0 con 0x7fc8000602c0 2022-07-20T02:25:18.559 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.561+0000 7fc7f77fe700 1 -- 172.21.15.120:0/477079981 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7f0021070 con 0x7fc8000602c0 2022-07-20T02:25:18.560 INFO:teuthology.orchestra.run.smithi120.stdout:1 2022-07-20T02:25:18.563 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 msgr2=0x7fc8000b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.563 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc8000b2d70 tx=0x7fc7f00014a0).stop 2022-07-20T02:25:18.564 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 shutdown_connections 2022-07-20T02:25:18.564 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc800067060 0x7fc8000ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.564 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8000aeae0 0x7fc8000ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.565 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 --2- 172.21.15.120:0/477079981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8000602c0 0x7fc8000b4900 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.565 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 >> 172.21.15.120:0/477079981 conn(0x7fc8000b66e0 msgr2=0x7fc8000b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.566 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 shutdown_connections 2022-07-20T02:25:18.566 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.565+0000 7fc804ee3700 1 -- 172.21.15.120:0/477079981 wait complete. 2022-07-20T02:25:18.574 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new e7012e19-7787-4117-be57-55c92bd2e289 2 2022-07-20T02:25:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 Processor -- start 2022-07-20T02:25:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- start start 2022-07-20T02:25:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe20c0b3260 0x7fe20c0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.680 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0b3640 con 0x7fe20c097c50 2022-07-20T02:25:18.681 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c060e90 con 0x7fe20c0b3260 2022-07-20T02:25:18.681 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c095300 con 0x7fe20c0609e0 2022-07-20T02:25:18.681 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.681 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54060/0 (socket says 172.21.15.120:54060) 2022-07-20T02:25:18.682 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 -- 172.21.15.120:0/665022257 learned_addr learned my addr 172.21.15.120:0/665022257 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.682 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe210840700 1 --2- 172.21.15.120:0/665022257 >> v2:172.21.15.26:3300/0 conn(0x7fe20c0b3260 0x7fe20c0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.682 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 -- 172.21.15.120:0/665022257 >> v2:172.21.15.26:3300/0 conn(0x7fe20c0b3260 msgr2=0x7fe20c0604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.683 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 --2- 172.21.15.120:0/665022257 >> v2:172.21.15.26:3300/0 conn(0x7fe20c0b3260 0x7fe20c0604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.683 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 -- 172.21.15.120:0/665022257 >> v1:172.21.15.26:6789/0 conn(0x7fe20c097c50 legacy=0x7fe20c0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.683 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 -- 172.21.15.120:0/665022257 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c0955a0 con 0x7fe20c0609e0 2022-07-20T02:25:18.683 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211041700 1 --2- 172.21.15.120:0/665022257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe2040032f0 tx=0x7fe204009290).ready entity=mon.2 client_cookie=99aae3ae5dea851c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.684 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe20b7fe700 1 -- 172.21.15.120:0/665022257 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe20400b8a0 con 0x7fe20c0609e0 2022-07-20T02:25:18.684 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe20b7fe700 1 -- 172.21.15.120:0/665022257 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20400ba00 con 0x7fe20c0609e0 2022-07-20T02:25:18.684 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe20b7fe700 1 -- 172.21.15.120:0/665022257 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe20400bb70 con 0x7fe20c0609e0 2022-07-20T02:25:18.688 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- 172.21.15.120:0/665022257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 msgr2=0x7fe20c0b1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.688 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 --2- 172.21.15.120:0/665022257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe2040032f0 tx=0x7fe204009290).stop 2022-07-20T02:25:18.690 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- 172.21.15.120:0/665022257 shutdown_connections 2022-07-20T02:25:18.690 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 --2- 172.21.15.120:0/665022257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c0609e0 0x7fe20c0b1e30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.691 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 --2- 172.21.15.120:0/665022257 >> v2:172.21.15.26:3300/0 conn(0x7fe20c0b3260 0x7fe20c0604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.691 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- 172.21.15.120:0/665022257 >> 172.21.15.120:0/665022257 conn(0x7fe20c0b67f0 msgr2=0x7fe20c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.691 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- 172.21.15.120:0/665022257 shutdown_connections 2022-07-20T02:25:18.691 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.673+0000 7fe211842700 1 -- 172.21.15.120:0/665022257 wait complete. 2022-07-20T02:25:18.692 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 Processor -- start 2022-07-20T02:25:18.692 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- start start 2022-07-20T02:25:18.692 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 0x7fe20c0afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.692 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.693 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe20c0b3260 0x7fe20c152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.693 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c095a40 con 0x7fe20c0b3260 2022-07-20T02:25:18.693 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c095ce0 con 0x7fe20c0609e0 2022-07-20T02:25:18.693 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c095f80 con 0x7fe20c097c50 2022-07-20T02:25:18.694 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe210840700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 0x7fe20c0afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.694 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.694 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54064/0 (socket says 172.21.15.120:54064) 2022-07-20T02:25:18.694 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe210840700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 0x7fe20c0afcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44072/0 (socket says 172.21.15.120:44072) 2022-07-20T02:25:18.696 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 -- 172.21.15.120:0/405613883 learned_addr learned my addr 172.21.15.120:0/405613883 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.696 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211041700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe20c0b3260 0x7fe20c152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.696 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 -- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 msgr2=0x7fe20c0afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.698 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 0x7fe20c0afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.699 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 -- 172.21.15.120:0/405613883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe20c0b3260 msgr2=0x7fe20c152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.699 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe20c0b3260 0x7fe20c152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.699 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 -- 172.21.15.120:0/405613883 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204009530 con 0x7fe20c097c50 2022-07-20T02:25:18.700 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe20bfff700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe20c0603d0 tx=0x7fe1f8007c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.701 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1f8012400 con 0x7fe20c097c50 2022-07-20T02:25:18.701 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f8012560 con 0x7fe20c097c50 2022-07-20T02:25:18.701 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c096220 con 0x7fe20c097c50 2022-07-20T02:25:18.702 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1f80128c0 con 0x7fe20c097c50 2022-07-20T02:25:18.703 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c0966c0 con 0x7fe20c097c50 2022-07-20T02:25:18.703 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fe1f8019030 con 0x7fe20c097c50 2022-07-20T02:25:18.703 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.677+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fe1f801b800 con 0x7fe20c097c50 2022-07-20T02:25:18.704 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.681+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c0963b0 con 0x7fe20c097c50 2022-07-20T02:25:18.704 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.681+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fe1f8017070 con 0x7fe20c097c50 2022-07-20T02:25:18.750 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.749+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", "id": 2} v 0) v1 -- 0x7fe20c095840 con 0x7fe20c097c50 2022-07-20T02:25:18.756 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.757+0000 7fe209ffb700 1 -- 172.21.15.120:0/405613883 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe1f801b3b0 con 0x7fe20c097c50 2022-07-20T02:25:18.758 INFO:teuthology.orchestra.run.smithi120.stdout:2 2022-07-20T02:25:18.766 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 msgr2=0x7fe20c1525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.766 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe20c0603d0 tx=0x7fe1f8007c50).stop 2022-07-20T02:25:18.767 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 shutdown_connections 2022-07-20T02:25:18.767 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe20c0b3260 0x7fe20c152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.768 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe20c097c50 0x7fe20c1525a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.769 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 --2- 172.21.15.120:0/405613883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20c0609e0 0x7fe20c0afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.769 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 >> 172.21.15.120:0/405613883 conn(0x7fe20c0b67f0 msgr2=0x7fe20c1468c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.770 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 shutdown_connections 2022-07-20T02:25:18.770 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.761+0000 7fe211842700 1 -- 172.21.15.120:0/405613883 wait complete. 2022-07-20T02:25:18.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 8abca3ca-a3f0-44e2-80e9-a3d7ff287442 3 2022-07-20T02:25:18.924 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 Processor -- start 2022-07-20T02:25:18.925 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 -- start start 2022-07-20T02:25:18.925 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f38ac0add30 0x7f38ac0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.926 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac0aa1e0 con 0x7f38ac060950 2022-07-20T02:25:18.926 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac0ae110 con 0x7f38ac0af6d0 2022-07-20T02:25:18.927 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b109a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac0a9f40 con 0x7f38ac0add30 2022-07-20T02:25:18.929 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b0899700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f38ac0af6d0 0x7f38ac0afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:57386/0 (socket says 172.21.15.120:57386) 2022-07-20T02:25:18.929 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38b0899700 1 -- 172.21.15.120:0/426333117 learned_addr learned my addr 172.21.15.120:0/426333117 (peer_addr_for_me v1:172.21.15.120:0/0) 2022-07-20T02:25:18.930 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38abfff700 1 --2- 172.21.15.120:0/426333117 >> v2:172.21.15.26:3300/0 conn(0x7f38ac0add30 0x7f38ac0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.930 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917455396 0 0) 0x7f38ac0ae110 con 0x7f38ac0af6d0 2022-07-20T02:25:18.931 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898003260 con 0x7f38ac0af6d0 2022-07-20T02:25:18.931 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699561475 0 0) 0x7f38ac0aa1e0 con 0x7f38ac060950 2022-07-20T02:25:18.931 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ac0ae110 con 0x7f38ac060950 2022-07-20T02:25:18.932 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654236772 0 0) 0x7f3898003260 con 0x7f38ac0af6d0 2022-07-20T02:25:18.932 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 >> v2:172.21.15.26:3300/0 conn(0x7f38ac0add30 msgr2=0x7f38ac0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.933 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 --2- 172.21.15.120:0/426333117 >> v2:172.21.15.26:3300/0 conn(0x7f38ac0add30 0x7f38ac0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.933 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 >> v1:172.21.15.26:6789/0 conn(0x7f38ac060950 legacy=0x7f38ac060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.934 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.877+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0b1110 con 0x7f38ac0af6d0 2022-07-20T02:25:18.934 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f389c001660 con 0x7f38ac0af6d0 2022-07-20T02:25:18.935 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 >> v1:172.21.15.120:6789/0 conn(0x7f38ac0af6d0 legacy=0x7f38ac0afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.935 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38980038e0 0x7f3898003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.936 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f3898004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.936 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3898004e70 0x7f389800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.937 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0a9f40 con 0x7f3898004e70 2022-07-20T02:25:18.938 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389800bcf0 con 0x7f3898004500 2022-07-20T02:25:18.938 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389800beb0 con 0x7f38980038e0 2022-07-20T02:25:18.939 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38abfff700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38980038e0 0x7f3898003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.939 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f3898004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.940 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b0899700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3898004e70 0x7f389800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.940 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 -- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38980038e0 msgr2=0x7f3898003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.941 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38980038e0 0x7f3898003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.941 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 -- 172.21.15.120:0/426333117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3898004e70 msgr2=0x7f389800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:18.941 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3898004e70 0x7f389800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.955 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b0899700 1 -- 172.21.15.120:0/426333117 reap_dead start 2022-07-20T02:25:18.956 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 -- 172.21.15.120:0/426333117 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898003260 con 0x7f3898004500 2022-07-20T02:25:18.956 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38abfff700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38980038e0 0x7f3898003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:18.956 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f3898004980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f38980054d0 tx=0x7f38a0007a00).ready entity=mon.1 client_cookie=d4089ee9b3d317a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.957 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a0007d10 con 0x7f3898004500 2022-07-20T02:25:18.957 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0007070 con 0x7f3898004500 2022-07-20T02:25:18.958 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38aaffd700 1 -- 172.21.15.120:0/426333117 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a00073d0 con 0x7f3898004500 2022-07-20T02:25:18.959 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 -- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 msgr2=0x7f3898004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.959 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f3898004980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f38980054d0 tx=0x7f38a0007a00).stop 2022-07-20T02:25:18.960 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 -- 172.21.15.120:0/426333117 shutdown_connections 2022-07-20T02:25:18.960 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 --2- 172.21.15.120:0/426333117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f3898004980 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.961 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 -- 172.21.15.120:0/426333117 >> 172.21.15.120:0/426333117 conn(0x7f38ac0b3050 msgr2=0x7f38ac0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.961 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 -- 172.21.15.120:0/426333117 shutdown_connections 2022-07-20T02:25:18.962 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.881+0000 7f38b109a700 1 -- 172.21.15.120:0/426333117 wait complete. 2022-07-20T02:25:18.962 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 Processor -- start 2022-07-20T02:25:18.963 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- start start 2022-07-20T02:25:18.963 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f38ac143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.964 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.964 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38ac0aae40 0x7f38ac063800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:18.965 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0ab550 con 0x7f38ac0aae40 2022-07-20T02:25:18.965 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac065160 con 0x7f3898004500 2022-07-20T02:25:18.966 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0a9220 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.966 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.971 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38abfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f38ac143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.971 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54082/0 (socket says 172.21.15.120:54082) 2022-07-20T02:25:18.972 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 -- 172.21.15.120:0/3601640553 learned_addr learned my addr 172.21.15.120:0/3601640553 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:18.972 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b0899700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38ac0aae40 0x7f38ac063800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:18.973 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 -- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 msgr2=0x7f38ac143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.973 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f38ac143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.973 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 -- 172.21.15.120:0/3601640553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38ac0aae40 msgr2=0x7f38ac063800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.973 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38ac0aae40 0x7f38ac063800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.974 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 -- 172.21.15.120:0/3601640553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac066830 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.974 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38ab7fe700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f38980054d0 tx=0x7f38a0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:18.974 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a0018030 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.974 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0014410 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.975 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac064c50 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.975 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a0014770 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.975 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac065f20 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.975 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f38a0026070 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.976 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f38a001e860 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.976 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.885+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac0a9080 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.976 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.889+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f38ac0a9080 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.977 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.957+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", "id": 3} v 0) v1 -- 0x7f38ac065320 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.977 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38a97fa700 1 -- 172.21.15.120:0/3601640553 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f38a0024060 con 0x7f38ac0aa4d0 2022-07-20T02:25:18.978 INFO:teuthology.orchestra.run.smithi120.stdout:3 2022-07-20T02:25:18.982 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 msgr2=0x7f38ac0aa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:18.982 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f38980054d0 tx=0x7f38a0015000).stop 2022-07-20T02:25:18.982 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 shutdown_connections 2022-07-20T02:25:18.983 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38ac0aae40 0x7f38ac063800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.983 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38ac0aa4d0 0x7f38ac0aa950 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.983 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 --2- 172.21.15.120:0/3601640553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3898004500 0x7f38ac143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:18.983 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 >> 172.21.15.120:0/3601640553 conn(0x7f38ac0b3050 msgr2=0x7f38ac061590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:18.983 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 shutdown_connections 2022-07-20T02:25:18.984 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:18.965+0000 7f38b109a700 1 -- 172.21.15.120:0/3601640553 wait complete. 2022-07-20T02:25:18.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf 4 2022-07-20T02:25:19.084 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 Processor -- start 2022-07-20T02:25:19.092 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 -- start start 2022-07-20T02:25:19.092 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 0x7fb4880b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.093 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.093 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4880aa1e0 con 0x7fb488060950 2022-07-20T02:25:19.094 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.081+0000 7fb48e391700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4880ae110 con 0x7fb4880add30 2022-07-20T02:25:19.094 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880a9f40 con 0x7fb4880af6d0 2022-07-20T02:25:19.095 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.095 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48cb8e700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb488060950 0x7fb488060dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:57402/0 (socket says 172.21.15.120:57402) 2022-07-20T02:25:19.102 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54090/0 (socket says 172.21.15.120:54090) 2022-07-20T02:25:19.103 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48cb8e700 1 -- 172.21.15.120:0/2610330061 learned_addr learned my addr 172.21.15.120:0/2610330061 (peer_addr_for_me v1:172.21.15.120:0/0) 2022-07-20T02:25:19.103 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48d38f700 1 --2- 172.21.15.120:0/2610330061 >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 0x7fb4880b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.103 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 -- 172.21.15.120:0/2610330061 >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 msgr2=0x7fb4880b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:19.103 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb477fff700 1 -- 172.21.15.120:0/2610330061 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050029017 0 0) 0x7fb4880aa1e0 con 0x7fb488060950 2022-07-20T02:25:19.104 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 --2- 172.21.15.120:0/2610330061 >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 0x7fb4880b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.104 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 -- 172.21.15.120:0/2610330061 >> v1:172.21.15.120:6789/0 conn(0x7fb488060950 legacy=0x7fb488060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.104 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 -- 172.21.15.120:0/2610330061 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4880b1110 con 0x7fb4880af6d0 2022-07-20T02:25:19.104 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48d38f700 1 --2- 172.21.15.120:0/2610330061 >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 0x7fb4880b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:19.110 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48db90700 1 --2- 172.21.15.120:0/2610330061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb4800032f0 tx=0x7fb4800091d0).ready entity=mon.2 client_cookie=a31ae361e823532d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.111 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb477fff700 1 -- 172.21.15.120:0/2610330061 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb480009ea0 con 0x7fb4880af6d0 2022-07-20T02:25:19.111 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb477fff700 1 -- 172.21.15.120:0/2610330061 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48000b8a0 con 0x7fb4880af6d0 2022-07-20T02:25:19.112 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb477fff700 1 -- 172.21.15.120:0/2610330061 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb48000ba10 con 0x7fb4880af6d0 2022-07-20T02:25:19.112 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- 172.21.15.120:0/2610330061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 msgr2=0x7fb4880afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.112 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 --2- 172.21.15.120:0/2610330061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb4800032f0 tx=0x7fb4800091d0).stop 2022-07-20T02:25:19.113 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- 172.21.15.120:0/2610330061 shutdown_connections 2022-07-20T02:25:19.113 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 --2- 172.21.15.120:0/2610330061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4880af6d0 0x7fb4880afb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.114 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 --2- 172.21.15.120:0/2610330061 >> v2:172.21.15.26:3300/0 conn(0x7fb4880add30 0x7fb4880b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.114 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- 172.21.15.120:0/2610330061 >> 172.21.15.120:0/2610330061 conn(0x7fb4880b3050 msgr2=0x7fb4880b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.115 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- 172.21.15.120:0/2610330061 shutdown_connections 2022-07-20T02:25:19.115 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- 172.21.15.120:0/2610330061 wait complete. 2022-07-20T02:25:19.115 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 Processor -- start 2022-07-20T02:25:19.116 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.085+0000 7fb48e391700 1 -- start start 2022-07-20T02:25:19.116 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.117 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4880add30 0x7fb48813eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.117 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 0x7fb48813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.117 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488064a70 con 0x7fb4880add30 2022-07-20T02:25:19.118 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48813daf0 con 0x7fb4880af6d0 2022-07-20T02:25:19.118 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48813f490 con 0x7fb488060950 2022-07-20T02:25:19.119 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48d38f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.119 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48d38f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54092/0 (socket says 172.21.15.120:54092) 2022-07-20T02:25:19.120 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48d38f700 1 -- 172.21.15.120:0/2718875566 learned_addr learned my addr 172.21.15.120:0/2718875566 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.120 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 0x7fb48813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.120 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48cb8e700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4880add30 0x7fb48813eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.121 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 -- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 msgr2=0x7fb488140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.121 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.122 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 -- 172.21.15.120:0/2718875566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4880add30 msgr2=0x7fb48813eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.122 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4880add30 0x7fb48813eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.123 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 -- 172.21.15.120:0/2718875566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480009470 con 0x7fb4880af6d0 2022-07-20T02:25:19.123 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48d38f700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:19.124 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48db90700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 0x7fb48813efa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb4800032f0 tx=0x7fb480016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.124 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb480009ea0 con 0x7fb4880af6d0 2022-07-20T02:25:19.124 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4800205e0 con 0x7fb4880af6d0 2022-07-20T02:25:19.125 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4881417d0 con 0x7fb4880af6d0 2022-07-20T02:25:19.125 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb480020940 con 0x7fb4880af6d0 2022-07-20T02:25:19.126 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48813dd20 con 0x7fb4880af6d0 2022-07-20T02:25:19.126 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fb48001e070 con 0x7fb4880af6d0 2022-07-20T02:25:19.127 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7fb480029980 con 0x7fb4880af6d0 2022-07-20T02:25:19.127 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.089+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4880659d0 con 0x7fb4880af6d0 2022-07-20T02:25:19.128 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.093+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fb48001b070 con 0x7fb4880af6d0 2022-07-20T02:25:19.157 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.161+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", "id": 4} v 0) v1 -- 0x7fb4880b13b0 con 0x7fb4880af6d0 2022-07-20T02:25:19.165 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb4767fc700 1 -- 172.21.15.120:0/2718875566 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb480015020 con 0x7fb4880af6d0 2022-07-20T02:25:19.166 INFO:teuthology.orchestra.run.smithi120.stdout:4 2022-07-20T02:25:19.167 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 msgr2=0x7fb48813efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.168 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 0x7fb48813efa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb4800032f0 tx=0x7fb480016000).stop 2022-07-20T02:25:19.168 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 shutdown_connections 2022-07-20T02:25:19.168 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4880add30 0x7fb48813eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.169 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4880af6d0 0x7fb48813efa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.169 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 --2- 172.21.15.120:0/2718875566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb488060950 0x7fb488140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.170 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 >> 172.21.15.120:0/2718875566 conn(0x7fb4880b3050 msgr2=0x7fb488143920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.170 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 shutdown_connections 2022-07-20T02:25:19.170 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.169+0000 7fb48e391700 1 -- 172.21.15.120:0/2718875566 wait complete. 2022-07-20T02:25:19.189 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3 5 2022-07-20T02:25:19.293 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.293+0000 7f2550818700 1 Processor -- start 2022-07-20T02:25:19.293 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.293+0000 7f2550818700 1 -- start start 2022-07-20T02:25:19.302 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f254c147000 0x7f254c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.303 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c1474e0 con 0x7f254c05cfd0 2022-07-20T02:25:19.304 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c0b4e40 con 0x7f254c060f00 2022-07-20T02:25:19.304 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c05d3b0 con 0x7f254c147000 2022-07-20T02:25:19.304 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f254c147000 0x7f254c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.305 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f254c147000 0x7f254c143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44110/0 (socket says 172.21.15.120:44110) 2022-07-20T02:25:19.305 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 -- 172.21.15.120:0/2655883976 learned_addr learned my addr 172.21.15.120:0/2655883976 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.306 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 -- 172.21.15.120:0/2655883976 >> v1:172.21.15.120:6789/0 conn(0x7f254c060f00 legacy=0x7f254c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.306 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 -- 172.21.15.120:0/2655883976 >> v1:172.21.15.26:6789/0 conn(0x7f254c05cfd0 legacy=0x7f254c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:19.306 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 -- 172.21.15.120:0/2655883976 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c1431e0 con 0x7f254c147000 2022-07-20T02:25:19.307 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254affd700 1 --2- 172.21.15.120:0/2655883976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c147000 0x7f254c143bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f254c068660 tx=0x7f2540006eb0).ready entity=mon.1 client_cookie=16311e280ca187fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.307 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254a7fc700 1 -- 172.21.15.120:0/2655883976 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f254000d610 con 0x7f254c147000 2022-07-20T02:25:19.308 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254a7fc700 1 -- 172.21.15.120:0/2655883976 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2540000e90 con 0x7f254c147000 2022-07-20T02:25:19.308 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f254a7fc700 1 -- 172.21.15.120:0/2655883976 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f254000d960 con 0x7f254c147000 2022-07-20T02:25:19.309 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- 172.21.15.120:0/2655883976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c147000 msgr2=0x7f254c143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.309 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 --2- 172.21.15.120:0/2655883976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c147000 0x7f254c143bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f254c068660 tx=0x7f2540006eb0).stop 2022-07-20T02:25:19.310 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- 172.21.15.120:0/2655883976 shutdown_connections 2022-07-20T02:25:19.310 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 --2- 172.21.15.120:0/2655883976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c147000 0x7f254c143bc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.310 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- 172.21.15.120:0/2655883976 >> 172.21.15.120:0/2655883976 conn(0x7f254c0620c0 msgr2=0x7f254c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.311 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- 172.21.15.120:0/2655883976 shutdown_connections 2022-07-20T02:25:19.311 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 -- 172.21.15.120:0/2655883976 wait complete. 2022-07-20T02:25:19.312 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.297+0000 7f2550818700 1 Processor -- start 2022-07-20T02:25:19.319 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- start start 2022-07-20T02:25:19.319 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.319 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 0x7f254c0ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.320 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f254c0ad690 0x7f254c0aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.320 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c146060 con 0x7f254c147000 2022-07-20T02:25:19.321 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1565f0 con 0x7f254c060f00 2022-07-20T02:25:19.321 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0ae820 con 0x7f254c0ad690 2022-07-20T02:25:19.322 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.323 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44114/0 (socket says 172.21.15.120:44114) 2022-07-20T02:25:19.323 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 -- 172.21.15.120:0/391843246 learned_addr learned my addr 172.21.15.120:0/391843246 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.324 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254bfff700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f254c0ad690 0x7f254c0aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.325 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254affd700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 0x7f254c0ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.325 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 -- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f254c0ad690 msgr2=0x7f254c0aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.325 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f254c0ad690 0x7f254c0aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.326 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 -- 172.21.15.120:0/391843246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 msgr2=0x7f254c0ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.326 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 0x7f254c0ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.326 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 -- 172.21.15.120:0/391843246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254000b1e0 con 0x7f254c060f00 2022-07-20T02:25:19.326 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254affd700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 0x7f254c0ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:19.327 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f254b7fe700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f254c0ae9b0 tx=0x7f253c006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.327 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253c009e10 con 0x7f254c060f00 2022-07-20T02:25:19.327 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c002320 con 0x7f254c060f00 2022-07-20T02:25:19.327 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c156850 con 0x7f254c060f00 2022-07-20T02:25:19.328 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253c002680 con 0x7f254c060f00 2022-07-20T02:25:19.328 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c146290 con 0x7f254c060f00 2022-07-20T02:25:19.329 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f253c00f030 con 0x7f254c060f00 2022-07-20T02:25:19.329 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f253c0129c0 con 0x7f254c060f00 2022-07-20T02:25:19.329 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.301+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c143480 con 0x7f254c060f00 2022-07-20T02:25:19.330 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.305+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f253c012bd0 con 0x7f254c060f00 2022-07-20T02:25:19.375 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.373+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", "id": 5} v 0) v1 -- 0x7f254c0574e0 con 0x7f254c060f00 2022-07-20T02:25:19.375 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.373+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7f253c012d70 con 0x7f254c060f00 2022-07-20T02:25:19.381 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2548ff9700 1 -- 172.21.15.120:0/391843246 <== mon.1 v2:172.21.15.120:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f253c01b3e0 con 0x7f254c060f00 2022-07-20T02:25:19.387 INFO:teuthology.orchestra.run.smithi120.stdout:5 2022-07-20T02:25:19.389 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 msgr2=0x7f254c142460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.389 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f254c0ae9b0 tx=0x7f253c006d70).stop 2022-07-20T02:25:19.390 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 shutdown_connections 2022-07-20T02:25:19.390 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f254c147000 0x7f254c0ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.391 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f254c0ad690 0x7f254c0aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.391 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 --2- 172.21.15.120:0/391843246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f254c060f00 0x7f254c142460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.392 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 >> 172.21.15.120:0/391843246 conn(0x7f254c05cfd0 msgr2=0x7f254c0b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.392 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 shutdown_connections 2022-07-20T02:25:19.393 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.385+0000 7f2550818700 1 -- 172.21.15.120:0/391843246 wait complete. 2022-07-20T02:25:19.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 87a7cb28-e512-4d57-910f-5b4b8bc96eef 6 2022-07-20T02:25:19.500 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 Processor -- start 2022-07-20T02:25:19.500 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 -- start start 2022-07-20T02:25:19.517 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.518 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ea8149ce0 0x7f5ea814a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.519 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea814b470 con 0x7f5ea80b4b80 2022-07-20T02:25:19.519 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80b4fc0 con 0x7f5ea8149ce0 2022-07-20T02:25:19.519 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ead5f8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8064650 con 0x7f5ea80b7d80 2022-07-20T02:25:19.520 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.520 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54110/0 (socket says 172.21.15.120:54110) 2022-07-20T02:25:19.521 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5ea7fff700 1 -- 172.21.15.120:0/590378069 learned_addr learned my addr 172.21.15.120:0/590378069 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.521 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.501+0000 7f5eacdf7700 1 --2- 172.21.15.120:0/590378069 >> v2:172.21.15.120:3300/0 conn(0x7f5ea8149ce0 0x7f5ea814a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.522 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5eacdf7700 1 -- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 msgr2=0x7f5ea80aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:19.522 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5eacdf7700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.523 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5eacdf7700 1 -- 172.21.15.120:0/590378069 >> v1:172.21.15.26:6789/0 conn(0x7f5ea80b4b80 legacy=0x7f5ea814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.523 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5eacdf7700 1 -- 172.21.15.120:0/590378069 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80b41a0 con 0x7f5ea8149ce0 2022-07-20T02:25:19.524 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ea7fff700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:19.524 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5eacdf7700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea8149ce0 0x7f5ea814a130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5ea00032f0 tx=0x7f5ea00091d0).ready entity=mon.1 client_cookie=b070d6fe3ea5233e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.524 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ea6ffd700 1 -- 172.21.15.120:0/590378069 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ea000b6c0 con 0x7f5ea8149ce0 2022-07-20T02:25:19.525 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ea6ffd700 1 -- 172.21.15.120:0/590378069 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea000b820 con 0x7f5ea8149ce0 2022-07-20T02:25:19.525 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ea6ffd700 1 -- 172.21.15.120:0/590378069 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ea000b990 con 0x7f5ea8149ce0 2022-07-20T02:25:19.526 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea8149ce0 msgr2=0x7f5ea814a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.526 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea8149ce0 0x7f5ea814a130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5ea00032f0 tx=0x7f5ea00091d0).stop 2022-07-20T02:25:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- 172.21.15.120:0/590378069 shutdown_connections 2022-07-20T02:25:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea8149ce0 0x7f5ea814a130 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/590378069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea80aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- 172.21.15.120:0/590378069 >> 172.21.15.120:0/590378069 conn(0x7f5ea805f840 msgr2=0x7f5ea805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- 172.21.15.120:0/590378069 shutdown_connections 2022-07-20T02:25:19.528 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- 172.21.15.120:0/590378069 wait complete. 2022-07-20T02:25:19.528 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 Processor -- start 2022-07-20T02:25:19.529 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.505+0000 7f5ead5f8700 1 -- start start 2022-07-20T02:25:19.529 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.529 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.530 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 0x7f5ea809f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.530 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80b1aa0 con 0x7f5ea8149ce0 2022-07-20T02:25:19.530 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80b13d0 con 0x7f5ea80b4b80 2022-07-20T02:25:19.530 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81571f0 con 0x7f5ea80b7d80 2022-07-20T02:25:19.531 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.531 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.531 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54118/0 (socket says 172.21.15.120:54118) 2022-07-20T02:25:19.531 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea77fe700 1 -- 172.21.15.120:0/3194539420 learned_addr learned my addr 172.21.15.120:0/3194539420 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.532 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44126/0 (socket says 172.21.15.120:44126) 2022-07-20T02:25:19.532 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5eacdf7700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 0x7f5ea809f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.532 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 -- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 msgr2=0x7f5ea809ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.532 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.532 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 -- 172.21.15.120:0/3194539420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 msgr2=0x7f5ea809f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.533 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 0x7f5ea809f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.533 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 -- 172.21.15.120:0/3194539420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9800b1e0 con 0x7f5ea80b4b80 2022-07-20T02:25:19.533 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5eacdf7700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 0x7f5ea809f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:19.533 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea77fe700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:19.534 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea7fff700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5ea80b1be0 tx=0x7f5e98006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.534 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e98012600 con 0x7f5ea80b4b80 2022-07-20T02:25:19.534 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98012760 con 0x7f5ea80b4b80 2022-07-20T02:25:19.534 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea0009470 con 0x7f5ea80b4b80 2022-07-20T02:25:19.535 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e98012ac0 con 0x7f5ea80b4b80 2022-07-20T02:25:19.535 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea80b1600 con 0x7f5ea80b4b80 2022-07-20T02:25:19.535 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7f5e98012c20 con 0x7f5ea80b4b80 2022-07-20T02:25:19.535 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f5e9803a8d0 con 0x7f5ea80b4b80 2022-07-20T02:25:19.535 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.509+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea805db10 con 0x7f5ea80b4b80 2022-07-20T02:25:19.536 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.513+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f5ea805db10 con 0x7f5ea80b4b80 2022-07-20T02:25:19.581 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.585+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", "id": 6} v 0) v1 -- 0x7f5ea805db10 con 0x7f5ea80b4b80 2022-07-20T02:25:19.588 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.589+0000 7f5ea57fa700 1 -- 172.21.15.120:0/3194539420 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5ea805db10 con 0x7f5ea80b4b80 2022-07-20T02:25:19.588 INFO:teuthology.orchestra.run.smithi120.stdout:6 2022-07-20T02:25:19.590 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 msgr2=0x7f5ea80b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.590 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5ea80b1be0 tx=0x7f5e98006eb0).stop 2022-07-20T02:25:19.591 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 shutdown_connections 2022-07-20T02:25:19.591 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea8149ce0 0x7f5ea809f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.592 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea80b7d80 0x7f5ea809ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.592 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 --2- 172.21.15.120:0/3194539420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea80b4b80 0x7f5ea80b00b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.592 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 >> 172.21.15.120:0/3194539420 conn(0x7f5ea805f840 msgr2=0x7f5ea80b5d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.594 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 shutdown_connections 2022-07-20T02:25:19.595 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.593+0000 7f5ead5f8700 1 -- 172.21.15.120:0/3194539420 wait complete. 2022-07-20T02:25:19.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new b1607f07-e4fb-42df-84d9-b5c855babf2e 7 2022-07-20T02:25:19.725 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 Processor -- start 2022-07-20T02:25:19.726 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 -- start start 2022-07-20T02:25:19.726 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 0x7ff0c40b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.727 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.727 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40b2e40 con 0x7ff0c40b7d80 2022-07-20T02:25:19.727 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40b4c20 con 0x7ff0c40b3060 2022-07-20T02:25:19.727 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40617a0 con 0x7ff0c4064220 2022-07-20T02:25:19.727 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.728 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54126/0 (socket says 172.21.15.120:54126) 2022-07-20T02:25:19.728 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c37fe700 1 -- 172.21.15.120:0/259559809 learned_addr learned my addr 172.21.15.120:0/259559809 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.728 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.709+0000 7ff0c27fc700 1 --2- 172.21.15.120:0/259559809 >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 0x7ff0c40b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.728 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 -- 172.21.15.120:0/259559809 >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 msgr2=0x7ff0c40b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:19.729 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 --2- 172.21.15.120:0/259559809 >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 0x7ff0c40b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.730 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 -- 172.21.15.120:0/259559809 >> v1:172.21.15.26:6789/0 conn(0x7ff0c40b7d80 legacy=0x7ff0c40b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.730 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 -- 172.21.15.120:0/259559809 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4060b20 con 0x7ff0c4064220 2022-07-20T02:25:19.730 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 --2- 172.21.15.120:0/259559809 >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 0x7ff0c40b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:19.731 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 --2- 172.21.15.120:0/259559809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff0bc0032f0 tx=0x7ff0bc009290).ready entity=mon.2 client_cookie=78bb8a6e5e0e5387 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.731 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c1ffb700 1 -- 172.21.15.120:0/259559809 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc00b8a0 con 0x7ff0c4064220 2022-07-20T02:25:19.731 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c1ffb700 1 -- 172.21.15.120:0/259559809 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc00ba00 con 0x7ff0c4064220 2022-07-20T02:25:19.732 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c1ffb700 1 -- 172.21.15.120:0/259559809 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc00bb70 con 0x7ff0c4064220 2022-07-20T02:25:19.732 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- 172.21.15.120:0/259559809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 msgr2=0x7ff0c41434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.732 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/259559809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff0bc0032f0 tx=0x7ff0bc009290).stop 2022-07-20T02:25:19.733 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- 172.21.15.120:0/259559809 shutdown_connections 2022-07-20T02:25:19.733 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/259559809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c4064220 0x7ff0c41434f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.733 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/259559809 >> v2:172.21.15.26:3300/0 conn(0x7ff0c40b3060 0x7ff0c40b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.733 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- 172.21.15.120:0/259559809 >> 172.21.15.120:0/259559809 conn(0x7ff0c405f840 msgr2=0x7ff0c405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.734 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- 172.21.15.120:0/259559809 shutdown_connections 2022-07-20T02:25:19.734 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- 172.21.15.120:0/259559809 wait complete. 2022-07-20T02:25:19.734 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 Processor -- start 2022-07-20T02:25:19.735 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- start start 2022-07-20T02:25:19.735 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 0x7ff0c4145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.735 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.735 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0c40b7d80 0x7ff0c41562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:19.736 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40ab5f0 con 0x7ff0c40b7d80 2022-07-20T02:25:19.736 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40ab890 con 0x7ff0c4064220 2022-07-20T02:25:19.737 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40abb30 con 0x7ff0c40b3060 2022-07-20T02:25:19.737 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.737 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54130/0 (socket says 172.21.15.120:54130) 2022-07-20T02:25:19.738 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c2ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 0x7ff0c4145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.738 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 -- 172.21.15.120:0/1137848036 learned_addr learned my addr 172.21.15.120:0/1137848036 (peer_addr_for_me v2:172.21.15.120:0/0) 2022-07-20T02:25:19.738 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c37fe700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0c40b7d80 0x7ff0c41562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:19.738 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 -- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 msgr2=0x7ff0c4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.739 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 0x7ff0c4145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.739 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 -- 172.21.15.120:0/1137848036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0c40b7d80 msgr2=0x7ff0c41562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.740 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0c40b7d80 0x7ff0c41562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.741 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c27fc700 1 -- 172.21.15.120:0/1137848036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc009530 con 0x7ff0c40b3060 2022-07-20T02:25:19.741 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.713+0000 7ff0c2ffd700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 0x7ff0c4145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:19.741 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0c27fc700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff0b8006940 tx=0x7ff0b8008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:19.741 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0b8011780 con 0x7ff0c40b3060 2022-07-20T02:25:19.742 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b80118e0 con 0x7ff0c40b3060 2022-07-20T02:25:19.743 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0b8011c40 con 0x7ff0c40b3060 2022-07-20T02:25:19.743 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40abdd0 con 0x7ff0c40b3060 2022-07-20T02:25:19.744 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c41494e0 con 0x7ff0c40b3060 2022-07-20T02:25:19.744 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7ff0b801c460 con 0x7ff0c40b3060 2022-07-20T02:25:19.744 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1935+0+0 (secure 0 0 0) 0x7ff0b803a750 con 0x7ff0c40b3060 2022-07-20T02:25:19.745 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c40abf60 con 0x7ff0c40b3060 2022-07-20T02:25:19.745 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.717+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7ff0b801a360 con 0x7ff0c40b3060 2022-07-20T02:25:19.788 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.785+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", "id": 7} v 0) v1 -- 0x7ff0c4060dc0 con 0x7ff0c40b3060 2022-07-20T02:25:19.793 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0b3fff700 1 -- 172.21.15.120:0/1137848036 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff0b802a0b0 con 0x7ff0c40b3060 2022-07-20T02:25:19.793 INFO:teuthology.orchestra.run.smithi120.stdout:7 2022-07-20T02:25:19.797 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 msgr2=0x7ff0c4155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:19.798 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff0b8006940 tx=0x7ff0b8008040).stop 2022-07-20T02:25:19.799 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 shutdown_connections 2022-07-20T02:25:19.799 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0c40b7d80 0x7ff0c41562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.800 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0c40b3060 0x7ff0c4155de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.800 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 --2- 172.21.15.120:0/1137848036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0c4064220 0x7ff0c4145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:19.801 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 >> 172.21.15.120:0/1137848036 conn(0x7ff0c405f840 msgr2=0x7ff0c4065390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:19.801 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 shutdown_connections 2022-07-20T02:25:19.802 INFO:teuthology.orchestra.run.smithi120.stderr:2022-07-20T02:25:19.797+0000 7ff0c3fff700 1 -- 172.21.15.120:0/1137848036 wait complete. 2022-07-20T02:25:19.807 INFO:tasks.ceph.osd.0:Restarting daemon 2022-07-20T02:25:19.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-07-20T02:25:19.814 INFO:tasks.ceph.osd.0:Started 2022-07-20T02:25:19.815 INFO:tasks.ceph.osd.1:Restarting daemon 2022-07-20T02:25:19.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-07-20T02:25:19.821 INFO:tasks.ceph.osd.1:Started 2022-07-20T02:25:19.822 INFO:tasks.ceph.osd.2:Restarting daemon 2022-07-20T02:25:19.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-07-20T02:25:19.825 INFO:tasks.ceph.osd.2:Started 2022-07-20T02:25:19.826 INFO:tasks.ceph.osd.3:Restarting daemon 2022-07-20T02:25:19.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-07-20T02:25:19.830 INFO:tasks.ceph.osd.3:Started 2022-07-20T02:25:19.832 INFO:tasks.ceph.osd.4:Restarting daemon 2022-07-20T02:25:19.833 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-07-20T02:25:19.837 INFO:tasks.ceph.osd.4:Started 2022-07-20T02:25:19.838 INFO:tasks.ceph.osd.5:Restarting daemon 2022-07-20T02:25:19.838 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-07-20T02:25:19.842 INFO:tasks.ceph.osd.5:Started 2022-07-20T02:25:19.843 INFO:tasks.ceph.osd.6:Restarting daemon 2022-07-20T02:25:19.845 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-07-20T02:25:19.850 INFO:tasks.ceph.osd.6:Started 2022-07-20T02:25:19.851 INFO:tasks.ceph.osd.7:Restarting daemon 2022-07-20T02:25:19.852 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-07-20T02:25:19.855 INFO:tasks.ceph.osd.7:Started 2022-07-20T02:25:19.856 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-20T02:25:20.202 INFO:tasks.ceph.osd.0.smithi026.stderr:2022-07-20T02:25:20.199+0000 7f328e7a0d80 -1 Falling back to public interface 2022-07-20T02:25:20.435 INFO:tasks.ceph.osd.5.smithi120.stderr:2022-07-20T02:25:20.437+0000 7fd3cea9fd80 -1 Falling back to public interface 2022-07-20T02:25:20.454 INFO:tasks.ceph.osd.1.smithi026.stderr:2022-07-20T02:25:20.451+0000 7f0ac007fd80 -1 Falling back to public interface 2022-07-20T02:25:20.464 INFO:tasks.ceph.osd.6.smithi120.stderr:2022-07-20T02:25:20.465+0000 7f759787ed80 -1 Falling back to public interface 2022-07-20T02:25:20.466 INFO:tasks.ceph.osd.4.smithi120.stderr:2022-07-20T02:25:20.469+0000 7f023761ad80 -1 Falling back to public interface 2022-07-20T02:25:20.481 INFO:tasks.ceph.osd.2.smithi026.stderr:2022-07-20T02:25:20.479+0000 7fc7f8fadd80 -1 Falling back to public interface 2022-07-20T02:25:20.486 INFO:tasks.ceph.osd.7.smithi120.stderr:2022-07-20T02:25:20.489+0000 7f382cfbcd80 -1 Falling back to public interface 2022-07-20T02:25:20.505 INFO:tasks.ceph.osd.3.smithi026.stderr:2022-07-20T02:25:20.503+0000 7fe40e97ed80 -1 Falling back to public interface 2022-07-20T02:25:20.666 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:20.781 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.755+0000 7f0b51b44700 1 Processor -- start 2022-07-20T02:25:20.782 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.755+0000 7f0b51b44700 1 -- start start 2022-07-20T02:25:20.782 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b4c098de0 0x7f0b4c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.782 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0b4c147590 0x7f0b4c149980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.783 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c062dd0 con 0x7f0b4c0b3190 2022-07-20T02:25:20.783 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c0b3570 con 0x7f0b4c147590 2022-07-20T02:25:20.784 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c062b30 con 0x7f0b4c098de0 2022-07-20T02:25:20.784 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0b4c147590 0x7f0b4c149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.785 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0b4c147590 0x7f0b4c149980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34456/0 (socket says 172.21.15.26:34456) 2022-07-20T02:25:20.785 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 -- 172.21.15.26:0/2550034803 learned_addr learned my addr 172.21.15.26:0/2550034803 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:20.785 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b4bfff700 1 --2- 172.21.15.26:0/2550034803 >> v2:172.21.15.120:3300/0 conn(0x7f0b4c098de0 0x7f0b4c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.785 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 -- 172.21.15.26:0/2550034803 >> v2:172.21.15.120:3300/0 conn(0x7f0b4c098de0 msgr2=0x7f0b4c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:20.786 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 --2- 172.21.15.26:0/2550034803 >> v2:172.21.15.120:3300/0 conn(0x7f0b4c098de0 0x7f0b4c09ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.786 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 -- 172.21.15.26:0/2550034803 >> v1:172.21.15.26:6789/0 conn(0x7f0b4c0b3190 legacy=0x7f0b4c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.786 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 -- 172.21.15.26:0/2550034803 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c0b1eb0 con 0x7f0b4c147590 2022-07-20T02:25:20.786 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51343700 1 --2- 172.21.15.26:0/2550034803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c149980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f0b440032f0 tx=0x7f0b44009290).ready entity=mon.0 client_cookie=16c8542398f6d5c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:20.787 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b4b7fe700 1 -- 172.21.15.26:0/2550034803 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b4400ba60 con 0x7f0b4c147590 2022-07-20T02:25:20.787 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b4b7fe700 1 -- 172.21.15.26:0/2550034803 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4400bbc0 con 0x7f0b4c147590 2022-07-20T02:25:20.787 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b4b7fe700 1 -- 172.21.15.26:0/2550034803 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b44010470 con 0x7f0b4c147590 2022-07-20T02:25:20.787 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- 172.21.15.26:0/2550034803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 msgr2=0x7f0b4c149980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.788 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2550034803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c149980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f0b440032f0 tx=0x7f0b44009290).stop 2022-07-20T02:25:20.788 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- 172.21.15.26:0/2550034803 shutdown_connections 2022-07-20T02:25:20.788 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2550034803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c149980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.788 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2550034803 >> v2:172.21.15.120:3300/0 conn(0x7f0b4c098de0 0x7f0b4c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.788 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- 172.21.15.26:0/2550034803 >> 172.21.15.26:0/2550034803 conn(0x7f0b4c063100 msgr2=0x7f0b4c0634e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:20.789 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- 172.21.15.26:0/2550034803 shutdown_connections 2022-07-20T02:25:20.789 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- 172.21.15.26:0/2550034803 wait complete. 2022-07-20T02:25:20.789 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 Processor -- start 2022-07-20T02:25:20.789 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.759+0000 7f0b51b44700 1 -- start start 2022-07-20T02:25:20.790 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 0x7f0b4c063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.790 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b4c0b3190 0x7f0b4c0903d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.790 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.791 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0908c0 con 0x7f0b4c147590 2022-07-20T02:25:20.791 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c14efb0 con 0x7f0b4c098de0 2022-07-20T02:25:20.791 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c090af0 con 0x7f0b4c0b3190 2022-07-20T02:25:20.791 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.791 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34462/0 (socket says 172.21.15.26:34462) 2022-07-20T02:25:20.792 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 -- 172.21.15.26:0/2901011125 learned_addr learned my addr 172.21.15.26:0/2901011125 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:20.792 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b50b42700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 0x7f0b4c063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.792 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b4bfff700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b4c0b3190 0x7f0b4c0903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.792 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 -- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b4c0b3190 msgr2=0x7f0b4c0903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.793 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b4c0b3190 0x7f0b4c0903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.793 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 -- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 msgr2=0x7f0b4c063e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.793 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 0x7f0b4c063e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.793 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 -- 172.21.15.26:0/2901011125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44009530 con 0x7f0b4c147590 2022-07-20T02:25:20.794 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b50b42700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 0x7f0b4c063e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:20.804 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51343700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f0b440032f0 tx=0x7f0b44018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:20.804 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b4401f070 con 0x7f0b4c147590 2022-07-20T02:25:20.805 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b440185f0 con 0x7f0b4c147590 2022-07-20T02:25:20.805 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c14ecd0 con 0x7f0b4c147590 2022-07-20T02:25:20.805 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b44018950 con 0x7f0b4c147590 2022-07-20T02:25:20.806 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c067390 con 0x7f0b4c147590 2022-07-20T02:25:20.806 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f0b4400f3d0 con 0x7f0b4c147590 2022-07-20T02:25:20.807 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.763+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c14f1b0 con 0x7f0b4c147590 2022-07-20T02:25:20.807 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.767+0000 7f0b49ffb700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 0x7f0b2c055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:20.808 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.767+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2074+0+0 (secure 0 0 0) 0x7f0b4406df20 con 0x7f0b4c147590 2022-07-20T02:25:20.808 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.767+0000 7f0b50b42700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 0x7f0b2c055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:20.809 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.767+0000 7f0b50b42700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 0x7f0b2c055e00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0b4000b000 tx=0x7f0b40009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:20.817 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.771+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b44045e10 con 0x7f0b4c147590 2022-07-20T02:25:20.911 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.907+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b4c090cb0 con 0x7f0b4c147590 2022-07-20T02:25:20.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.911+0000 7f0b49ffb700 1 -- 172.21.15.26:0/2901011125 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+8275 (secure 0 0 0) 0x7f0b44049430 con 0x7f0b4c147590 2022-07-20T02:25:20.916 INFO:teuthology.misc.health.smithi026.stdout: 2022-07-20T02:25:20.917 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:19.786630+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:20.920 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.911+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 msgr2=0x7f0b2c055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.920 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.911+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 0x7f0b2c055e00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0b4000b000 tx=0x7f0b40009040).stop 2022-07-20T02:25:20.921 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.911+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 msgr2=0x7f0b4c066ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:20.921 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.911+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f0b440032f0 tx=0x7f0b44018000).stop 2022-07-20T02:25:20.924 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 shutdown_connections 2022-07-20T02:25:20.924 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b4c147590 0x7f0b4c066ea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.925 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0b2c053940 0x7f0b2c055e00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.927 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b4c0b3190 0x7f0b4c0903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.928 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 --2- 172.21.15.26:0/2901011125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b4c098de0 0x7f0b4c063e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:20.928 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 >> 172.21.15.26:0/2901011125 conn(0x7f0b4c063100 msgr2=0x7f0b4c05f780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:20.928 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 shutdown_connections 2022-07-20T02:25:20.929 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:20.915+0000 7f0b51b44700 1 -- 172.21.15.26:0/2901011125 wait complete. 2022-07-20T02:25:20.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-07-20T02:25:21.398 INFO:tasks.ceph.osd.0.smithi026.stderr:2022-07-20T02:25:21.395+0000 7f328e7a0d80 -1 osd.0 0 log_to_monitors {default=true} 2022-07-20T02:25:21.570 INFO:tasks.ceph.osd.5.smithi120.stderr:2022-07-20T02:25:21.569+0000 7fd3cea9fd80 -1 osd.5 0 log_to_monitors {default=true} 2022-07-20T02:25:21.654 INFO:tasks.ceph.osd.1.smithi026.stderr:2022-07-20T02:25:21.647+0000 7f0ac007fd80 -1 osd.1 0 log_to_monitors {default=true} 2022-07-20T02:25:21.690 INFO:tasks.ceph.osd.4.smithi120.stderr:2022-07-20T02:25:21.685+0000 7f023761ad80 -1 osd.4 0 log_to_monitors {default=true} 2022-07-20T02:25:21.692 INFO:tasks.ceph.osd.7.smithi120.stderr:2022-07-20T02:25:21.693+0000 7f382cfbcd80 -1 osd.7 0 log_to_monitors {default=true} 2022-07-20T02:25:21.699 INFO:tasks.ceph.osd.2.smithi026.stderr:2022-07-20T02:25:21.691+0000 7fc7f8fadd80 -1 osd.2 0 log_to_monitors {default=true} 2022-07-20T02:25:21.711 INFO:tasks.ceph.osd.3.smithi026.stderr:2022-07-20T02:25:21.707+0000 7fe40e97ed80 -1 osd.3 0 log_to_monitors {default=true} 2022-07-20T02:25:21.920 INFO:tasks.ceph.osd.6.smithi120.stderr:2022-07-20T02:25:21.921+0000 7f759787ed80 -1 osd.6 0 log_to_monitors {default=true} 2022-07-20T02:25:23.300 INFO:tasks.ceph.osd.3.smithi026.stderr:2022-07-20T02:25:23.295+0000 7fe400a27700 -1 osd.3 0 waiting for initial osdmap 2022-07-20T02:25:23.301 INFO:tasks.ceph.osd.2.smithi026.stderr:2022-07-20T02:25:23.295+0000 7fc7eb056700 -1 osd.2 0 waiting for initial osdmap 2022-07-20T02:25:23.302 INFO:tasks.ceph.osd.1.smithi026.stderr:2022-07-20T02:25:23.299+0000 7f0ab2128700 -1 osd.1 0 waiting for initial osdmap 2022-07-20T02:25:23.303 INFO:tasks.ceph.osd.5.smithi120.stderr:2022-07-20T02:25:23.301+0000 7fd3c0b48700 -1 osd.5 0 waiting for initial osdmap 2022-07-20T02:25:23.304 INFO:tasks.ceph.osd.0.smithi026.stderr:2022-07-20T02:25:23.299+0000 7f3280849700 -1 osd.0 0 waiting for initial osdmap 2022-07-20T02:25:23.304 INFO:tasks.ceph.osd.7.smithi120.stderr:2022-07-20T02:25:23.301+0000 7f381f065700 -1 osd.7 0 waiting for initial osdmap 2022-07-20T02:25:23.305 INFO:tasks.ceph.osd.6.smithi120.stderr:2022-07-20T02:25:23.305+0000 7f7589927700 -1 osd.6 0 waiting for initial osdmap 2022-07-20T02:25:23.306 INFO:tasks.ceph.osd.4.smithi120.stderr:2022-07-20T02:25:23.305+0000 7f02296c3700 -1 osd.4 0 waiting for initial osdmap 2022-07-20T02:25:23.307 INFO:tasks.ceph.osd.5.smithi120.stderr:2022-07-20T02:25:23.309+0000 7fd3c7fc3700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.309 INFO:tasks.ceph.osd.3.smithi026.stderr:2022-07-20T02:25:23.303+0000 7fe407ea2700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.310 INFO:tasks.ceph.osd.4.smithi120.stderr:2022-07-20T02:25:23.309+0000 7f0230b3e700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.311 INFO:tasks.ceph.osd.6.smithi120.stderr:2022-07-20T02:25:23.309+0000 7f7590da2700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.312 INFO:tasks.ceph.osd.7.smithi120.stderr:2022-07-20T02:25:23.313+0000 7f38264e0700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.313 INFO:tasks.ceph.osd.1.smithi026.stderr:2022-07-20T02:25:23.307+0000 7f0ab95a3700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.315 INFO:tasks.ceph.osd.2.smithi026.stderr:2022-07-20T02:25:23.307+0000 7fc7f24d1700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:23.317 INFO:tasks.ceph.osd.0.smithi026.stderr:2022-07-20T02:25:23.311+0000 7f3287cc4700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T02:25:27.759 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:27.882 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 Processor -- start 2022-07-20T02:25:27.882 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 -- start start 2022-07-20T02:25:27.883 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55cc061640 0x7f55cc143c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.883 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f55cc0b5b50 0x7f55cc0b9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.883 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc146690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.883 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0b97d0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.884 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0b68f0 con 0x7f55cc061640 2022-07-20T02:25:27.884 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d22de700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0b6b90 con 0x7f55cc0ba120 2022-07-20T02:25:27.888 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d0adb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f55cc0b5b50 0x7f55cc0b9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.888 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d0adb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f55cc0b5b50 0x7f55cc0b9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34680/0 (socket says 172.21.15.26:34680) 2022-07-20T02:25:27.888 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d0adb700 1 -- 172.21.15.26:0/3407070988 learned_addr learned my addr 172.21.15.26:0/3407070988 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:27.889 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d12dc700 1 --2- 172.21.15.26:0/3407070988 >> v2:172.21.15.120:3300/0 conn(0x7f55cc061640 0x7f55cc143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.889 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.863+0000 7f55d1add700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.890 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 -- 172.21.15.26:0/3407070988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 msgr2=0x7f55cc146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:27.890 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.890 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 -- 172.21.15.26:0/3407070988 >> v2:172.21.15.120:3300/0 conn(0x7f55cc061640 msgr2=0x7f55cc143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:27.891 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3407070988 >> v2:172.21.15.120:3300/0 conn(0x7f55cc061640 0x7f55cc143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.891 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 -- 172.21.15.26:0/3407070988 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0b6e30 con 0x7f55cc0b5b50 2022-07-20T02:25:27.893 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc0b9c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f55cc143170 tx=0x7f55bc006eb0).ready entity=mon.0 client_cookie=57ae53de0d5038b5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:27.893 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55bbfff700 1 -- 172.21.15.26:0/3407070988 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc00e610 con 0x7f55cc0b5b50 2022-07-20T02:25:27.894 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55bbfff700 1 -- 172.21.15.26:0/3407070988 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc000e90 con 0x7f55cc0b5b50 2022-07-20T02:25:27.894 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55bbfff700 1 -- 172.21.15.26:0/3407070988 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc00e9a0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.895 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- 172.21.15.26:0/3407070988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 msgr2=0x7f55cc0b9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:27.895 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc0b9c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f55cc143170 tx=0x7f55bc006eb0).stop 2022-07-20T02:25:27.896 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- 172.21.15.26:0/3407070988 shutdown_connections 2022-07-20T02:25:27.896 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.897 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 --2- 172.21.15.26:0/3407070988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc0b9c30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.897 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 --2- 172.21.15.26:0/3407070988 >> v2:172.21.15.120:3300/0 conn(0x7f55cc061640 0x7f55cc143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.898 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- 172.21.15.26:0/3407070988 >> 172.21.15.26:0/3407070988 conn(0x7f55cc05c5b0 msgr2=0x7f55cc05c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:27.898 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- 172.21.15.26:0/3407070988 shutdown_connections 2022-07-20T02:25:27.899 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- 172.21.15.26:0/3407070988 wait complete. 2022-07-20T02:25:27.899 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 Processor -- start 2022-07-20T02:25:27.899 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.867+0000 7f55d22de700 1 -- start start 2022-07-20T02:25:27.900 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55cc061640 0x7f55cc0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.900 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.901 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.901 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1524a0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.902 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc142b80 con 0x7f55cc061640 2022-07-20T02:25:27.902 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc152690 con 0x7f55cc0ba120 2022-07-20T02:25:27.903 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.914 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34686/0 (socket says 172.21.15.26:34686) 2022-07-20T02:25:27.914 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 -- 172.21.15.26:0/3330109561 learned_addr learned my addr 172.21.15.26:0/3330109561 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:27.914 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d1add700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d12dc700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55cc061640 0x7f55cc0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 -- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 msgr2=0x7f55cc152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:27.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 -- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55cc061640 msgr2=0x7f55cc0b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:27.915 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55cc061640 0x7f55cc0b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:27.916 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 -- 172.21.15.26:0/3330109561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc00b1e0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.916 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d0adb700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f55cc143170 tx=0x7f55bc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:27.916 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc00e070 con 0x7f55cc0b5b50 2022-07-20T02:25:27.916 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc00e990 con 0x7f55cc0b5b50 2022-07-20T02:25:27.917 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc011de0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.917 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc142640 con 0x7f55cc0b5b50 2022-07-20T02:25:27.917 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.871+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc0b3290 con 0x7f55cc0b5b50 2022-07-20T02:25:27.917 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc0ba5b0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.946 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f55bc025030 con 0x7f55cc0b5b50 2022-07-20T02:25:27.946 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55ba7fc700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 0x7f55ac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:27.946 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f55bc06cde0 con 0x7f55cc0b5b50 2022-07-20T02:25:27.947 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55d12dc700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 0x7f55ac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:27.948 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.875+0000 7f55d12dc700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 0x7f55ac055ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f55c800b000 tx=0x7f55c8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:27.949 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:27.883+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55bc044080 con 0x7f55cc0b5b50 2022-07-20T02:25:28.020 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55cc0574e0 con 0x7f55cc0b5b50 2022-07-20T02:25:28.020 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55ba7fc700 1 -- 172.21.15.26:0/3330109561 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11853 (secure 0 0 0) 0x7f55bc023020 con 0x7f55cc0b5b50 2022-07-20T02:25:28.021 INFO:teuthology.misc.health.smithi026.stdout: 2022-07-20T02:25:28.038 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":15,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:26.314322+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:28.048 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 msgr2=0x7f55ac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.049 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 0x7f55ac055ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f55c800b000 tx=0x7f55c8006df0).stop 2022-07-20T02:25:28.049 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 msgr2=0x7f55cc152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.050 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.011+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f55cc143170 tx=0x7f55bc015000).stop 2022-07-20T02:25:28.051 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 shutdown_connections 2022-07-20T02:25:28.057 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55cc0b5b50 0x7f55cc152af0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.058 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55ac0539e0 0x7f55ac055ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.058 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55cc0ba120 0x7f55cc152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.058 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 --2- 172.21.15.26:0/3330109561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55cc061640 0x7f55cc0b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.058 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 >> 172.21.15.26:0/3330109561 conn(0x7f55cc05c5b0 msgr2=0x7f55cc0b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:28.059 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 shutdown_connections 2022-07-20T02:25:28.059 INFO:teuthology.misc.health.smithi026.stderr:2022-07-20T02:25:28.015+0000 7f55d22de700 1 -- 172.21.15.26:0/3330109561 wait complete. 2022-07-20T02:25:28.068 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-07-20T02:25:28.068 INFO:tasks.ceph:Creating RBD pool 2022-07-20T02:25:28.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool create rbd 8 2022-07-20T02:25:28.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.175+0000 7fb51bfff700 1 Processor -- start 2022-07-20T02:25:28.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.175+0000 7fb51bfff700 1 -- start start 2022-07-20T02:25:28.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb51c066c00 0x7fb51c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 0x7fb51c061110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c146bb0 con 0x7fb51c05ce30 2022-07-20T02:25:28.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c146e50 con 0x7fb51c066c00 2022-07-20T02:25:28.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c1470f0 con 0x7fb51c060c80 2022-07-20T02:25:28.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb51c066c00 0x7fb51c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51affd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb51c05ce30 0x7fb51c0ac6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33972/0 (socket says 172.21.15.26:33972) 2022-07-20T02:25:28.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51affd700 1 -- 172.21.15.26:0/2837410460 learned_addr learned my addr 172.21.15.26:0/2837410460 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:28.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51b7fe700 1 --2- 172.21.15.26:0/2837410460 >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 0x7fb51c061110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 -- 172.21.15.26:0/2837410460 >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 msgr2=0x7fb51c061110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:28.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 --2- 172.21.15.26:0/2837410460 >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 0x7fb51c061110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 -- 172.21.15.26:0/2837410460 >> v1:172.21.15.26:6789/0 conn(0x7fb51c05ce30 legacy=0x7fb51c0ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51a7fc700 1 -- 172.21.15.26:0/2837410460 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157744800 0 0) 0x7fb51c146bb0 con 0x7fb51c05ce30 2022-07-20T02:25:28.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 -- 172.21.15.26:0/2837410460 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c147390 con 0x7fb51c066c00 2022-07-20T02:25:28.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51b7fe700 1 --2- 172.21.15.26:0/2837410460 >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 0x7fb51c061110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:28.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb513fff700 1 --2- 172.21.15.26:0/2837410460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c066c00 0x7fb51c0ace00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb504002f70 tx=0x7fb504006df0).ready entity=mon.0 client_cookie=d964b97c620bc2f8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51a7fc700 1 -- 172.21.15.26:0/2837410460 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb50400d570 con 0x7fb51c066c00 2022-07-20T02:25:28.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51a7fc700 1 -- 172.21.15.26:0/2837410460 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504000e90 con 0x7fb51c066c00 2022-07-20T02:25:28.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51a7fc700 1 -- 172.21.15.26:0/2837410460 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb50400d8c0 con 0x7fb51c066c00 2022-07-20T02:25:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- 172.21.15.26:0/2837410460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c066c00 msgr2=0x7fb51c0ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 --2- 172.21.15.26:0/2837410460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c066c00 0x7fb51c0ace00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb504002f70 tx=0x7fb504006df0).stop 2022-07-20T02:25:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- 172.21.15.26:0/2837410460 shutdown_connections 2022-07-20T02:25:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 --2- 172.21.15.26:0/2837410460 >> v2:172.21.15.120:3300/0 conn(0x7fb51c060c80 0x7fb51c061110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 --2- 172.21.15.26:0/2837410460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c066c00 0x7fb51c0ace00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- 172.21.15.26:0/2837410460 >> 172.21.15.26:0/2837410460 conn(0x7fb51c0b7090 msgr2=0x7fb51c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:28.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- 172.21.15.26:0/2837410460 shutdown_connections 2022-07-20T02:25:28.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.179+0000 7fb51bfff700 1 -- 172.21.15.26:0/2837410460 wait complete. 2022-07-20T02:25:28.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 Processor -- start 2022-07-20T02:25:28.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 -- start start 2022-07-20T02:25:28.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb51c05ce30 0x7fb51c0ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c066c00 0x7fb51c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0b1910 con 0x7fb51c060c80 2022-07-20T02:25:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c066900 con 0x7fb51c066c00 2022-07-20T02:25:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0addc0 con 0x7fb51c05ce30 2022-07-20T02:25:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34700/0 (socket says 172.21.15.26:34700) 2022-07-20T02:25:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 -- 172.21.15.26:0/535545769 learned_addr learned my addr 172.21.15.26:0/535545769 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51b7fe700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c066c00 0x7fb51c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb51affd700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb51c05ce30 0x7fb51c0ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 -- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb51c05ce30 msgr2=0x7fb51c0ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb51c05ce30 0x7fb51c0ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 -- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c066c00 msgr2=0x7fb51c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c066c00 0x7fb51c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 -- 172.21.15.26:0/535545769 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb514009470 con 0x7fb51c060c80 2022-07-20T02:25:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.183+0000 7fb513fff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb51c066a90 tx=0x7fb504014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb504016070 con 0x7fb51c060c80 2022-07-20T02:25:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504013980 con 0x7fb51c060c80 2022-07-20T02:25:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50400b180 con 0x7fb51c060c80 2022-07-20T02:25:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb504013ce0 con 0x7fb51c060c80 2022-07-20T02:25:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c065cd0 con 0x7fb51c060c80 2022-07-20T02:25:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c065b40 con 0x7fb51c060c80 2022-07-20T02:25:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb504025030 con 0x7fb51c060c80 2022-07-20T02:25:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 0x7fb4f4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7fb50406c990 con 0x7fb51c060c80 2022-07-20T02:25:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb51affd700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 0x7fb4f4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.187+0000 7fb51affd700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 0x7fb4f4055e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb50c00d000 tx=0x7fb50c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.195+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb504043d40 con 0x7fb51c060c80 2022-07-20T02:25:28.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.319+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fb51c066110 con 0x7fb51c060c80 2022-07-20T02:25:28.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.431+0000 7fb518ff9700 1 -- 172.21.15.26:0/535545769 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7fb504023070 con 0x7fb51c060c80 2022-07-20T02:25:28.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 msgr2=0x7fb4f4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 0x7fb4f4055e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb50c00d000 tx=0x7fb50c009040).stop 2022-07-20T02:25:28.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 msgr2=0x7fb51c066410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb51c066a90 tx=0x7fb504014000).stop 2022-07-20T02:25:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 shutdown_connections 2022-07-20T02:25:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb51c060c80 0x7fb51c066410 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4f4053950 0x7fb4f4055e10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c066c00 0x7fb51c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 --2- 172.21.15.26:0/535545769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb51c05ce30 0x7fb51c0ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 >> 172.21.15.26:0/535545769 conn(0x7fb51c0b7090 msgr2=0x7fb51c0b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:28.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 shutdown_connections 2022-07-20T02:25:28.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.435+0000 7fb51bfff700 1 -- 172.21.15.26:0/535545769 wait complete. 2022-07-20T02:25:28.444 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2022-07-20T02:25:28.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-20T02:25:28.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.555+0000 7fd851e5b700 1 Processor -- start 2022-07-20T02:25:28.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.555+0000 7fd851e5b700 1 -- start start 2022-07-20T02:25:28.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 0x7fd84c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd84c0b1160 0x7fd84c140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c05d3b0 con 0x7fd84c146e80 2022-07-20T02:25:28.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c1435b0 con 0x7fd84c0b1160 2022-07-20T02:25:28.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c0b4bc0 con 0x7fd84c05cfd0 2022-07-20T02:25:28.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd84c0b1160 0x7fd84c140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd84c0b1160 0x7fd84c140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34708/0 (socket says 172.21.15.26:34708) 2022-07-20T02:25:28.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 -- 172.21.15.26:0/508379555 learned_addr learned my addr 172.21.15.26:0/508379555 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:28.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd84b7fe700 1 --2- 172.21.15.26:0/508379555 >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 0x7fd84c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 -- 172.21.15.26:0/508379555 >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 msgr2=0x7fd84c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:28.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 --2- 172.21.15.26:0/508379555 >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 0x7fd84c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 -- 172.21.15.26:0/508379555 >> v1:172.21.15.120:6789/0 conn(0x7fd84c146e80 legacy=0x7fd84c147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 -- 172.21.15.26:0/508379555 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c0b4e60 con 0x7fd84c0b1160 2022-07-20T02:25:28.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd84b7fe700 1 --2- 172.21.15.26:0/508379555 >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 0x7fd84c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:28.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd842ffd700 1 --2- 172.21.15.26:0/508379555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c140cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd834002f70 tx=0x7fd834006eb0).ready entity=mon.0 client_cookie=4803e1e8cdba972c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd84affd700 1 -- 172.21.15.26:0/508379555 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83400d5c0 con 0x7fd84c0b1160 2022-07-20T02:25:28.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd84affd700 1 -- 172.21.15.26:0/508379555 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd834000e90 con 0x7fd84c0b1160 2022-07-20T02:25:28.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd84affd700 1 -- 172.21.15.26:0/508379555 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83400d950 con 0x7fd84c0b1160 2022-07-20T02:25:28.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- 172.21.15.26:0/508379555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 msgr2=0x7fd84c140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 --2- 172.21.15.26:0/508379555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c140cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd834002f70 tx=0x7fd834006eb0).stop 2022-07-20T02:25:28.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- 172.21.15.26:0/508379555 shutdown_connections 2022-07-20T02:25:28.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 --2- 172.21.15.26:0/508379555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c140cb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 --2- 172.21.15.26:0/508379555 >> v2:172.21.15.120:3300/0 conn(0x7fd84c05cfd0 0x7fd84c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- 172.21.15.26:0/508379555 >> 172.21.15.26:0/508379555 conn(0x7fd84c066f70 msgr2=0x7fd84c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:28.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- 172.21.15.26:0/508379555 shutdown_connections 2022-07-20T02:25:28.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.559+0000 7fd851e5b700 1 -- 172.21.15.26:0/508379555 wait complete. 2022-07-20T02:25:28.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 Processor -- start 2022-07-20T02:25:28.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- start start 2022-07-20T02:25:28.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c05cfd0 0x7fd84c13bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 0x7fd84c0b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c138070 con 0x7fd84c0b1160 2022-07-20T02:25:28.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c1560c0 con 0x7fd84c146e80 2022-07-20T02:25:28.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c13f8f0 con 0x7fd84c05cfd0 2022-07-20T02:25:28.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd842ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 0x7fd84c0b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd842ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34714/0 (socket says 172.21.15.26:34714) 2022-07-20T02:25:28.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd842ffd700 1 -- 172.21.15.26:0/4122117804 learned_addr learned my addr 172.21.15.26:0/4122117804 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:28.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84b7fe700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c05cfd0 0x7fd84c13bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 -- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c05cfd0 msgr2=0x7fd84c13bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c05cfd0 0x7fd84c13bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 -- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 msgr2=0x7fd84c156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:28.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:28.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 -- 172.21.15.26:0/4122117804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c00b180 con 0x7fd84c146e80 2022-07-20T02:25:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd84bfff700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 0x7fd84c0b7ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd84c0b27d0 tx=0x7fd844007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd84400ce10 con 0x7fd84c146e80 2022-07-20T02:25:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd844006aa0 con 0x7fd84c146e80 2022-07-20T02:25:28.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83400b1e0 con 0x7fd84c146e80 2022-07-20T02:25:28.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd844006e00 con 0x7fd84c146e80 2022-07-20T02:25:28.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.563+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c13c990 con 0x7fd84c146e80 2022-07-20T02:25:28.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c0574e0 con 0x7fd84c146e80 2022-07-20T02:25:28.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd844010030 con 0x7fd84c146e80 2022-07-20T02:25:28.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd8497fa700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 0x7fd830055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:28.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd84406b8b0 con 0x7fd84c146e80 2022-07-20T02:25:28.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd84b7fe700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 0x7fd830055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:28.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.567+0000 7fd84b7fe700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 0x7fd830055f30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd83c00b000 tx=0x7fd83c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:28.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.575+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd844042b10 con 0x7fd84c146e80 2022-07-20T02:25:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:28.707+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd84c141230 con 0x7fd84c146e80 2022-07-20T02:25:29.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.439+0000 7fd8497fa700 1 -- 172.21.15.26:0/4122117804 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd844046130 con 0x7fd84c146e80 2022-07-20T02:25:29.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 msgr2=0x7fd830055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 0x7fd830055f30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd83c00b000 tx=0x7fd83c006df0).stop 2022-07-20T02:25:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 msgr2=0x7fd84c0b7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 0x7fd84c0b7ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd84c0b27d0 tx=0x7fd844007f10).stop 2022-07-20T02:25:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 shutdown_connections 2022-07-20T02:25:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd84c0b1160 0x7fd84c156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd830053a70 0x7fd830055f30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84c146e80 0x7fd84c0b7ab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 --2- 172.21.15.26:0/4122117804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84c05cfd0 0x7fd84c13bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.447+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 >> 172.21.15.26:0/4122117804 conn(0x7fd84c066f70 msgr2=0x7fd84c0b99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:29.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.451+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 shutdown_connections 2022-07-20T02:25:29.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.451+0000 7fd851e5b700 1 -- 172.21.15.26:0/4122117804 wait complete. 2022-07-20T02:25:29.474 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-20T02:25:29.476 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-07-20T02:25:29.477 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:25:29.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:25:29.490 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:25:29.491 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:25:29.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:25:29.507 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:25:29.507 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:25:29.508 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:25:29.527 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:25:29.528 INFO:tasks.ceph:Setting up CephFS filesystem... 2022-07-20T02:25:29.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:29.562 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:25:29.579 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:25:29.604 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:25:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 Processor -- start 2022-07-20T02:25:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 -- start start 2022-07-20T02:25:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5100143090 0x7f5100098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51000b0c50 0x7f51000957f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100143470 con 0x7f51000b0810 2022-07-20T02:25:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510005fd90 con 0x7f51000b0c50 2022-07-20T02:25:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5100097d50 con 0x7f5100143090 2022-07-20T02:25:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51000b0c50 0x7f51000957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51000b0c50 0x7f51000957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34746/0 (socket says 172.21.15.26:34746) 2022-07-20T02:25:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 -- 172.21.15.26:0/2774366678 learned_addr learned my addr 172.21.15.26:0/2774366678 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5105850700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5100143090 0x7f5100098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f510484e700 1 -- 172.21.15.26:0/2774366678 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702222147 0 0) 0x7f5100143470 con 0x7f51000b0810 2022-07-20T02:25:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f510484e700 1 -- 172.21.15.26:0/2774366678 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec003260 con 0x7f51000b0810 2022-07-20T02:25:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 -- 172.21.15.26:0/2774366678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5100143090 msgr2=0x7f5100098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5100143090 0x7f5100098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 -- 172.21.15.26:0/2774366678 >> v1:172.21.15.120:6789/0 conn(0x7f51000b0810 legacy=0x7f51000950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 -- 172.21.15.26:0/2774366678 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100097ff0 con 0x7f51000b0c50 2022-07-20T02:25:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5106051700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51000b0c50 0x7f51000957f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5100098c00 tx=0x7f50f8006eb0).ready entity=mon.0 client_cookie=38accc27ce6c38ff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f510484e700 1 -- 172.21.15.26:0/2774366678 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f800d5c0 con 0x7f51000b0c50 2022-07-20T02:25:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f510484e700 1 -- 172.21.15.26:0/2774366678 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f8000e90 con 0x7f51000b0c50 2022-07-20T02:25:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f510484e700 1 -- 172.21.15.26:0/2774366678 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f800d950 con 0x7f51000b0c50 2022-07-20T02:25:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 -- 172.21.15.26:0/2774366678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51000b0c50 msgr2=0x7f51000957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.623+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51000b0c50 0x7f51000957f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5100098c00 tx=0x7f50f8006eb0).stop 2022-07-20T02:25:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 -- 172.21.15.26:0/2774366678 shutdown_connections 2022-07-20T02:25:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51000b0c50 0x7f51000957f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2774366678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5100143090 0x7f5100098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 -- 172.21.15.26:0/2774366678 >> 172.21.15.26:0/2774366678 conn(0x7f51000b5e70 msgr2=0x7f51000b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 -- 172.21.15.26:0/2774366678 shutdown_connections 2022-07-20T02:25:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.627+0000 7f5107ab4700 1 -- 172.21.15.26:0/2774366678 wait complete. 2022-07-20T02:25:29.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 Processor -- start 2022-07-20T02:25:29.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 -- start start 2022-07-20T02:25:29.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51000b0810 0x7f51000944a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51000b0c50 0x7f51000ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5100095440 con 0x7f5100143090 2022-07-20T02:25:29.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5100065310 con 0x7f51000b0c50 2022-07-20T02:25:29.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5107ab4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51000655b0 con 0x7f51000b0810 2022-07-20T02:25:29.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5106051700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5106051700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34754/0 (socket says 172.21.15.26:34754) 2022-07-20T02:25:29.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5106051700 1 -- 172.21.15.26:0/2464658214 learned_addr learned my addr 172.21.15.26:0/2464658214 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:29.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f5105850700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51000b0810 0x7f51000944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.631+0000 7f510504f700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51000b0c50 0x7f51000ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 -- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51000b0810 msgr2=0x7f51000944a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51000b0810 0x7f51000944a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 -- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51000b0c50 msgr2=0x7f51000ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51000b0c50 0x7f51000ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 -- 172.21.15.26:0/2464658214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f800b1e0 con 0x7f5100143090 2022-07-20T02:25:29.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5106051700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5100098c00 tx=0x7f50f8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:29.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f8021070 con 0x7f5100143090 2022-07-20T02:25:29.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f8002f70 con 0x7f5100143090 2022-07-20T02:25:29.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5100065850 con 0x7f5100143090 2022-07-20T02:25:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f8011de0 con 0x7f5100143090 2022-07-20T02:25:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5100063e10 con 0x7f5100143090 2022-07-20T02:25:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5100065cf0 con 0x7f5100143090 2022-07-20T02:25:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.635+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f50f800f070 con 0x7f5100143090 2022-07-20T02:25:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.639+0000 7f50feffd700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 0x7f50e0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.639+0000 7f5105850700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 0x7f50e0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.639+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f50f806cf40 con 0x7f5100143090 2022-07-20T02:25:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.639+0000 7f5105850700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 0x7f50e0055e10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f50f400b000 tx=0x7f50f4006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:29.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.643+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5100065cf0 con 0x7f5100143090 2022-07-20T02:25:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.763+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5100065cf0 con 0x7f5100143090 2022-07-20T02:25:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f50feffd700 1 -- 172.21.15.26:0/2464658214 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f5100065cf0 con 0x7f5100143090 2022-07-20T02:25:29.774 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:29.775 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":17,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:29.429440+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 msgr2=0x7f50e0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 0x7f50e0055e10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f50f400b000 tx=0x7f50f4006df0).stop 2022-07-20T02:25:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 msgr2=0x7f51000ad6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5100098c00 tx=0x7f50f8015000).stop 2022-07-20T02:25:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 shutdown_connections 2022-07-20T02:25:29.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5100143090 0x7f51000ad6d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f50e0053950 0x7f50e0055e10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51000b0c50 0x7f51000ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 --2- 172.21.15.26:0/2464658214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51000b0810 0x7f51000944a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 >> 172.21.15.26:0/2464658214 conn(0x7f51000b5e70 msgr2=0x7f51001422e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:29.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 shutdown_connections 2022-07-20T02:25:29.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.767+0000 7f5107ab4700 1 -- 172.21.15.26:0/2464658214 wait complete. 2022-07-20T02:25:29.797 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:25:29.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 Processor -- start 2022-07-20T02:25:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 -- start start 2022-07-20T02:25:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77e81465b0 0x7f77e8066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f77e80b24c0 0x7f77e80604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e80672a0 0x7f77e80609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:29.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80b2e20 con 0x7f77e81465b0 2022-07-20T02:25:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80b1ae0 con 0x7f77e80b24c0 2022-07-20T02:25:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ef1d5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b1d80 con 0x7f77e80672a0 2022-07-20T02:25:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77e81465b0 0x7f77e8066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77e81465b0 0x7f77e8066db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34758/0 (socket says 172.21.15.26:34758) 2022-07-20T02:25:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 -- 172.21.15.26:0/26739078 learned_addr learned my addr 172.21.15.26:0/26739078 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee9d4700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e80672a0 0x7f77e80609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ed9d2700 1 --2- 172.21.15.26:0/26739078 >> v2:172.21.15.120:3300/0 conn(0x7f77e80b24c0 0x7f77e80604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 -- 172.21.15.26:0/26739078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e80672a0 msgr2=0x7f77e80609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e80672a0 0x7f77e80609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 -- 172.21.15.26:0/26739078 >> v2:172.21.15.120:3300/0 conn(0x7f77e80b24c0 msgr2=0x7f77e80604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/26739078 >> v2:172.21.15.120:3300/0 conn(0x7f77e80b24c0 0x7f77e80604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 -- 172.21.15.26:0/26739078 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80b2020 con 0x7f77e81465b0 2022-07-20T02:25:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e81465b0 0x7f77e8066db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f77e4002f70 tx=0x7f77e4006eb0).ready entity=mon.0 client_cookie=548447331ead0b08 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ed1d1700 1 -- 172.21.15.26:0/26739078 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e400d570 con 0x7f77e81465b0 2022-07-20T02:25:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ed1d1700 1 -- 172.21.15.26:0/26739078 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e4000e90 con 0x7f77e81465b0 2022-07-20T02:25:29.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.963+0000 7f77ed1d1700 1 -- 172.21.15.26:0/26739078 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e400d900 con 0x7f77e81465b0 2022-07-20T02:25:30.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- 172.21.15.26:0/26739078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e81465b0 msgr2=0x7f77e8066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e81465b0 0x7f77e8066db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f77e4002f70 tx=0x7f77e4006eb0).stop 2022-07-20T02:25:30.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- 172.21.15.26:0/26739078 shutdown_connections 2022-07-20T02:25:30.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e80672a0 0x7f77e80609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/26739078 >> v2:172.21.15.120:3300/0 conn(0x7f77e80b24c0 0x7f77e80604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/26739078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e81465b0 0x7f77e8066db0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- 172.21.15.26:0/26739078 >> 172.21.15.26:0/26739078 conn(0x7f77e80b3050 msgr2=0x7f77e80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- 172.21.15.26:0/26739078 shutdown_connections 2022-07-20T02:25:30.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- 172.21.15.26:0/26739078 wait complete. 2022-07-20T02:25:30.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 Processor -- start 2022-07-20T02:25:30.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 -- start start 2022-07-20T02:25:30.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.967+0000 7f77ef1d5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e81465b0 0x7f77e8152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80650c0 0x7f77e8152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8064160 con 0x7f77e80672a0 2022-07-20T02:25:30.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8064eb0 con 0x7f77e80650c0 2022-07-20T02:25:30.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80644d0 con 0x7f77e81465b0 2022-07-20T02:25:30.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34764/0 (socket says 172.21.15.26:34764) 2022-07-20T02:25:30.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 -- 172.21.15.26:0/1818790293 learned_addr learned my addr 172.21.15.26:0/1818790293 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:30.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ed9d2700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e81465b0 0x7f77e8152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee9d4700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80650c0 0x7f77e8152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 -- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e81465b0 msgr2=0x7f77e8152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e81465b0 0x7f77e8152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 -- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80650c0 msgr2=0x7f77e8152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80650c0 0x7f77e8152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 -- 172.21.15.26:0/1818790293 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e400b1e0 con 0x7f77e80672a0 2022-07-20T02:25:30.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ee1d3700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f77e4002f70 tx=0x7f77e4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e40117e0 con 0x7f77e80672a0 2022-07-20T02:25:30.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e4011940 con 0x7f77e80672a0 2022-07-20T02:25:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e4011ca0 con 0x7f77e80672a0 2022-07-20T02:25:30.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e8064770 con 0x7f77e80672a0 2022-07-20T02:25:30.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e8152df0 con 0x7f77e80672a0 2022-07-20T02:25:30.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f77e401a040 con 0x7f77e80672a0 2022-07-20T02:25:30.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e805f840 con 0x7f77e80672a0 2022-07-20T02:25:30.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.971+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f77e406ca40 con 0x7f77e80672a0 2022-07-20T02:25:30.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.975+0000 7f77ed9d2700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.975+0000 7f77ed9d2700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 0x7f77c8055f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f77d800b000 tx=0x7f77d8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:29.979+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77e4043d10 con 0x7f77e80672a0 2022-07-20T02:25:30.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.103+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f77e80574e0 con 0x7f77e80672a0 2022-07-20T02:25:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.107+0000 7f77d77fe700 1 -- 172.21.15.26:0/1818790293 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f77e4023020 con 0x7f77e80672a0 2022-07-20T02:25:30.177 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.107+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 msgr2=0x7f77c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.107+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 0x7f77c8055f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f77d800b000 tx=0x7f77d8006df0).stop 2022-07-20T02:25:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.107+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 msgr2=0x7f77e81411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.107+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f77e4002f70 tx=0x7f77e4015000).stop 2022-07-20T02:25:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 shutdown_connections 2022-07-20T02:25:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80650c0 0x7f77e8152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e81465b0 0x7f77e8152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 --2- 172.21.15.26:0/1818790293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80672a0 0x7f77e81411c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 >> 172.21.15.26:0/1818790293 conn(0x7f77e80b3050 msgr2=0x7f77e80b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 shutdown_connections 2022-07-20T02:25:30.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.111+0000 7f77ef1d5700 1 -- 172.21.15.26:0/1818790293 wait complete. 2022-07-20T02:25:30.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:30.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.315+0000 7f29dffff700 1 Processor -- start 2022-07-20T02:25:30.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.315+0000 7f29dffff700 1 -- start start 2022-07-20T02:25:30.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 0x7f29e0147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e005d4d0 con 0x7f29e00b1200 2022-07-20T02:25:30.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e00a9ff0 con 0x7f29e005c720 2022-07-20T02:25:30.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0067e60 con 0x7f29e00afe40 2022-07-20T02:25:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29df7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29e00b1200 0x7f29e00abbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34054/0 (socket says 172.21.15.26:34054) 2022-07-20T02:25:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29df7fe700 1 -- 172.21.15.26:0/427746328 learned_addr learned my addr 172.21.15.26:0/427746328 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 --2- 172.21.15.26:0/427746328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29deffd700 1 --2- 172.21.15.26:0/427746328 >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 0x7f29e0147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29ddffb700 1 -- 172.21.15.26:0/427746328 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809120237 0 0) 0x7f29e005d4d0 con 0x7f29e00b1200 2022-07-20T02:25:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29ddffb700 1 -- 172.21.15.26:0/427746328 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c4003260 con 0x7f29e00b1200 2022-07-20T02:25:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 -- 172.21.15.26:0/427746328 >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 msgr2=0x7f29e0147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 --2- 172.21.15.26:0/427746328 >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 0x7f29e0147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 -- 172.21.15.26:0/427746328 >> v1:172.21.15.26:6789/0 conn(0x7f29e00b1200 legacy=0x7f29e00abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 -- 172.21.15.26:0/427746328 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0067bc0 con 0x7f29e00afe40 2022-07-20T02:25:30.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29deffd700 1 --2- 172.21.15.26:0/427746328 >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 0x7f29e0147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:30.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29de7fc700 1 --2- 172.21.15.26:0/427746328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0147a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f29d4002f70 tx=0x7f29d4006df0).ready entity=mon.2 client_cookie=d5a34734cea61ff3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29ddffb700 1 -- 172.21.15.26:0/427746328 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d400d4a0 con 0x7f29e00afe40 2022-07-20T02:25:30.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29ddffb700 1 -- 172.21.15.26:0/427746328 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4000e90 con 0x7f29e00afe40 2022-07-20T02:25:30.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29ddffb700 1 -- 172.21.15.26:0/427746328 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d400d7f0 con 0x7f29e00afe40 2022-07-20T02:25:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- 172.21.15.26:0/427746328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 msgr2=0x7f29e0147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 --2- 172.21.15.26:0/427746328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0147a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f29d4002f70 tx=0x7f29d4006df0).stop 2022-07-20T02:25:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- 172.21.15.26:0/427746328 shutdown_connections 2022-07-20T02:25:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 --2- 172.21.15.26:0/427746328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0147a40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 --2- 172.21.15.26:0/427746328 >> v2:172.21.15.120:3300/0 conn(0x7f29e005c720 0x7f29e0147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- 172.21.15.26:0/427746328 >> 172.21.15.26:0/427746328 conn(0x7f29e0060310 msgr2=0x7f29e00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- 172.21.15.26:0/427746328 shutdown_connections 2022-07-20T02:25:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.319+0000 7f29dffff700 1 -- 172.21.15.26:0/427746328 wait complete. 2022-07-20T02:25:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 Processor -- start 2022-07-20T02:25:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- start start 2022-07-20T02:25:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0066d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e00b1200 0x7f29e0067250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00652c0 con 0x7f29e005c720 2022-07-20T02:25:30.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0067740 con 0x7f29e00b1200 2022-07-20T02:25:30.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00654f0 con 0x7f29e00afe40 2022-07-20T02:25:30.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34778/0 (socket says 172.21.15.26:34778) 2022-07-20T02:25:30.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 -- 172.21.15.26:0/875545951 learned_addr learned my addr 172.21.15.26:0/875545951 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:30.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29de7fc700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0066d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29df7fe700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e00b1200 0x7f29e0067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 -- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 msgr2=0x7f29e0066d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0066d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 -- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e00b1200 msgr2=0x7f29e0067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e00b1200 0x7f29e0067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 -- 172.21.15.26:0/875545951 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8009470 con 0x7f29e005c720 2022-07-20T02:25:30.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29deffd700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f29d800c000 tx=0x7f29d80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d800c850 con 0x7f29e005c720 2022-07-20T02:25:30.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d800c9b0 con 0x7f29e005c720 2022-07-20T02:25:30.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d400b180 con 0x7f29e005c720 2022-07-20T02:25:30.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d800cd10 con 0x7f29e005c720 2022-07-20T02:25:30.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.323+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e0159c10 con 0x7f29e005c720 2022-07-20T02:25:30.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e00aa250 con 0x7f29e005c720 2022-07-20T02:25:30.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29d8022070 con 0x7f29e005c720 2022-07-20T02:25:30.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29cffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 0x7f29bc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f29d806c380 con 0x7f29e005c720 2022-07-20T02:25:30.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29de7fc700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 0x7f29bc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.327+0000 7f29de7fc700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 0x7f29bc055f30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f29d401a000 tx=0x7f29d4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.335+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29d80434c0 con 0x7f29e005c720 2022-07-20T02:25:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.463+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f29e00574e0 con 0x7f29e005c720 2022-07-20T02:25:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.463+0000 7f29cffff700 1 -- 172.21.15.26:0/875545951 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f29d8046ae0 con 0x7f29e005c720 2022-07-20T02:25:30.468 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 msgr2=0x7f29bc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 0x7f29bc055f30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f29d401a000 tx=0x7f29d4014000).stop 2022-07-20T02:25:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 msgr2=0x7f29e014d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f29d800c000 tx=0x7f29d80091d0).stop 2022-07-20T02:25:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 shutdown_connections 2022-07-20T02:25:30.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29bc053a70 0x7f29bc055f30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e00b1200 0x7f29e0067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e00afe40 0x7f29e0066d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 --2- 172.21.15.26:0/875545951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29e005c720 0x7f29e014d530 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 >> 172.21.15.26:0/875545951 conn(0x7f29e0060310 msgr2=0x7f29e00a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 shutdown_connections 2022-07-20T02:25:30.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.467+0000 7f29dffff700 1 -- 172.21.15.26:0/875545951 wait complete. 2022-07-20T02:25:30.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:25:30.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 Processor -- start 2022-07-20T02:25:30.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 -- start start 2022-07-20T02:25:30.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbfc40b3260 0x7fbfc4060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfc40a95a0 0x7fbfc40676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc40b3640 con 0x7fbfc40b1720 2022-07-20T02:25:30.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40615c0 con 0x7fbfc40a95a0 2022-07-20T02:25:30.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.579+0000 7fbfca8b4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc4061860 con 0x7fbfc40b3260 2022-07-20T02:25:30.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca0b3700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbfc40b1720 0x7fbfc40b1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34068/0 (socket says 172.21.15.26:34068) 2022-07-20T02:25:30.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca0b3700 1 -- 172.21.15.26:0/2587761533 learned_addr learned my addr 172.21.15.26:0/2587761533 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:30.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc90b1700 1 --2- 172.21.15.26:0/2587761533 >> v2:172.21.15.26:3300/0 conn(0x7fbfc40a95a0 0x7fbfc40676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2587761533 >> v2:172.21.15.120:3300/0 conn(0x7fbfc40b3260 0x7fbfc4060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc88b0700 1 -- 172.21.15.26:0/2587761533 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210220412 0 0) 0x7fbfc40b3640 con 0x7fbfc40b1720 2022-07-20T02:25:30.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc88b0700 1 -- 172.21.15.26:0/2587761533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac003260 con 0x7fbfc40b1720 2022-07-20T02:25:30.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 -- 172.21.15.26:0/2587761533 >> v2:172.21.15.26:3300/0 conn(0x7fbfc40a95a0 msgr2=0x7fbfc40676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:30.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2587761533 >> v2:172.21.15.26:3300/0 conn(0x7fbfc40a95a0 0x7fbfc40676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 -- 172.21.15.26:0/2587761533 >> v1:172.21.15.26:6789/0 conn(0x7fbfc40b1720 legacy=0x7fbfc40b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 -- 172.21.15.26:0/2587761533 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4061b00 con 0x7fbfc40b3260 2022-07-20T02:25:30.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2587761533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40b3260 0x7fbfc4060950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbfc0002f70 tx=0x7fbfc0006df0).ready entity=mon.1 client_cookie=2828872c58e34335 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc88b0700 1 -- 172.21.15.26:0/2587761533 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfc000d4f0 con 0x7fbfc40b3260 2022-07-20T02:25:30.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc88b0700 1 -- 172.21.15.26:0/2587761533 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0000e90 con 0x7fbfc40b3260 2022-07-20T02:25:30.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfc88b0700 1 -- 172.21.15.26:0/2587761533 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfc000d840 con 0x7fbfc40b3260 2022-07-20T02:25:30.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2587761533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40b3260 msgr2=0x7fbfc4060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2587761533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40b3260 0x7fbfc4060950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbfc0002f70 tx=0x7fbfc0006df0).stop 2022-07-20T02:25:30.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2587761533 shutdown_connections 2022-07-20T02:25:30.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2587761533 >> v2:172.21.15.26:3300/0 conn(0x7fbfc40a95a0 0x7fbfc40676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2587761533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40b3260 0x7fbfc4060950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2587761533 >> 172.21.15.26:0/2587761533 conn(0x7fbfc40b67f0 msgr2=0x7fbfc40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2587761533 shutdown_connections 2022-07-20T02:25:30.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.583+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2587761533 wait complete. 2022-07-20T02:25:30.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 Processor -- start 2022-07-20T02:25:30.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- start start 2022-07-20T02:25:30.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 0x7fbfc4093210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40b3260 0x7fbfc40921d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40b57d0 con 0x7fbfc40b1720 2022-07-20T02:25:30.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40916d0 con 0x7fbfc40a95a0 2022-07-20T02:25:30.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4091d70 con 0x7fbfc40b3260 2022-07-20T02:25:30.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34794/0 (socket says 172.21.15.26:34794) 2022-07-20T02:25:30.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 -- 172.21.15.26:0/2947949978 learned_addr learned my addr 172.21.15.26:0/2947949978 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:30.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 0x7fbfc4093210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca0b3700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40b3260 0x7fbfc40921d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 -- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40b3260 msgr2=0x7fbfc40921d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40b3260 0x7fbfc40921d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 -- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 msgr2=0x7fbfc4093210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 0x7fbfc4093210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 -- 172.21.15.26:0/2947949978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc000b180 con 0x7fbfc40b1720 2022-07-20T02:25:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 0x7fbfc4093210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfc90b1700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbfb400c000 tx=0x7fbfb4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb4010780 con 0x7fbfc40b1720 2022-07-20T02:25:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb40108e0 con 0x7fbfc40b1720 2022-07-20T02:25:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb4010c40 con 0x7fbfc40b1720 2022-07-20T02:25:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc4150210 con 0x7fbfc40b1720 2022-07-20T02:25:30.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.587+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc414ff70 con 0x7fbfc40b1720 2022-07-20T02:25:30.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc4067210 con 0x7fbfc40b1720 2022-07-20T02:25:30.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbfb401c460 con 0x7fbfc40b1720 2022-07-20T02:25:30.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfb2ffd700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 0x7fbfa4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fbfb406c210 con 0x7fbfc40b1720 2022-07-20T02:25:30.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 0x7fbfa4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.591+0000 7fbfc98b2700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 0x7fbfa4055e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbfc0018000 tx=0x7fbfc0014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.595+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfb40432f0 con 0x7fbfc40b1720 2022-07-20T02:25:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.727+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbfc40574e0 con 0x7fbfc40b1720 2022-07-20T02:25:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.727+0000 7fbfb2ffd700 1 -- 172.21.15.26:0/2947949978 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7fbfb4046910 con 0x7fbfc40b1720 2022-07-20T02:25:30.736 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:30.737 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:25:30.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:25:30.738 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:25:30.739 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:25:30.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:25:30.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:25:30.740 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:25:30.740 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:25:30.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:25:30.741 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:25:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 msgr2=0x7fbfa4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 0x7fbfa4055e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbfc0018000 tx=0x7fbfc0014000).stop 2022-07-20T02:25:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 msgr2=0x7fbfc40b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbfb400c000 tx=0x7fbfb4006df0).stop 2022-07-20T02:25:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 shutdown_connections 2022-07-20T02:25:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40b1720 0x7fbfc40b4e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfa4053950 0x7fbfa4055e10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40b3260 0x7fbfc40921d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 --2- 172.21.15.26:0/2947949978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc40a95a0 0x7fbfc4093210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 >> 172.21.15.26:0/2947949978 conn(0x7fbfc40b67f0 msgr2=0x7fbfc4066920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 shutdown_connections 2022-07-20T02:25:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.731+0000 7fbfca8b4700 1 -- 172.21.15.26:0/2947949978 wait complete. 2022-07-20T02:25:30.755 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:25:30.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:25:30.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 Processor -- start 2022-07-20T02:25:30.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- start start 2022-07-20T02:25:30.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0678147000 0x7f0678143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0678060f00 0x7f067813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06781474e0 con 0x7f067805cfd0 2022-07-20T02:25:30.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780b4e40 con 0x7f0678147000 2022-07-20T02:25:30.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067805d3b0 con 0x7f0678060f00 2022-07-20T02:25:30.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0678147000 0x7f0678143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0678147000 0x7f0678143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34800/0 (socket says 172.21.15.26:34800) 2022-07-20T02:25:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 -- 172.21.15.26:0/411107410 learned_addr learned my addr 172.21.15.26:0/411107410 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0676d9d700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0678060f00 0x7f067813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 -- 172.21.15.26:0/411107410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0678060f00 msgr2=0x7f067813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0678060f00 0x7f067813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 -- 172.21.15.26:0/411107410 >> v1:172.21.15.120:6789/0 conn(0x7f067805cfd0 legacy=0x7f06780b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 -- 172.21.15.26:0/411107410 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06781431e0 con 0x7f0678147000 2022-07-20T02:25:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f0675d9b700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678147000 0x7f0678143bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f066c002f70 tx=0x7f066c006eb0).ready entity=mon.0 client_cookie=d6ac38872eb91a5c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067559a700 1 -- 172.21.15.26:0/411107410 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c00d5c0 con 0x7f0678147000 2022-07-20T02:25:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067559a700 1 -- 172.21.15.26:0/411107410 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c000e90 con 0x7f0678147000 2022-07-20T02:25:30.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067559a700 1 -- 172.21.15.26:0/411107410 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c00d950 con 0x7f0678147000 2022-07-20T02:25:30.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- 172.21.15.26:0/411107410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678147000 msgr2=0x7f0678143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678147000 0x7f0678143bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f066c002f70 tx=0x7f066c006eb0).stop 2022-07-20T02:25:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- 172.21.15.26:0/411107410 shutdown_connections 2022-07-20T02:25:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0678060f00 0x7f067813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 --2- 172.21.15.26:0/411107410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678147000 0x7f0678143bc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- 172.21.15.26:0/411107410 >> 172.21.15.26:0/411107410 conn(0x7f06780620c0 msgr2=0x7f067805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- 172.21.15.26:0/411107410 shutdown_connections 2022-07-20T02:25:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.867+0000 7f067759e700 1 -- 172.21.15.26:0/411107410 wait complete. 2022-07-20T02:25:30.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 Processor -- start 2022-07-20T02:25:30.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- start start 2022-07-20T02:25:30.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0678147000 0x7f06780b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06780ba200 0x7f06781522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06780af780 con 0x7f0678060f00 2022-07-20T02:25:30.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678156390 con 0x7f0678147000 2022-07-20T02:25:30.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780b3400 con 0x7f06780ba200 2022-07-20T02:25:30.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34806/0 (socket says 172.21.15.26:34806) 2022-07-20T02:25:30.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 -- 172.21.15.26:0/358514747 learned_addr learned my addr 172.21.15.26:0/358514747 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:30.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f0675d9b700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0678147000 0x7f06780b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f0676d9d700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06780ba200 0x7f06781522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 -- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06780ba200 msgr2=0x7f06781522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06780ba200 0x7f06781522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 -- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0678147000 msgr2=0x7f06780b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0678147000 0x7f06780b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 -- 172.21.15.26:0/358514747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c00b1e0 con 0x7f0678060f00 2022-07-20T02:25:30.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067659c700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f0678068660 tx=0x7f0668007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0668012400 con 0x7f0678060f00 2022-07-20T02:25:30.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0668012560 con 0x7f0678060f00 2022-07-20T02:25:30.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06780b5060 con 0x7f0678060f00 2022-07-20T02:25:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06680128c0 con 0x7f0678060f00 2022-07-20T02:25:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.871+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06780afbb0 con 0x7f0678060f00 2022-07-20T02:25:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06780af910 con 0x7f0678060f00 2022-07-20T02:25:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0668019030 con 0x7f0678060f00 2022-07-20T02:25:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f06677fe700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 0x7f0654055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f066806ae80 con 0x7f0678060f00 2022-07-20T02:25:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f0675d9b700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 0x7f0654055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.875+0000 7f0675d9b700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 0x7f0654055ff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f066c01c000 tx=0x7f066c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:30.883+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06780af910 con 0x7f0678060f00 2022-07-20T02:25:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.007+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f06780af910 con 0x7f0678060f00 2022-07-20T02:25:31.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.607+0000 7f06677fe700 1 -- 172.21.15.26:0/358514747 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f06780af910 con 0x7f0678060f00 2022-07-20T02:25:31.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 msgr2=0x7f0654055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:31.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 0x7f0654055ff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f066c01c000 tx=0x7f066c015000).stop 2022-07-20T02:25:31.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 msgr2=0x7f0678140170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:31.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f0678068660 tx=0x7f0668007c50).stop 2022-07-20T02:25:31.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 shutdown_connections 2022-07-20T02:25:31.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0654053b30 0x7f0654055ff0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06780ba200 0x7f06781522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0678147000 0x7f06780b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 --2- 172.21.15.26:0/358514747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0678060f00 0x7f0678140170 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 >> 172.21.15.26:0/358514747 conn(0x7f067805cfd0 msgr2=0x7f0678138f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:31.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 shutdown_connections 2022-07-20T02:25:31.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.615+0000 7f067759e700 1 -- 172.21.15.26:0/358514747 wait complete. 2022-07-20T02:25:31.626 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:25:31.635 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2022-07-20T02:25:31.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2022-07-20T02:25:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 Processor -- start 2022-07-20T02:25:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 -- start start 2022-07-20T02:25:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd3200ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd320060850 0x7fd3200abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 0x7fd3201468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200610e0 con 0x7fd320060850 2022-07-20T02:25:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200b2ea0 con 0x7fd3200ac260 2022-07-20T02:25:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200a8e20 con 0x7fd3200b2ac0 2022-07-20T02:25:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd320060850 0x7fd3200abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd320060850 0x7fd3200abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34814/0 (socket says 172.21.15.26:34814) 2022-07-20T02:25:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 -- 172.21.15.26:0/3516387765 learned_addr learned my addr 172.21.15.26:0/3516387765 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd324baf700 1 --2- 172.21.15.26:0/3516387765 >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 0x7fd3201468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31ffff700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd3200ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 -- 172.21.15.26:0/3516387765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 msgr2=0x7fd3200ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd3200ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 -- 172.21.15.26:0/3516387765 >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 msgr2=0x7fd3201468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3516387765 >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 0x7fd3201468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 -- 172.21.15.26:0/3516387765 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200a95e0 con 0x7fd320060850 2022-07-20T02:25:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd324baf700 1 --2- 172.21.15.26:0/3516387765 >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 0x7fd3201468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd3200abd70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd310002f70 tx=0x7fd310006eb0).ready entity=mon.0 client_cookie=21a0706c51e86902 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31effd700 1 -- 172.21.15.26:0/3516387765 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd31000d570 con 0x7fd320060850 2022-07-20T02:25:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31effd700 1 -- 172.21.15.26:0/3516387765 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310000e90 con 0x7fd320060850 2022-07-20T02:25:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd31effd700 1 -- 172.21.15.26:0/3516387765 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd31000d900 con 0x7fd320060850 2022-07-20T02:25:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 -- 172.21.15.26:0/3516387765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 msgr2=0x7fd3200abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.783+0000 7fd326612700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd3200abd70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd310002f70 tx=0x7fd310006eb0).stop 2022-07-20T02:25:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3516387765 shutdown_connections 2022-07-20T02:25:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- 172.21.15.26:0/3516387765 >> v2:172.21.15.120:3300/0 conn(0x7fd3200ac260 0x7fd3201468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd3200abd70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- 172.21.15.26:0/3516387765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd3200ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3516387765 >> 172.21.15.26:0/3516387765 conn(0x7fd3200b60a0 msgr2=0x7fd3200b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3516387765 shutdown_connections 2022-07-20T02:25:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3516387765 wait complete. 2022-07-20T02:25:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 Processor -- start 2022-07-20T02:25:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- start start 2022-07-20T02:25:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 0x7fd32013f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd32013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b1f40 con 0x7fd320060850 2022-07-20T02:25:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320152890 con 0x7fd3200ac260 2022-07-20T02:25:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd320152b30 con 0x7fd3200b2ac0 2022-07-20T02:25:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34820/0 (socket says 172.21.15.26:34820) 2022-07-20T02:25:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 -- 172.21.15.26:0/3868515598 learned_addr learned my addr 172.21.15.26:0/3868515598 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd324baf700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd32013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 0x7fd32013f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 -- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 msgr2=0x7fd32013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd32013fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 -- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 msgr2=0x7fd32013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 0x7fd32013f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 -- 172.21.15.26:0/3868515598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318009470 con 0x7fd320060850 2022-07-20T02:25:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 0x7fd32013f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:25:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31ffff700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd31400c000 tx=0x7fd314006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd314010780 con 0x7fd320060850 2022-07-20T02:25:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3140108e0 con 0x7fd320060850 2022-07-20T02:25:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31000b1e0 con 0x7fd320060850 2022-07-20T02:25:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd314010c40 con 0x7fd320060850 2022-07-20T02:25:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.787+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3200ae340 con 0x7fd320060850 2022-07-20T02:25:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd320060d60 con 0x7fd320060850 2022-07-20T02:25:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd31401c460 con 0x7fd320060850 2022-07-20T02:25:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd31d7fa700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 0x7fd300055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fd31406c190 con 0x7fd320060850 2022-07-20T02:25:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 0x7fd300055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.791+0000 7fd31f7fe700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 0x7fd300055f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd310017000 tx=0x7fd310015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.795+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd314043220 con 0x7fd320060850 2022-07-20T02:25:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:31.931+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fd3200574e0 con 0x7fd320060850 2022-07-20T02:25:32.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.611+0000 7fd31d7fa700 1 -- 172.21.15.26:0/3868515598 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 154+0+0 (secure 0 0 0) 0x7fd314046840 con 0x7fd320060850 2022-07-20T02:25:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 msgr2=0x7fd300055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 0x7fd300055f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd310017000 tx=0x7fd310015000).stop 2022-07-20T02:25:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 msgr2=0x7fd320142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd31400c000 tx=0x7fd314006df0).stop 2022-07-20T02:25:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 shutdown_connections 2022-07-20T02:25:32.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd300053aa0 0x7fd300055f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200b2ac0 0x7fd32013fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200ac260 0x7fd32013f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 --2- 172.21.15.26:0/3868515598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd320060850 0x7fd320142770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 >> 172.21.15.26:0/3868515598 conn(0x7fd3200b60a0 msgr2=0x7fd320143400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:32.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 shutdown_connections 2022-07-20T02:25:32.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.615+0000 7fd326612700 1 -- 172.21.15.26:0/3868515598 wait complete. 2022-07-20T02:25:32.633 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2022-07-20T02:25:32.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.735+0000 7f9f89a1c700 1 Processor -- start 2022-07-20T02:25:32.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.735+0000 7f9f89a1c700 1 -- start start 2022-07-20T02:25:32.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f89a1c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 0x7f9f84067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f89a1c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9f84060250 0x7f9f84060700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f89a1c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f840b0f50 con 0x7f9f840aff40 2022-07-20T02:25:32.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f89a1c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8413e7f0 con 0x7f9f84060250 2022-07-20T02:25:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f89a1c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840b0320 con 0x7f9f840621f0 2022-07-20T02:25:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f88a1a700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9f840aff40 0x7f9f84066b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34108/0 (socket says 172.21.15.26:34108) 2022-07-20T02:25:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f88a1a700 1 -- 172.21.15.26:0/1591074418 learned_addr learned my addr 172.21.15.26:0/1591074418 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1591074418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 0x7f9f84067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f8921b700 1 --2- 172.21.15.26:0/1591074418 >> v2:172.21.15.26:3300/0 conn(0x7f9f84060250 0x7f9f84060700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 -- 172.21.15.26:0/1591074418 >> v2:172.21.15.26:3300/0 conn(0x7f9f84060250 msgr2=0x7f9f84060700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1591074418 >> v2:172.21.15.26:3300/0 conn(0x7f9f84060250 0x7f9f84060700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 -- 172.21.15.26:0/1591074418 >> v1:172.21.15.26:6789/0 conn(0x7f9f840aff40 legacy=0x7f9f84066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f837fe700 1 -- 172.21.15.26:0/1591074418 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569069193 0 0) 0x7f9f840b0f50 con 0x7f9f840aff40 2022-07-20T02:25:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 -- 172.21.15.26:0/1591074418 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8405f8e0 con 0x7f9f840621f0 2022-07-20T02:25:32.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.739+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1591074418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 0x7f9f84067240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9f70002f70 tx=0x7f9f70006df0).ready entity=mon.2 client_cookie=dbde7af5673e9d7c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:32.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f837fe700 1 -- 172.21.15.26:0/1591074418 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7000d570 con 0x7f9f840621f0 2022-07-20T02:25:32.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f837fe700 1 -- 172.21.15.26:0/1591074418 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70000e90 con 0x7f9f840621f0 2022-07-20T02:25:32.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f837fe700 1 -- 172.21.15.26:0/1591074418 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7000d8c0 con 0x7f9f840621f0 2022-07-20T02:25:32.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1591074418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 msgr2=0x7f9f84067240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1591074418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 0x7f9f84067240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9f70002f70 tx=0x7f9f70006df0).stop 2022-07-20T02:25:32.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1591074418 shutdown_connections 2022-07-20T02:25:32.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1591074418 >> v2:172.21.15.26:3300/0 conn(0x7f9f84060250 0x7f9f84060700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1591074418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840621f0 0x7f9f84067240 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1591074418 >> 172.21.15.26:0/1591074418 conn(0x7f9f840b3050 msgr2=0x7f9f840b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:32.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1591074418 shutdown_connections 2022-07-20T02:25:32.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.743+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1591074418 wait complete. 2022-07-20T02:25:32.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 Processor -- start 2022-07-20T02:25:32.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- start start 2022-07-20T02:25:32.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f84060250 0x7f9f8413efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840aff40 0x7f9f8413bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840b0f20 con 0x7f9f840621f0 2022-07-20T02:25:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8414ffb0 con 0x7f9f84060250 2022-07-20T02:25:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840ad0b0 con 0x7f9f840aff40 2022-07-20T02:25:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34836/0 (socket says 172.21.15.26:34836) 2022-07-20T02:25:32.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 -- 172.21.15.26:0/1488446156 learned_addr learned my addr 172.21.15.26:0/1488446156 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:32.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f88a1a700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f84060250 0x7f9f8413efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f8921b700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840aff40 0x7f9f8413bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 -- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840aff40 msgr2=0x7f9f8413bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840aff40 0x7f9f8413bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 -- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f84060250 msgr2=0x7f9f8413efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:32.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f84060250 0x7f9f8413efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:32.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 -- 172.21.15.26:0/1488446156 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7000b180 con 0x7f9f840621f0 2022-07-20T02:25:32.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f83fff700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9f70002f70 tx=0x7f9f70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:32.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7000de10 con 0x7f9f840621f0 2022-07-20T02:25:32.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70011ce0 con 0x7f9f840621f0 2022-07-20T02:25:32.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7001e650 con 0x7f9f840621f0 2022-07-20T02:25:32.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f84153700 con 0x7f9f840621f0 2022-07-20T02:25:32.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.747+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f840b1470 con 0x7f9f840621f0 2022-07-20T02:25:32.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f840ad2b0 con 0x7f9f840621f0 2022-07-20T02:25:32.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9f7001a040 con 0x7f9f840621f0 2022-07-20T02:25:32.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f81ffb700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 0x7f9f64055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:32.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5863+0+0 (secure 0 0 0) 0x7f9f7006cfe0 con 0x7f9f840621f0 2022-07-20T02:25:32.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f88a1a700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 0x7f9f64055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.751+0000 7f9f88a1a700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 0x7f9f64055fc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9f7800b000 tx=0x7f9f78009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.755+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f70043ec0 con 0x7f9f840621f0 2022-07-20T02:25:32.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:32.879+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"} v 0) v1 -- 0x7f9f8405fb10 con 0x7f9f840621f0 2022-07-20T02:25:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.615+0000 7f9f81ffb700 1 -- 172.21.15.26:0/1488446156 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"}]=0 pool 'cephfs_data' created v22) v1 ==== 188+0+0 (secure 0 0 0) 0x7f9f70023020 con 0x7f9f840621f0 2022-07-20T02:25:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 msgr2=0x7f9f64055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 0x7f9f64055fc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9f7800b000 tx=0x7f9f78009040).stop 2022-07-20T02:25:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 msgr2=0x7f9f8413b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9f70002f70 tx=0x7f9f70015000).stop 2022-07-20T02:25:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 shutdown_connections 2022-07-20T02:25:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f840621f0 0x7f9f8413b900 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0x7f9f70002f70 tx=0x7f9f70015000).stop 2022-07-20T02:25:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f64053b00 0x7f9f64055fc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f840aff40 0x7f9f8413bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 --2- 172.21.15.26:0/1488446156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f84060250 0x7f9f8413efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.623+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 >> 172.21.15.26:0/1488446156 conn(0x7f9f840b3050 msgr2=0x7f9f84140080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.627+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 shutdown_connections 2022-07-20T02:25:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.627+0000 7f9f89a1c700 1 -- 172.21.15.26:0/1488446156 wait complete. 2022-07-20T02:25:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:25:34.642 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2022-07-20T02:25:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.751+0000 7f8a8d98b700 1 Processor -- start 2022-07-20T02:25:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.751+0000 7f8a8d98b700 1 -- start start 2022-07-20T02:25:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a8d98b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a880b14b0 0x7f8a880b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a8d98b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880614b0 0x7f8a88061960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a8d98b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880ae370 con 0x7f8a880adf90 2022-07-20T02:25:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a8d98b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a880b28b0 con 0x7f8a880b14b0 2022-07-20T02:25:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a8d98b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880b0610 con 0x7f8a880614b0 2022-07-20T02:25:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a877fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880614b0 0x7f8a88061960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a880b14b0 0x7f8a880b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a880b14b0 0x7f8a880b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:54538/0 (socket says 172.21.15.26:54538) 2022-07-20T02:25:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 -- 172.21.15.26:0/2734253807 learned_addr learned my addr 172.21.15.26:0/2734253807 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 -- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880614b0 msgr2=0x7f8a88061960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 --2- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880614b0 0x7f8a88061960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 -- 172.21.15.26:0/2734253807 >> v1:172.21.15.120:6789/0 conn(0x7f8a880adf90 legacy=0x7f8a88060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 -- 172.21.15.26:0/2734253807 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880b08b0 con 0x7f8a880b14b0 2022-07-20T02:25:34.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a7ffff700 1 --2- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880b14b0 0x7f8a880b2c60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f8a880b1900 tx=0x7f8a70006eb0).ready entity=mon.1 client_cookie=18b4e47f3537a65a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:34.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a867fc700 1 -- 172.21.15.26:0/2734253807 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7000d5c0 con 0x7f8a880b14b0 2022-07-20T02:25:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a867fc700 1 -- 172.21.15.26:0/2734253807 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70000e90 con 0x7f8a880b14b0 2022-07-20T02:25:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.755+0000 7f8a867fc700 1 -- 172.21.15.26:0/2734253807 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7000d950 con 0x7f8a880b14b0 2022-07-20T02:25:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880b14b0 msgr2=0x7f8a880b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880b14b0 0x7f8a880b2c60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f8a880b1900 tx=0x7f8a70006eb0).stop 2022-07-20T02:25:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- 172.21.15.26:0/2734253807 shutdown_connections 2022-07-20T02:25:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880614b0 0x7f8a88061960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/2734253807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880b14b0 0x7f8a880b2c60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- 172.21.15.26:0/2734253807 >> 172.21.15.26:0/2734253807 conn(0x7f8a880b66e0 msgr2=0x7f8a880b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- 172.21.15.26:0/2734253807 shutdown_connections 2022-07-20T02:25:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- 172.21.15.26:0/2734253807 wait complete. 2022-07-20T02:25:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 Processor -- start 2022-07-20T02:25:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.759+0000 7f8a8d98b700 1 -- start start 2022-07-20T02:25:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880614b0 0x7f8a880b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880b14b0 0x7f8a88145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88066be0 con 0x7f8a880adf90 2022-07-20T02:25:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880a2520 con 0x7f8a880614b0 2022-07-20T02:25:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880a27c0 con 0x7f8a880b14b0 2022-07-20T02:25:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34852/0 (socket says 172.21.15.26:34852) 2022-07-20T02:25:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 -- 172.21.15.26:0/1147647980 learned_addr learned my addr 172.21.15.26:0/1147647980 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a86ffd700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880614b0 0x7f8a880b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a877fe700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880b14b0 0x7f8a88145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 -- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880b14b0 msgr2=0x7f8a88145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880b14b0 0x7f8a88145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 -- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880614b0 msgr2=0x7f8a880b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880614b0 0x7f8a880b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 -- 172.21.15.26:0/1147647980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7000b1e0 con 0x7f8a880adf90 2022-07-20T02:25:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a7ffff700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a880b1900 tx=0x7f8a70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a70021070 con 0x7f8a880adf90 2022-07-20T02:25:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70002f70 con 0x7f8a880adf90 2022-07-20T02:25:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a880a2a60 con 0x7f8a880adf90 2022-07-20T02:25:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a70011cf0 con 0x7f8a880adf90 2022-07-20T02:25:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.763+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a880a0310 con 0x7f8a880adf90 2022-07-20T02:25:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a880574e0 con 0x7f8a880adf90 2022-07-20T02:25:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a7000f070 con 0x7f8a880adf90 2022-07-20T02:25:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a84ff9700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 0x7f8a6c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6315+0+0 (secure 0 0 0) 0x7f8a7006d650 con 0x7f8a880adf90 2022-07-20T02:25:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a86ffd700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 0x7f8a6c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.767+0000 7f8a86ffd700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 0x7f8a6c055f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f8a7800d000 tx=0x7f8a78007b50).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:34.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.775+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a700442f0 con 0x7f8a880adf90 2022-07-20T02:25:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:34.907+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f8a880a22e0 con 0x7f8a880adf90 2022-07-20T02:25:35.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.619+0000 7f8a84ff9700 1 -- 172.21.15.26:0/1147647980 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8a70047910 con 0x7f8a880adf90 2022-07-20T02:25:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 msgr2=0x7f8a6c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 0x7f8a6c055f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f8a7800d000 tx=0x7f8a78007b50).stop 2022-07-20T02:25:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 msgr2=0x7f8a88145a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a880b1900 tx=0x7f8a70015000).stop 2022-07-20T02:25:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 shutdown_connections 2022-07-20T02:25:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a880adf90 0x7f8a88145a60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a6c053a70 0x7f8a6c055f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a880b14b0 0x7f8a88145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 --2- 172.21.15.26:0/1147647980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a880614b0 0x7f8a880b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 >> 172.21.15.26:0/1147647980 conn(0x7f8a880b66e0 msgr2=0x7f8a880aa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 shutdown_connections 2022-07-20T02:25:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.627+0000 7f8a8d98b700 1 -- 172.21.15.26:0/1147647980 wait complete. 2022-07-20T02:25:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 4 allow_ec_overwrites to true 2022-07-20T02:25:35.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:25:35.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.755+0000 7f70965f1700 1 Processor -- start 2022-07-20T02:25:35.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.755+0000 7f70965f1700 1 -- start start 2022-07-20T02:25:35.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f70901465b0 0x7f7090066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70900b24c0 0x7f70900604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70900b1ae0 con 0x7f70900672a0 2022-07-20T02:25:35.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70900b2e20 con 0x7f70900b24c0 2022-07-20T02:25:35.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70900b1d80 con 0x7f70901465b0 2022-07-20T02:25:35.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70900b24c0 0x7f70900604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70900b24c0 0x7f70900604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34860/0 (socket says 172.21.15.26:34860) 2022-07-20T02:25:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 -- 172.21.15.26:0/2084009726 learned_addr learned my addr 172.21.15.26:0/2084009726 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70955ef700 1 --2- 172.21.15.26:0/2084009726 >> v2:172.21.15.120:3300/0 conn(0x7f70901465b0 0x7f7090066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 -- 172.21.15.26:0/2084009726 >> v2:172.21.15.120:3300/0 conn(0x7f70901465b0 msgr2=0x7f7090066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:35.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 --2- 172.21.15.26:0/2084009726 >> v2:172.21.15.120:3300/0 conn(0x7f70901465b0 0x7f7090066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 -- 172.21.15.26:0/2084009726 >> v1:172.21.15.120:6789/0 conn(0x7f70900672a0 legacy=0x7f7090146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 -- 172.21.15.26:0/2084009726 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70900b2020 con 0x7f70900b24c0 2022-07-20T02:25:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f7094dee700 1 --2- 172.21.15.26:0/2084009726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900b24c0 0x7f70900604e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7080002f70 tx=0x7f7080006eb0).ready entity=mon.0 client_cookie=ba205cc1dcd4e230 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f707ffff700 1 -- 172.21.15.26:0/2084009726 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708000d5c0 con 0x7f70900b24c0 2022-07-20T02:25:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f707ffff700 1 -- 172.21.15.26:0/2084009726 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080000e90 con 0x7f70900b24c0 2022-07-20T02:25:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f707ffff700 1 -- 172.21.15.26:0/2084009726 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708000d950 con 0x7f70900b24c0 2022-07-20T02:25:35.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 -- 172.21.15.26:0/2084009726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900b24c0 msgr2=0x7f70900604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 --2- 172.21.15.26:0/2084009726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900b24c0 0x7f70900604e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7080002f70 tx=0x7f7080006eb0).stop 2022-07-20T02:25:35.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.759+0000 7f70965f1700 1 -- 172.21.15.26:0/2084009726 shutdown_connections 2022-07-20T02:25:35.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 --2- 172.21.15.26:0/2084009726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900b24c0 0x7f70900604e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 --2- 172.21.15.26:0/2084009726 >> v2:172.21.15.120:3300/0 conn(0x7f70901465b0 0x7f7090066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 -- 172.21.15.26:0/2084009726 >> 172.21.15.26:0/2084009726 conn(0x7f70900b3050 msgr2=0x7f70900b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 -- 172.21.15.26:0/2084009726 shutdown_connections 2022-07-20T02:25:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 -- 172.21.15.26:0/2084009726 wait complete. 2022-07-20T02:25:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 Processor -- start 2022-07-20T02:25:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.763+0000 7f70965f1700 1 -- start start 2022-07-20T02:25:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900672a0 0x7f70901411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70901465b0 0x7f7090064cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090156270 con 0x7f70900ace10 2022-07-20T02:25:35.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090156740 con 0x7f70900672a0 2022-07-20T02:25:35.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70901564a0 con 0x7f70901465b0 2022-07-20T02:25:35.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34868/0 (socket says 172.21.15.26:34868) 2022-07-20T02:25:35.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70955ef700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900672a0 0x7f70901411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 -- 172.21.15.26:0/4210361325 learned_addr learned my addr 172.21.15.26:0/4210361325 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7094dee700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70901465b0 0x7f7090064cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 -- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70901465b0 msgr2=0x7f7090064cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70901465b0 0x7f7090064cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 -- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900672a0 msgr2=0x7f70901411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900672a0 0x7f70901411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 -- 172.21.15.26:0/4210361325 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708000b1e0 con 0x7f70900ace10 2022-07-20T02:25:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f7095df0700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f70900b0dc0 tx=0x7f7088007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708800f470 con 0x7f70900ace10 2022-07-20T02:25:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70900aa450 con 0x7f70900ace10 2022-07-20T02:25:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708800f5d0 con 0x7f70900ace10 2022-07-20T02:25:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708800f930 con 0x7f70900ace10 2022-07-20T02:25:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.767+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70900aca70 con 0x7f70900ace10 2022-07-20T02:25:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70901419f0 con 0x7f70900ace10 2022-07-20T02:25:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7088019030 con 0x7f70900ace10 2022-07-20T02:25:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f707e7fc700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 0x7f7070055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6315+0+0 (secure 0 0 0) 0x7f708806ba10 con 0x7f70900ace10 2022-07-20T02:25:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f70955ef700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 0x7f7070055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.771+0000 7f70955ef700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 0x7f7070055fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f708c00b000 tx=0x7f708c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.779+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70880426b0 con 0x7f70900ace10 2022-07-20T02:25:35.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.943+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f70900aa5e0 con 0x7f70900ace10 2022-07-20T02:25:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.947+0000 7f707e7fc700 1 -- 172.21.15.26:0/4210361325 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v5) v1 ==== 187+0+0 (secure 0 0 0) 0x7f7088045cd0 con 0x7f70900ace10 2022-07-20T02:25:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 msgr2=0x7f7070055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 0x7f7070055fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f708c00b000 tx=0x7f708c006df0).stop 2022-07-20T02:25:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 msgr2=0x7f70900651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f70900b0dc0 tx=0x7f7088007f10).stop 2022-07-20T02:25:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 shutdown_connections 2022-07-20T02:25:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70900ace10 0x7f70900651e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7070053b00 0x7f7070055fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70901465b0 0x7f7090064cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 --2- 172.21.15.26:0/4210361325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70900672a0 0x7f70901411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 >> 172.21.15.26:0/4210361325 conn(0x7f70900b3050 msgr2=0x7f709013ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 shutdown_connections 2022-07-20T02:25:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:35.951+0000 7f70965f1700 1 -- 172.21.15.26:0/4210361325 wait complete. 2022-07-20T02:25:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 3 and data pool 4 2022-07-20T02:25:35.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 Processor -- start 2022-07-20T02:25:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 -- start start 2022-07-20T02:25:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f366405cfd0 0x7f3664143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664060f00 0x7f366413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36641474e0 con 0x7f3664147000 2022-07-20T02:25:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36640b4e40 con 0x7f366405cfd0 2022-07-20T02:25:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3663fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366405d3b0 con 0x7f3664060f00 2022-07-20T02:25:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f36627fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3664147000 0x7f36640b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34164/0 (socket says 172.21.15.26:34164) 2022-07-20T02:25:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f36627fc700 1 -- 172.21.15.26:0/516365258 learned_addr learned my addr 172.21.15.26:0/516365258 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:36.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 --2- 172.21.15.26:0/516365258 >> v2:172.21.15.120:3300/0 conn(0x7f366405cfd0 0x7f3664143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f36637fe700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664060f00 0x7f366413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3661ffb700 1 -- 172.21.15.26:0/516365258 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506054330 0 0) 0x7f36641474e0 con 0x7f3664147000 2022-07-20T02:25:36.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3661ffb700 1 -- 172.21.15.26:0/516365258 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3644003260 con 0x7f3664147000 2022-07-20T02:25:36.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 -- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664060f00 msgr2=0x7f366413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:36.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664060f00 0x7f366413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 -- 172.21.15.26:0/516365258 >> v1:172.21.15.26:6789/0 conn(0x7f3664147000 legacy=0x7f36640b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 -- 172.21.15.26:0/516365258 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36641431e0 con 0x7f366405cfd0 2022-07-20T02:25:36.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.063+0000 7f3662ffd700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f366405cfd0 0x7f3664143bc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f365c0032f0 tx=0x7f365c0091d0).ready entity=mon.1 client_cookie=9c00568b4464d263 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3661ffb700 1 -- 172.21.15.26:0/516365258 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365c00b600 con 0x7f366405cfd0 2022-07-20T02:25:36.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3661ffb700 1 -- 172.21.15.26:0/516365258 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c00b760 con 0x7f366405cfd0 2022-07-20T02:25:36.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3661ffb700 1 -- 172.21.15.26:0/516365258 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365c00b8d0 con 0x7f366405cfd0 2022-07-20T02:25:36.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f366405cfd0 msgr2=0x7f3664143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f366405cfd0 0x7f3664143bc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f365c0032f0 tx=0x7f365c0091d0).stop 2022-07-20T02:25:36.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- 172.21.15.26:0/516365258 shutdown_connections 2022-07-20T02:25:36.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664060f00 0x7f366413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 --2- 172.21.15.26:0/516365258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f366405cfd0 0x7f3664143bc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- 172.21.15.26:0/516365258 >> 172.21.15.26:0/516365258 conn(0x7f36640620c0 msgr2=0x7f366405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:36.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- 172.21.15.26:0/516365258 shutdown_connections 2022-07-20T02:25:36.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- 172.21.15.26:0/516365258 wait complete. 2022-07-20T02:25:36.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 Processor -- start 2022-07-20T02:25:36.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.067+0000 7f3663fff700 1 -- start start 2022-07-20T02:25:36.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664147000 0x7f3664137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 0x7f36641381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36640ba240 con 0x7f3664060f00 2022-07-20T02:25:36.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36641386a0 con 0x7f3664146270 2022-07-20T02:25:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36640b0a30 con 0x7f3664147000 2022-07-20T02:25:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34888/0 (socket says 172.21.15.26:34888) 2022-07-20T02:25:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 -- 172.21.15.26:0/3452680242 learned_addr learned my addr 172.21.15.26:0/3452680242 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f36637fe700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 0x7f36641381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f36627fc700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664147000 0x7f3664137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 -- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664147000 msgr2=0x7f3664137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664147000 0x7f3664137cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 -- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 msgr2=0x7f36641381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 0x7f36641381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 -- 172.21.15.26:0/3452680242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009470 con 0x7f3664060f00 2022-07-20T02:25:36.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f36637fe700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 0x7f36641381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:36.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3662ffd700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f365c0032f0 tx=0x7f365c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365c00b8d0 con 0x7f3664060f00 2022-07-20T02:25:36.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c01f5e0 con 0x7f3664060f00 2022-07-20T02:25:36.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36640b4320 con 0x7f3664060f00 2022-07-20T02:25:36.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365c00b160 con 0x7f3664060f00 2022-07-20T02:25:36.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.071+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3664144ff0 con 0x7f3664060f00 2022-07-20T02:25:36.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f365c026030 con 0x7f3664060f00 2022-07-20T02:25:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3664144e30 con 0x7f3664060f00 2022-07-20T02:25:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f365bfff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 0x7f3648055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f365c06ce60 con 0x7f3664060f00 2022-07-20T02:25:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f36627fc700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 0x7f3648055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.075+0000 7f36627fc700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 0x7f3648055fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f364c00b000 tx=0x7f364c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.079+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f365c047540 con 0x7f3664060f00 2022-07-20T02:25:36.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.207+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3664143480 con 0x7f3664060f00 2022-07-20T02:25:36.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.207+0000 7f365bfff700 1 -- 172.21.15.26:0/3452680242 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f365c024090 con 0x7f3664060f00 2022-07-20T02:25:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:36.216 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:36.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 msgr2=0x7f3648055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 0x7f3648055fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f364c00b000 tx=0x7f364c009040).stop 2022-07-20T02:25:36.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 msgr2=0x7f3664142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f365c0032f0 tx=0x7f365c016000).stop 2022-07-20T02:25:36.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 shutdown_connections 2022-07-20T02:25:36.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3648053b00 0x7f3648055fc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3664146270 0x7f36641381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3664147000 0x7f3664137cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 --2- 172.21.15.26:0/3452680242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3664060f00 0x7f3664142360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 >> 172.21.15.26:0/3452680242 conn(0x7f366405cfd0 msgr2=0x7f366413b160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:36.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 shutdown_connections 2022-07-20T02:25:36.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.211+0000 7f3663fff700 1 -- 172.21.15.26:0/3452680242 wait complete. 2022-07-20T02:25:36.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 Processor -- start 2022-07-20T02:25:36.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 -- start start 2022-07-20T02:25:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0100a7fc0 0x7fe0100af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0100b62d0 con 0x7fe0100b5ef0 2022-07-20T02:25:36.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01014aac0 con 0x7fe010147be0 2022-07-20T02:25:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe01487b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01014ad60 con 0x7fe0100a7fc0 2022-07-20T02:25:36.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe00f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0100b5ef0 0x7fe0101474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34176/0 (socket says 172.21.15.26:34176) 2022-07-20T02:25:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe00f7fe700 1 -- 172.21.15.26:0/2325593405 learned_addr learned my addr 172.21.15.26:0/2325593405 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe00ffff700 1 --2- 172.21.15.26:0/2325593405 >> v2:172.21.15.120:3300/0 conn(0x7fe0100a7fc0 0x7fe0100af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe00e7fc700 1 -- 172.21.15.26:0/2325593405 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625374030 0 0) 0x7fe0100b62d0 con 0x7fe0100b5ef0 2022-07-20T02:25:36.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.335+0000 7fe00e7fc700 1 -- 172.21.15.26:0/2325593405 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc003260 con 0x7fe0100b5ef0 2022-07-20T02:25:36.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00ffff700 1 -- 172.21.15.26:0/2325593405 >> v1:172.21.15.120:6789/0 conn(0x7fe010147be0 legacy=0x7fe0100aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:36.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00ffff700 1 -- 172.21.15.26:0/2325593405 >> v1:172.21.15.26:6789/0 conn(0x7fe0100b5ef0 legacy=0x7fe0101474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00ffff700 1 -- 172.21.15.26:0/2325593405 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01014b000 con 0x7fe0100a7fc0 2022-07-20T02:25:36.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00ffff700 1 --2- 172.21.15.26:0/2325593405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100a7fc0 0x7fe0100af550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe008002f70 tx=0x7fe008006df0).ready entity=mon.1 client_cookie=5149806d9bf88221 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00e7fc700 1 -- 172.21.15.26:0/2325593405 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe00800d610 con 0x7fe0100a7fc0 2022-07-20T02:25:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00e7fc700 1 -- 172.21.15.26:0/2325593405 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008000e90 con 0x7fe0100a7fc0 2022-07-20T02:25:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe00e7fc700 1 -- 172.21.15.26:0/2325593405 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe00800d960 con 0x7fe0100a7fc0 2022-07-20T02:25:36.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- 172.21.15.26:0/2325593405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100a7fc0 msgr2=0x7fe0100af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 --2- 172.21.15.26:0/2325593405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100a7fc0 0x7fe0100af550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe008002f70 tx=0x7fe008006df0).stop 2022-07-20T02:25:36.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- 172.21.15.26:0/2325593405 shutdown_connections 2022-07-20T02:25:36.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 --2- 172.21.15.26:0/2325593405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100a7fc0 0x7fe0100af550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- 172.21.15.26:0/2325593405 >> 172.21.15.26:0/2325593405 conn(0x7fe010060310 msgr2=0x7fe0100b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:36.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- 172.21.15.26:0/2325593405 shutdown_connections 2022-07-20T02:25:36.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- 172.21.15.26:0/2325593405 wait complete. 2022-07-20T02:25:36.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 Processor -- start 2022-07-20T02:25:36.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- start start 2022-07-20T02:25:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 0x7fe010159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100b5ef0 0x7fe010066850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01015a8f0 con 0x7fe010147be0 2022-07-20T02:25:36.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100651c0 con 0x7fe0100b5ef0 2022-07-20T02:25:36.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.339+0000 7fe01487b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01015a650 con 0x7fe0100a7fc0 2022-07-20T02:25:36.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34906/0 (socket says 172.21.15.26:34906) 2022-07-20T02:25:36.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 -- 172.21.15.26:0/3796774410 learned_addr learned my addr 172.21.15.26:0/3796774410 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:36.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00f7fe700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 0x7fe010159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00effd700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100b5ef0 0x7fe010066850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 -- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 msgr2=0x7fe010159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 0x7fe010159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 -- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100b5ef0 msgr2=0x7fe010066850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100b5ef0 0x7fe010066850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 -- 172.21.15.26:0/3796774410 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00800b180 con 0x7fe010147be0 2022-07-20T02:25:36.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00f7fe700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 0x7fe010159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:36.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00ffff700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe008002f70 tx=0x7fe008014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe008013de0 con 0x7fe010147be0 2022-07-20T02:25:36.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00800d290 con 0x7fe010147be0 2022-07-20T02:25:36.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010065420 con 0x7fe010147be0 2022-07-20T02:25:36.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe00801e5f0 con 0x7fe010147be0 2022-07-20T02:25:36.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe010062be0 con 0x7fe010147be0 2022-07-20T02:25:36.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.343+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0100655b0 con 0x7fe010147be0 2022-07-20T02:25:36.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.347+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe008025030 con 0x7fe010147be0 2022-07-20T02:25:36.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.347+0000 7fe00cff9700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 0x7fdff0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.347+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe00806ced0 con 0x7fe010147be0 2022-07-20T02:25:36.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.347+0000 7fe00f7fe700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 0x7fdff0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.347+0000 7fe00f7fe700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 0x7fdff0055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe00400d000 tx=0x7fe004009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.355+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe008043b80 con 0x7fe010147be0 2022-07-20T02:25:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.483+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe01014b2a0 con 0x7fe010147be0 2022-07-20T02:25:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.483+0000 7fe00cff9700 1 -- 172.21.15.26:0/3796774410 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7fe0080471a0 con 0x7fe010147be0 2022-07-20T02:25:36.505 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:36.506 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:36.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 msgr2=0x7fdff0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 0x7fdff0055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe00400d000 tx=0x7fe004009040).stop 2022-07-20T02:25:36.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 msgr2=0x7fe0100626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe008002f70 tx=0x7fe008014000).stop 2022-07-20T02:25:36.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 shutdown_connections 2022-07-20T02:25:36.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe010147be0 0x7fe0100626f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdff0053b00 0x7fdff0055fc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0100b5ef0 0x7fe010066850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 --2- 172.21.15.26:0/3796774410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0100a7fc0 0x7fe010159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 >> 172.21.15.26:0/3796774410 conn(0x7fe010060310 msgr2=0x7fe0100b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:36.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 shutdown_connections 2022-07-20T02:25:36.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.487+0000 7fe01487b700 1 -- 172.21.15.26:0/3796774410 wait complete. 2022-07-20T02:25:36.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:25:36.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.675+0000 7fb69d34d700 1 Processor -- start 2022-07-20T02:25:36.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.675+0000 7fb69d34d700 1 -- start start 2022-07-20T02:25:36.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb698065980 0x7fb698065d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 0x7fb698144e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69805d220 con 0x7fb69813dd40 2022-07-20T02:25:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6980b35d0 con 0x7fb698065980 2022-07-20T02:25:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6980b3870 con 0x7fb6981449d0 2022-07-20T02:25:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb698065980 0x7fb698065d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb6977fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb69813dd40 0x7fb69813e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34194/0 (socket says 172.21.15.26:34194) 2022-07-20T02:25:36.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb6977fe700 1 -- 172.21.15.26:0/846776166 learned_addr learned my addr 172.21.15.26:0/846776166 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:36.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb6967fc700 1 --2- 172.21.15.26:0/846776166 >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 0x7fb698144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb695ffb700 1 -- 172.21.15.26:0/846776166 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468752826 0 0) 0x7fb69805d220 con 0x7fb69813dd40 2022-07-20T02:25:36.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb695ffb700 1 -- 172.21.15.26:0/846776166 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678003260 con 0x7fb69813dd40 2022-07-20T02:25:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 -- 172.21.15.26:0/846776166 >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 msgr2=0x7fb698144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 --2- 172.21.15.26:0/846776166 >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 0x7fb698144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 -- 172.21.15.26:0/846776166 >> v1:172.21.15.26:6789/0 conn(0x7fb69813dd40 legacy=0x7fb69813e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 -- 172.21.15.26:0/846776166 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb698144530 con 0x7fb698065980 2022-07-20T02:25:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb6967fc700 1 --2- 172.21.15.26:0/846776166 >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 0x7fb698144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb696ffd700 1 --2- 172.21.15.26:0/846776166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb698065980 0x7fb698065d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb698065590 tx=0x7fb680006df0).ready entity=mon.0 client_cookie=28cc5b385dc0f4fb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb695ffb700 1 -- 172.21.15.26:0/846776166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb68000d4f0 con 0x7fb698065980 2022-07-20T02:25:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb695ffb700 1 -- 172.21.15.26:0/846776166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb680000e90 con 0x7fb698065980 2022-07-20T02:25:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb695ffb700 1 -- 172.21.15.26:0/846776166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb68000d840 con 0x7fb698065980 2022-07-20T02:25:36.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- 172.21.15.26:0/846776166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb698065980 msgr2=0x7fb698065d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 --2- 172.21.15.26:0/846776166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb698065980 0x7fb698065d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb698065590 tx=0x7fb680006df0).stop 2022-07-20T02:25:36.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- 172.21.15.26:0/846776166 shutdown_connections 2022-07-20T02:25:36.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 --2- 172.21.15.26:0/846776166 >> v2:172.21.15.120:3300/0 conn(0x7fb6981449d0 0x7fb698144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 --2- 172.21.15.26:0/846776166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb698065980 0x7fb698065d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- 172.21.15.26:0/846776166 >> 172.21.15.26:0/846776166 conn(0x7fb69805c580 msgr2=0x7fb69805c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- 172.21.15.26:0/846776166 shutdown_connections 2022-07-20T02:25:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.679+0000 7fb69d34d700 1 -- 172.21.15.26:0/846776166 wait complete. 2022-07-20T02:25:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 Processor -- start 2022-07-20T02:25:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 -- start start 2022-07-20T02:25:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb698065980 0x7fb69813aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6981449d0 0x7fb698143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6980b5480 con 0x7fb69813dd40 2022-07-20T02:25:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6981388e0 con 0x7fb698065980 2022-07-20T02:25:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb69d34d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb698138b50 con 0x7fb6981449d0 2022-07-20T02:25:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb6967fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.683+0000 7fb6967fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34918/0 (socket says 172.21.15.26:34918) 2022-07-20T02:25:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 -- 172.21.15.26:0/341515253 learned_addr learned my addr 172.21.15.26:0/341515253 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb696ffd700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb698065980 0x7fb69813aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6977fe700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6981449d0 0x7fb698143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 -- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6981449d0 msgr2=0x7fb698143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6981449d0 0x7fb698143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 -- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb698065980 msgr2=0x7fb69813aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:36.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb698065980 0x7fb69813aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:36.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 -- 172.21.15.26:0/341515253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68800b180 con 0x7fb69813dd40 2022-07-20T02:25:36.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb6967fc700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb68800c000 tx=0x7fb688006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb688010780 con 0x7fb69813dd40 2022-07-20T02:25:36.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6880108e0 con 0x7fb69813dd40 2022-07-20T02:25:36.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68000b180 con 0x7fb69813dd40 2022-07-20T02:25:36.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb688010c40 con 0x7fb69813dd40 2022-07-20T02:25:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb698138640 con 0x7fb69813dd40 2022-07-20T02:25:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb68801c460 con 0x7fb69813dd40 2022-07-20T02:25:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 0x7fb67c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb68806c5a0 con 0x7fb69813dd40 2022-07-20T02:25:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb698145370 con 0x7fb69813dd40 2022-07-20T02:25:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.687+0000 7fb696ffd700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 0x7fb67c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.691+0000 7fb696ffd700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 0x7fb67c055ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb68001a000 tx=0x7fb680014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.695+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb688043300 con 0x7fb69813dd40 2022-07-20T02:25:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:36.855+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb6980574e0 con 0x7fb69813dd40 2022-07-20T02:25:37.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.679+0000 7fb68ffff700 1 -- 172.21.15.26:0/341515253 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb688046920 con 0x7fb69813dd40 2022-07-20T02:25:37.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 msgr2=0x7fb67c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 0x7fb67c055ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb68001a000 tx=0x7fb680014000).stop 2022-07-20T02:25:37.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 msgr2=0x7fb6981439f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb68800c000 tx=0x7fb688006df0).stop 2022-07-20T02:25:37.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 shutdown_connections 2022-07-20T02:25:37.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb69813dd40 0x7fb6981439f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb67c0539e0 0x7fb67c055ea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6981449d0 0x7fb698143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 --2- 172.21.15.26:0/341515253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb698065980 0x7fb69813aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 >> 172.21.15.26:0/341515253 conn(0x7fb69805c580 msgr2=0x7fb698065d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:37.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 shutdown_connections 2022-07-20T02:25:37.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.683+0000 7fb69d34d700 1 -- 172.21.15.26:0/341515253 wait complete. 2022-07-20T02:25:37.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:25:37.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 Processor -- start 2022-07-20T02:25:37.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- start start 2022-07-20T02:25:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2cc147be0 0x7fc2cc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0b62d0 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc14aac0 con 0x7fc2cc147be0 2022-07-20T02:25:37.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc14ad60 con 0x7fc2cc0a7fc0 2022-07-20T02:25:37.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2cc147be0 0x7fc2cc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2cc147be0 0x7fc2cc0aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34924/0 (socket says 172.21.15.26:34924) 2022-07-20T02:25:37.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 -- 172.21.15.26:0/2401244162 learned_addr learned my addr 172.21.15.26:0/2401244162 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:37.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d1625700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2cc0b5ef0 0x7fc2cc1474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34204/0 (socket says 172.21.15.26:34204) 2022-07-20T02:25:37.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d1e26700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 -- 172.21.15.26:0/2401244162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 msgr2=0x7fc2cc0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:37.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 -- 172.21.15.26:0/2401244162 >> v1:172.21.15.26:6789/0 conn(0x7fc2cc0b5ef0 legacy=0x7fc2cc1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 -- 172.21.15.26:0/2401244162 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc14b000 con 0x7fc2cc147be0 2022-07-20T02:25:37.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d0e24700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aee40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc2bc002f70 tx=0x7fc2bc006eb0).ready entity=mon.0 client_cookie=52ecdccaad74cdb1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:37.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2c3fff700 1 -- 172.21.15.26:0/2401244162 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2bc00d5c0 con 0x7fc2cc147be0 2022-07-20T02:25:37.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2c3fff700 1 -- 172.21.15.26:0/2401244162 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc000e90 con 0x7fc2cc147be0 2022-07-20T02:25:37.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2c3fff700 1 -- 172.21.15.26:0/2401244162 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2bc00d950 con 0x7fc2cc147be0 2022-07-20T02:25:37.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- 172.21.15.26:0/2401244162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 msgr2=0x7fc2cc0aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aee40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc2bc002f70 tx=0x7fc2bc006eb0).stop 2022-07-20T02:25:37.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- 172.21.15.26:0/2401244162 shutdown_connections 2022-07-20T02:25:37.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 --2- 172.21.15.26:0/2401244162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aee40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- 172.21.15.26:0/2401244162 >> 172.21.15.26:0/2401244162 conn(0x7fc2cc060310 msgr2=0x7fc2cc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:37.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- 172.21.15.26:0/2401244162 shutdown_connections 2022-07-20T02:25:37.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.811+0000 7fc2d2627700 1 -- 172.21.15.26:0/2401244162 wait complete. 2022-07-20T02:25:37.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.815+0000 7fc2d2627700 1 Processor -- start 2022-07-20T02:25:37.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.815+0000 7fc2d2627700 1 -- start start 2022-07-20T02:25:37.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 0x7fc2cc0b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0631e0 con 0x7fc2cc147be0 2022-07-20T02:25:37.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0aaa70 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0ac220 con 0x7fc2cc0a7fc0 2022-07-20T02:25:37.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1e26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 0x7fc2cc0b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1625700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1e26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34934/0 (socket says 172.21.15.26:34934) 2022-07-20T02:25:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1e26700 1 -- 172.21.15.26:0/3071194963 learned_addr learned my addr 172.21.15.26:0/3071194963 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 -- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 msgr2=0x7fc2cc0ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 -- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 msgr2=0x7fc2cc0aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:37.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:37.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 -- 172.21.15.26:0/3071194963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc00b1e0 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1e26700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d1625700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d0e24700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 0x7fc2cc0b4370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc2bc002f70 tx=0x7fc2bc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2bc006b40 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00d940 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2bc011d30 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc0ab840 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.819+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc0abd80 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc14b2a0 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2bc01a040 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2c27fc700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 0x7fc2b4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:37.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc2bc06cf30 con 0x7fc2cc0b5ef0 2022-07-20T02:25:37.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2d1625700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 0x7fc2b4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:37.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.823+0000 7fc2d1625700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 0x7fc2b4055f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc2c800d000 tx=0x7fc2c8007b40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:37.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.831+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2bc043be0 con 0x7fc2cc0b5ef0 2022-07-20T02:25:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.995+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2cc0574e0 con 0x7fc2cc0b5ef0 2022-07-20T02:25:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2c27fc700 1 -- 172.21.15.26:0/3071194963 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2489 (secure 0 0 0) 0x7fc2bc047200 con 0x7fc2cc0b5ef0 2022-07-20T02:25:38.002 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:38.002 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:25:37.669039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/4183261198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4183261198},{"type":"v1","addr":"172.21.15.26:6837","nonce":4183261198}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:25:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 msgr2=0x7fc2b4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 0x7fc2b4055f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc2c800d000 tx=0x7fc2c8007b40).stop 2022-07-20T02:25:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 msgr2=0x7fc2cc0b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 0x7fc2cc0b4370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc2bc002f70 tx=0x7fc2bc015000).stop 2022-07-20T02:25:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 shutdown_connections 2022-07-20T02:25:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2cc147be0 0x7fc2cc0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b4053a70 0x7fc2b4055f30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2cc0b5ef0 0x7fc2cc0b4370 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 --2- 172.21.15.26:0/3071194963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2cc0a7fc0 0x7fc2cc0ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 >> 172.21.15.26:0/3071194963 conn(0x7fc2cc060310 msgr2=0x7fc2cc0aef10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 shutdown_connections 2022-07-20T02:25:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:37.999+0000 7fc2d2627700 1 -- 172.21.15.26:0/3071194963 wait complete. 2022-07-20T02:25:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2022-07-20T02:25:38.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 Processor -- start 2022-07-20T02:25:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 -- start start 2022-07-20T02:25:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 0x7f43181474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43180a7fc0 0x7f43181479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180b62d0 con 0x7f43180a8c40 2022-07-20T02:25:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4318147eb0 con 0x7f43180a7fc0 2022-07-20T02:25:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431e6ff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431814aac0 con 0x7f43180b5ef0 2022-07-20T02:25:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431cefc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43180a7fc0 0x7f43181479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431cefc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43180a7fc0 0x7f43181479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34938/0 (socket says 172.21.15.26:34938) 2022-07-20T02:25:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431cefc700 1 -- 172.21.15.26:0/2757569878 learned_addr learned my addr 172.21.15.26:0/2757569878 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.123+0000 7f431d6fd700 1 --2- 172.21.15.26:0/2757569878 >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 0x7f43181474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431cefc700 1 -- 172.21.15.26:0/2757569878 >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 msgr2=0x7f43181474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:38.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431cefc700 1 --2- 172.21.15.26:0/2757569878 >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 0x7f43181474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431cefc700 1 -- 172.21.15.26:0/2757569878 >> v1:172.21.15.120:6789/0 conn(0x7f43180a8c40 legacy=0x7f43180aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431cefc700 1 -- 172.21.15.26:0/2757569878 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431814ad60 con 0x7f43180a7fc0 2022-07-20T02:25:38.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431d6fd700 1 --2- 172.21.15.26:0/2757569878 >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 0x7f43181474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:38.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431cefc700 1 --2- 172.21.15.26:0/2757569878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181479c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f43140032f0 tx=0x7f4314009290).ready entity=mon.0 client_cookie=8daee1da72227589 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f430ffff700 1 -- 172.21.15.26:0/2757569878 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f431400b870 con 0x7f43180a7fc0 2022-07-20T02:25:38.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f430ffff700 1 -- 172.21.15.26:0/2757569878 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431400b9d0 con 0x7f43180a7fc0 2022-07-20T02:25:38.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f430ffff700 1 -- 172.21.15.26:0/2757569878 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f431400bb40 con 0x7f43180a7fc0 2022-07-20T02:25:38.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 -- 172.21.15.26:0/2757569878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 msgr2=0x7f43181479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 --2- 172.21.15.26:0/2757569878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181479c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f43140032f0 tx=0x7f4314009290).stop 2022-07-20T02:25:38.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 -- 172.21.15.26:0/2757569878 shutdown_connections 2022-07-20T02:25:38.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 --2- 172.21.15.26:0/2757569878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181479c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 --2- 172.21.15.26:0/2757569878 >> v2:172.21.15.120:3300/0 conn(0x7f43180b5ef0 0x7f43181474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 -- 172.21.15.26:0/2757569878 >> 172.21.15.26:0/2757569878 conn(0x7f4318060310 msgr2=0x7f43180b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 -- 172.21.15.26:0/2757569878 shutdown_connections 2022-07-20T02:25:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.127+0000 7f431e6ff700 1 -- 172.21.15.26:0/2757569878 wait complete. 2022-07-20T02:25:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 Processor -- start 2022-07-20T02:25:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- start start 2022-07-20T02:25:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43180a8c40 0x7f4318156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 0x7f4318156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318066750 con 0x7f43180a7fc0 2022-07-20T02:25:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180662f0 con 0x7f43180b5ef0 2022-07-20T02:25:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318064dc0 con 0x7f43180a8c40 2022-07-20T02:25:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34944/0 (socket says 172.21.15.26:34944) 2022-07-20T02:25:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 -- 172.21.15.26:0/1171237890 learned_addr learned my addr 172.21.15.26:0/1171237890 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431cefc700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43180a8c40 0x7f4318156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431defe700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 0x7f4318156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 -- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43180a8c40 msgr2=0x7f4318156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43180a8c40 0x7f4318156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 -- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 msgr2=0x7f4318156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 0x7f4318156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 -- 172.21.15.26:0/1171237890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4314009530 con 0x7f43180a7fc0 2022-07-20T02:25:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431defe700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 0x7f4318156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:25:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431d6fd700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4308006940 tx=0x7f4308008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308011780 con 0x7f43180a7fc0 2022-07-20T02:25:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43080118e0 con 0x7f43180a7fc0 2022-07-20T02:25:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318157050 con 0x7f43180a7fc0 2022-07-20T02:25:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308011c40 con 0x7f43180a7fc0 2022-07-20T02:25:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4318066980 con 0x7f43180a7fc0 2022-07-20T02:25:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.131+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318064310 con 0x7f43180a7fc0 2022-07-20T02:25:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.135+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4308012420 con 0x7f43180a7fc0 2022-07-20T02:25:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.135+0000 7f430e7fc700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.135+0000 7f431cefc700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.135+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f430806bfc0 con 0x7f43180a7fc0 2022-07-20T02:25:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.135+0000 7f431cefc700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 0x7f4300055f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4314018000 tx=0x7f4314016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.139+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4308042d20 con 0x7f43180a7fc0 2022-07-20T02:25:38.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.263+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4318064f80 con 0x7f43180a7fc0 2022-07-20T02:25:38.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.267+0000 7f430e7fc700 1 -- 172.21.15.26:0/1171237890 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f4308046340 con 0x7f43180a7fc0 2022-07-20T02:25:38.269 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:38.270 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:38.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 msgr2=0x7f4300055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 0x7f4300055f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4314018000 tx=0x7f4314016000).stop 2022-07-20T02:25:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 msgr2=0x7f43181493a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4308006940 tx=0x7f4308008040).stop 2022-07-20T02:25:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 shutdown_connections 2022-07-20T02:25:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43180b5ef0 0x7f4318156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43180a8c40 0x7f4318156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 --2- 172.21.15.26:0/1171237890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43180a7fc0 0x7f43181493a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 >> 172.21.15.26:0/1171237890 conn(0x7f4318060310 msgr2=0x7f43180b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 shutdown_connections 2022-07-20T02:25:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.271+0000 7f431e6ff700 1 -- 172.21.15.26:0/1171237890 wait complete. 2022-07-20T02:25:38.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:38.291 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-20T02:25:38.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.391+0000 7f1166d78700 1 Processor -- start 2022-07-20T02:25:38.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.391+0000 7f1166d78700 1 -- start start 2022-07-20T02:25:38.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803d930 0x7f1158042450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1158041400 0x7f11580d8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1158042940 con 0x7f115803c650 2022-07-20T02:25:38.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1158040e70 con 0x7f1158041400 2022-07-20T02:25:38.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158041d10 con 0x7f115803d930 2022-07-20T02:25:38.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1158041400 0x7f11580d8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1158041400 0x7f11580d8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34952/0 (socket says 172.21.15.26:34952) 2022-07-20T02:25:38.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 -- 172.21.15.26:0/2984331755 learned_addr learned my addr 172.21.15.26:0/2984331755 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165d76700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803d930 0x7f1158042450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 -- 172.21.15.26:0/2984331755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803d930 msgr2=0x7f1158042450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:38.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803d930 0x7f1158042450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 -- 172.21.15.26:0/2984331755 >> v1:172.21.15.120:6789/0 conn(0x7f115803c650 legacy=0x7f115803cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 -- 172.21.15.26:0/2984331755 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158041fb0 con 0x7f1158041400 2022-07-20T02:25:38.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1165575700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580d8420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1150002f70 tx=0x7f1150006eb0).ready entity=mon.0 client_cookie=29786edb1d8aca24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1164d74700 1 -- 172.21.15.26:0/2984331755 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f115000d5c0 con 0x7f1158041400 2022-07-20T02:25:38.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1164d74700 1 -- 172.21.15.26:0/2984331755 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150000e90 con 0x7f1158041400 2022-07-20T02:25:38.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1164d74700 1 -- 172.21.15.26:0/2984331755 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f115000d950 con 0x7f1158041400 2022-07-20T02:25:38.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- 172.21.15.26:0/2984331755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 msgr2=0x7f11580d8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580d8420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1150002f70 tx=0x7f1150006eb0).stop 2022-07-20T02:25:38.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- 172.21.15.26:0/2984331755 shutdown_connections 2022-07-20T02:25:38.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580d8420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 --2- 172.21.15.26:0/2984331755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803d930 0x7f1158042450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- 172.21.15.26:0/2984331755 >> 172.21.15.26:0/2984331755 conn(0x7f1158046350 msgr2=0x7f1158046760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- 172.21.15.26:0/2984331755 shutdown_connections 2022-07-20T02:25:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.395+0000 7f1166d78700 1 -- 172.21.15.26:0/2984331755 wait complete. 2022-07-20T02:25:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 Processor -- start 2022-07-20T02:25:38.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 -- start start 2022-07-20T02:25:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803c650 0x7f115803a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f115803d930 0x7f11580a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11580e83e0 con 0x7f1158041400 2022-07-20T02:25:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158039d20 con 0x7f115803d930 2022-07-20T02:25:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166d78700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158039fc0 con 0x7f115803c650 2022-07-20T02:25:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1165d76700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803c650 0x7f115803a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34962/0 (socket says 172.21.15.26:34962) 2022-07-20T02:25:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 -- 172.21.15.26:0/2686980243 learned_addr learned my addr 172.21.15.26:0/2686980243 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1165575700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f115803d930 0x7f11580a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 -- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803c650 msgr2=0x7f115803a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803c650 0x7f115803a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 -- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f115803d930 msgr2=0x7f11580a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f115803d930 0x7f11580a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 -- 172.21.15.26:0/2686980243 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115000b1e0 con 0x7f1158041400 2022-07-20T02:25:38.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.399+0000 7f1166577700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1160044f70 tx=0x7f116005f750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116005d1e0 con 0x7f1158041400 2022-07-20T02:25:38.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1160085070 con 0x7f1158041400 2022-07-20T02:25:38.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116005d570 con 0x7f1158041400 2022-07-20T02:25:38.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11580459b0 con 0x7f1158041400 2022-07-20T02:25:38.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158039860 con 0x7f1158041400 2022-07-20T02:25:38.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11580a1070 con 0x7f1158041400 2022-07-20T02:25:38.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11600683d0 con 0x7f1158041400 2022-07-20T02:25:38.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 0x7f1140055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11600da940 con 0x7f1158041400 2022-07-20T02:25:38.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f1165d76700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 0x7f1140055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.403+0000 7f1165d76700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 0x7f1140055ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f115c00c000 tx=0x7f115c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.407+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11600b16a0 con 0x7f1158041400 2022-07-20T02:25:38.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.535+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f115803a180 con 0x7f1158041400 2022-07-20T02:25:38.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f114effd700 1 -- 172.21.15.26:0/2686980243 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f11600b4cc0 con 0x7f1158041400 2022-07-20T02:25:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:38.543 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:38.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 msgr2=0x7f1140055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 0x7f1140055ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f115c00c000 tx=0x7f115c008040).stop 2022-07-20T02:25:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 msgr2=0x7f11580eac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1160044f70 tx=0x7f116005f750).stop 2022-07-20T02:25:38.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 shutdown_connections 2022-07-20T02:25:38.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158041400 0x7f11580eac60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1140053a10 0x7f1140055ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f115803d930 0x7f11580a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 --2- 172.21.15.26:0/2686980243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f115803c650 0x7f115803a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 >> 172.21.15.26:0/2686980243 conn(0x7f1158046350 msgr2=0x7f11580d3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 shutdown_connections 2022-07-20T02:25:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.539+0000 7f1166d78700 1 -- 172.21.15.26:0/2686980243 wait complete. 2022-07-20T02:25:38.568 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:38.569 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 Processor -- start 2022-07-20T02:25:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 -- start start 2022-07-20T02:25:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff08c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c1474e0 con 0x7ff08c05cfd0 2022-07-20T02:25:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0b4e40 con 0x7ff08c060f00 2022-07-20T02:25:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c05d3b0 con 0x7ff08c147000 2022-07-20T02:25:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08affd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff08c05cfd0 0x7ff08c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34246/0 (socket says 172.21.15.26:34246) 2022-07-20T02:25:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08affd700 1 -- 172.21.15.26:0/2636184363 learned_addr learned my addr 172.21.15.26:0/2636184363 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff083fff700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff08c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469004449 0 0) 0x7ff08c1474e0 con 0x7ff08c05cfd0 2022-07-20T02:25:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06c003260 con 0x7ff08c05cfd0 2022-07-20T02:25:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778644124 0 0) 0x7ff08c0b4e40 con 0x7ff08c060f00 2022-07-20T02:25:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08c1474e0 con 0x7ff08c060f00 2022-07-20T02:25:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859296625 0 0) 0x7ff06c003260 con 0x7ff08c05cfd0 2022-07-20T02:25:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 msgr2=0x7ff08c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff08c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 >> v1:172.21.15.120:6789/0 conn(0x7ff08c060f00 legacy=0x7ff08c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c1431e0 con 0x7ff08c05cfd0 2022-07-20T02:25:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7ff07c001660 con 0x7ff08c05cfd0 2022-07-20T02:25:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 >> v1:172.21.15.26:6789/0 conn(0x7ff08c05cfd0 legacy=0x7ff08c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.683+0000 7ff08a7fc700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff06c003910 0x7ff06c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff06c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c0049f0 0x7ff06c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c05d3b0 con 0x7ff06c003910 2022-07-20T02:25:38.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c00b800 con 0x7ff06c0049f0 2022-07-20T02:25:38.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff06c003910 0x7ff06c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c00b970 con 0x7ff08c147000 2022-07-20T02:25:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff083fff700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff06c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08b7fe700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c0049f0 0x7ff06c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 -- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 msgr2=0x7ff06c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff06c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 -- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c0049f0 msgr2=0x7ff06c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c0049f0 0x7ff06c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 -- 172.21.15.26:0/2636184363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c003260 con 0x7ff06c003910 2022-07-20T02:25:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff083fff700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c147000 0x7ff06c004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:38.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08b7fe700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff06c0049f0 0x7ff06c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:38.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08affd700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff06c003910 0x7ff06c003d10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff07c001430 tx=0x7ff08c0b1900).ready entity=mon.0 client_cookie=5f9cbda9d85e39d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c007710 con 0x7ff06c003910 2022-07-20T02:25:38.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c0070d0 con 0x7ff06c003910 2022-07-20T02:25:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08a7fc700 1 -- 172.21.15.26:0/2636184363 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c007430 con 0x7ff06c003910 2022-07-20T02:25:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- 172.21.15.26:0/2636184363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff06c003910 msgr2=0x7ff06c003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 --2- 172.21.15.26:0/2636184363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff06c003910 0x7ff06c003d10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff07c001430 tx=0x7ff08c0b1900).stop 2022-07-20T02:25:38.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08b7fe700 1 -- 172.21.15.26:0/2636184363 reap_dead start 2022-07-20T02:25:38.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- 172.21.15.26:0/2636184363 shutdown_connections 2022-07-20T02:25:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- 172.21.15.26:0/2636184363 >> 172.21.15.26:0/2636184363 conn(0x7ff08c0620c0 msgr2=0x7ff08c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- 172.21.15.26:0/2636184363 shutdown_connections 2022-07-20T02:25:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- 172.21.15.26:0/2636184363 wait complete. 2022-07-20T02:25:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 Processor -- start 2022-07-20T02:25:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.687+0000 7ff08bfff700 1 -- start start 2022-07-20T02:25:38.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c140650 0x7ff08c152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c152c00 0x7ff08c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c145d00 con 0x7ff08c145920 2022-07-20T02:25:38.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c1522b0 con 0x7ff08c152c00 2022-07-20T02:25:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0b9230 con 0x7ff08c140650 2022-07-20T02:25:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34978/0 (socket says 172.21.15.26:34978) 2022-07-20T02:25:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 -- 172.21.15.26:0/3273257876 learned_addr learned my addr 172.21.15.26:0/3273257876 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff083fff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c140650 0x7ff08c152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08b7fe700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c152c00 0x7ff08c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 -- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c140650 msgr2=0x7ff08c152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c140650 0x7ff08c152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 -- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c152c00 msgr2=0x7ff08c155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c152c00 0x7ff08c155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 -- 172.21.15.26:0/3273257876 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0b8cf0 con 0x7ff08c145920 2022-07-20T02:25:38.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08affd700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff08c0b93c0 tx=0x7ff07c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c007710 con 0x7ff08c145920 2022-07-20T02:25:38.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c000e90 con 0x7ff08c145920 2022-07-20T02:25:38.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c145d00 con 0x7ff08c145920 2022-07-20T02:25:38.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c004b10 con 0x7ff08c145920 2022-07-20T02:25:38.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c153080 con 0x7ff08c145920 2022-07-20T02:25:38.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c143480 con 0x7ff08c145920 2022-07-20T02:25:38.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.691+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff07c027070 con 0x7ff08c145920 2022-07-20T02:25:38.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.695+0000 7ff088ff9700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 0x7ff070055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.695+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff07c06cac0 con 0x7ff08c145920 2022-07-20T02:25:38.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.695+0000 7ff083fff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 0x7ff070055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.695+0000 7ff083fff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 0x7ff070055fc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff07400b000 tx=0x7ff074008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.699+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff07c043880 con 0x7ff08c145920 2022-07-20T02:25:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.819+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff08c0574e0 con 0x7ff08c145920 2022-07-20T02:25:38.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.819+0000 7ff088ff9700 1 -- 172.21.15.26:0/3273257876 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff07c046ea0 con 0x7ff08c145920 2022-07-20T02:25:38.832 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 msgr2=0x7ff070055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 0x7ff070055fc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff07400b000 tx=0x7ff074008040).stop 2022-07-20T02:25:38.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 msgr2=0x7ff08c0ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff08c0b93c0 tx=0x7ff07c015000).stop 2022-07-20T02:25:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 shutdown_connections 2022-07-20T02:25:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff070053b00 0x7ff070055fc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c152c00 0x7ff08c155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff08c140650 0x7ff08c152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 --2- 172.21.15.26:0/3273257876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff08c145920 0x7ff08c0ad670 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 >> 172.21.15.26:0/3273257876 conn(0x7ff06c003910 msgr2=0x7ff08c13a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 shutdown_connections 2022-07-20T02:25:38.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.823+0000 7ff08bfff700 1 -- 172.21.15.26:0/3273257876 wait complete. 2022-07-20T02:25:38.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:38.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.951+0000 7f276994b700 1 Processor -- start 2022-07-20T02:25:38.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.951+0000 7f276994b700 1 -- start start 2022-07-20T02:25:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 0x7f2764140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27640ab530 con 0x7f276405cf80 2022-07-20T02:25:38.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276405cc90 con 0x7f2764147020 2022-07-20T02:25:38.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27640b1ce0 con 0x7f2764140320 2022-07-20T02:25:38.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276914a700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f276405cf80 0x7f2764143040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34270/0 (socket says 172.21.15.26:34270) 2022-07-20T02:25:38.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276914a700 1 -- 172.21.15.26:0/3766377511 learned_addr learned my addr 172.21.15.26:0/3766377511 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f2768949700 1 --2- 172.21.15.26:0/3766377511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 0x7f2764140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458051717 0 0) 0x7f2754001110 con 0x7f276405cf80 2022-07-20T02:25:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2744003260 con 0x7f276405cf80 2022-07-20T02:25:38.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808735171 0 0) 0x7f276405cc90 con 0x7f2764147020 2022-07-20T02:25:38.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754001110 con 0x7f2764147020 2022-07-20T02:25:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f2768949700 1 -- 172.21.15.26:0/3766377511 >> v1:172.21.15.120:6789/0 conn(0x7f2764147020 legacy=0x7f276405c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f2768949700 1 -- 172.21.15.26:0/3766377511 >> v1:172.21.15.26:6789/0 conn(0x7f276405cf80 legacy=0x7f2764143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f2768949700 1 -- 172.21.15.26:0/3766377511 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27640aab50 con 0x7f2764140320 2022-07-20T02:25:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f2768949700 1 --2- 172.21.15.26:0/3766377511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 0x7f2764140700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f275c0032f0 tx=0x7f275c0091d0).ready entity=mon.2 client_cookie=3aca1c3b6adf5b4d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f275c00b640 con 0x7f2764140320 2022-07-20T02:25:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275c00b7a0 con 0x7f2764140320 2022-07-20T02:25:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f27637fe700 1 -- 172.21.15.26:0/3766377511 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f275c00b910 con 0x7f2764140320 2022-07-20T02:25:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- 172.21.15.26:0/3766377511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 msgr2=0x7f2764140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 --2- 172.21.15.26:0/3766377511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 0x7f2764140700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f275c0032f0 tx=0x7f275c0091d0).stop 2022-07-20T02:25:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- 172.21.15.26:0/3766377511 shutdown_connections 2022-07-20T02:25:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 --2- 172.21.15.26:0/3766377511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764140320 0x7f2764140700 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- 172.21.15.26:0/3766377511 >> 172.21.15.26:0/3766377511 conn(0x7f2764060310 msgr2=0x7f27640b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- 172.21.15.26:0/3766377511 shutdown_connections 2022-07-20T02:25:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.955+0000 7f276994b700 1 -- 172.21.15.26:0/3766377511 wait complete. 2022-07-20T02:25:38.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.959+0000 7f276994b700 1 Processor -- start 2022-07-20T02:25:38.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.959+0000 7f276994b700 1 -- start start 2022-07-20T02:25:38.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276405cf80 0x7f27640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764147020 0x7f2764140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27640ae730 con 0x7f2764140320 2022-07-20T02:25:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27640af470 con 0x7f276405cf80 2022-07-20T02:25:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27640af710 con 0x7f2764147020 2022-07-20T02:25:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34994/0 (socket says 172.21.15.26:34994) 2022-07-20T02:25:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 -- 172.21.15.26:0/2997079205 learned_addr learned my addr 172.21.15.26:0/2997079205 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2768949700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276405cf80 0x7f27640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276914a700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764147020 0x7f2764140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 -- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764147020 msgr2=0x7f2764140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764147020 0x7f2764140ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 -- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276405cf80 msgr2=0x7f27640b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:38.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276405cf80 0x7f27640b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:38.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 -- 172.21.15.26:0/2997079205 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c009470 con 0x7f2764140320 2022-07-20T02:25:38.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2763fff700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f274c001110 tx=0x7f274c007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f274c012400 con 0x7f2764140320 2022-07-20T02:25:38.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c012560 con 0x7f2764140320 2022-07-20T02:25:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27640af9b0 con 0x7f2764140320 2022-07-20T02:25:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f274c0128c0 con 0x7f2764140320 2022-07-20T02:25:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2764143570 con 0x7f2764140320 2022-07-20T02:25:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27640aadf0 con 0x7f2764140320 2022-07-20T02:25:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.963+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f274c019030 con 0x7f2764140320 2022-07-20T02:25:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.967+0000 7f2761ffb700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 0x7f2748055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.967+0000 7f2768949700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 0x7f2748055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.967+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f274c06b470 con 0x7f2764140320 2022-07-20T02:25:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.967+0000 7f2768949700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 0x7f2748055fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f275c019000 tx=0x7f275c016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:38.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:38.975+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f274c042120 con 0x7f2764140320 2022-07-20T02:25:39.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.099+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f27640afe50 con 0x7f2764140320 2022-07-20T02:25:39.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.099+0000 7f2761ffb700 1 -- 172.21.15.26:0/2997079205 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f274c045740 con 0x7f2764140320 2022-07-20T02:25:39.108 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:39.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 msgr2=0x7f2748055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 0x7f2748055fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f275c019000 tx=0x7f275c016000).stop 2022-07-20T02:25:39.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 msgr2=0x7f27640b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f274c001110 tx=0x7f274c007af0).stop 2022-07-20T02:25:39.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 shutdown_connections 2022-07-20T02:25:39.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2764140320 0x7f27640b0170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2748053b00 0x7f2748055fc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2764147020 0x7f2764140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 --2- 172.21.15.26:0/2997079205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276405cf80 0x7f27640b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 >> 172.21.15.26:0/2997079205 conn(0x7f2764060310 msgr2=0x7f276413b580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 shutdown_connections 2022-07-20T02:25:39.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.103+0000 7f276994b700 1 -- 172.21.15.26:0/2997079205 wait complete. 2022-07-20T02:25:39.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:39.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 Processor -- start 2022-07-20T02:25:39.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 -- start start 2022-07-20T02:25:39.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240b5ef0 0x7ff3241474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 0x7ff3241479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3240a8c40 0x7ff3240aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff324147eb0 con 0x7ff3240a8c40 2022-07-20T02:25:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240b62d0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff3288ed700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32414aac0 con 0x7ff3240b5ef0 2022-07-20T02:25:39.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff32359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3240a8c40 0x7ff3240aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff32359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3240a8c40 0x7ff3240aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35002/0 (socket says 172.21.15.26:35002) 2022-07-20T02:25:39.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff32359e700 1 -- 172.21.15.26:0/2774797667 learned_addr learned my addr 172.21.15.26:0/2774797667 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:39.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.247+0000 7ff32259c700 1 --2- 172.21.15.26:0/2774797667 >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 0x7ff3241479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff322d9d700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240b5ef0 0x7ff3241474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 -- 172.21.15.26:0/2774797667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240b5ef0 msgr2=0x7ff3241474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:39.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240b5ef0 0x7ff3241474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 -- 172.21.15.26:0/2774797667 >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 msgr2=0x7ff3241479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:39.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 --2- 172.21.15.26:0/2774797667 >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 0x7ff3241479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 -- 172.21.15.26:0/2774797667 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32414ad60 con 0x7ff3240a8c40 2022-07-20T02:25:39.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32259c700 1 --2- 172.21.15.26:0/2774797667 >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 0x7ff3241479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:39.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff32359e700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240a8c40 0x7ff3240aee40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff31c0032f0 tx=0x7ff31c009290).ready entity=mon.0 client_cookie=bd9e7b6468f6e83e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff321d9b700 1 -- 172.21.15.26:0/2774797667 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c00b870 con 0x7ff3240a8c40 2022-07-20T02:25:39.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff321d9b700 1 -- 172.21.15.26:0/2774797667 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c00b9d0 con 0x7ff3240a8c40 2022-07-20T02:25:39.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff321d9b700 1 -- 172.21.15.26:0/2774797667 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c00bb40 con 0x7ff3240a8c40 2022-07-20T02:25:39.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- 172.21.15.26:0/2774797667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240a8c40 msgr2=0x7ff3240aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240a8c40 0x7ff3240aee40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff31c0032f0 tx=0x7ff31c009290).stop 2022-07-20T02:25:39.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- 172.21.15.26:0/2774797667 shutdown_connections 2022-07-20T02:25:39.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240a8c40 0x7ff3240aee40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2774797667 >> v2:172.21.15.120:3300/0 conn(0x7ff3240a7fc0 0x7ff3241479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2774797667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240b5ef0 0x7ff3241474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- 172.21.15.26:0/2774797667 >> 172.21.15.26:0/2774797667 conn(0x7ff324060310 msgr2=0x7ff3240b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- 172.21.15.26:0/2774797667 shutdown_connections 2022-07-20T02:25:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- 172.21.15.26:0/2774797667 wait complete. 2022-07-20T02:25:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 Processor -- start 2022-07-20T02:25:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.251+0000 7ff3288ed700 1 -- start start 2022-07-20T02:25:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3240a8c40 0x7ff324156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240b5ef0 0x7ff324157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241567d0 con 0x7ff3240b5ef0 2022-07-20T02:25:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240aad20 con 0x7ff3240a8c40 2022-07-20T02:25:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff324062c00 con 0x7ff3240a7fc0 2022-07-20T02:25:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff32359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240b5ef0 0x7ff324157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:56522/0 (socket says 172.21.15.26:56522) 2022-07-20T02:25:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 -- 172.21.15.26:0/2423839053 learned_addr learned my addr 172.21.15.26:0/2423839053 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff32259c700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3240a8c40 0x7ff324156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 -- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3240a8c40 msgr2=0x7ff324156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3240a8c40 0x7ff324156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 -- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240b5ef0 msgr2=0x7ff324157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240b5ef0 0x7ff324157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 -- 172.21.15.26:0/2423839053 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c009530 con 0x7ff3240a7fc0 2022-07-20T02:25:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff322d9d700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff3180030b0 tx=0x7ff318008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff318011780 con 0x7ff3240a7fc0 2022-07-20T02:25:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3180118e0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff318011c40 con 0x7ff3240a7fc0 2022-07-20T02:25:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff324062ea0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3240a92b0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff324156bc0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.255+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff31801c460 con 0x7ff3240a7fc0 2022-07-20T02:25:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.259+0000 7ff313fff700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 0x7ff300055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.259+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff31806c3a0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.259+0000 7ff32259c700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 0x7ff300055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.259+0000 7ff32259c700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 0x7ff300055e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff31400c000 tx=0x7ff314008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.267+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff324156bc0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff324156bc0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff313fff700 1 -- 172.21.15.26:0/2423839053 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff324156bc0 con 0x7ff3240a7fc0 2022-07-20T02:25:39.400 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:39.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 msgr2=0x7ff300055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 0x7ff300055e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff31400c000 tx=0x7ff314008040).stop 2022-07-20T02:25:39.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 msgr2=0x7ff32414e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.395+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff3180030b0 tx=0x7ff318008040).stop 2022-07-20T02:25:39.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 shutdown_connections 2022-07-20T02:25:39.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3240b5ef0 0x7ff324157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff300053950 0x7ff300055e10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3240a8c40 0x7ff324156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 --2- 172.21.15.26:0/2423839053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3240a7fc0 0x7ff32414e1c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 >> 172.21.15.26:0/2423839053 conn(0x7ff324060310 msgr2=0x7ff3240b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 shutdown_connections 2022-07-20T02:25:39.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.399+0000 7ff3288ed700 1 -- 172.21.15.26:0/2423839053 wait complete. 2022-07-20T02:25:39.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:39.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 Processor -- start 2022-07-20T02:25:39.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 -- start start 2022-07-20T02:25:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 0x7ff1540aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1540b1cb0 0x7ff154142890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 0x7ff154143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15405d3b0 con 0x7ff15405cfd0 2022-07-20T02:25:39.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540b1a60 con 0x7ff1540b1cb0 2022-07-20T02:25:39.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff153fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540b2140 con 0x7ff154142d80 2022-07-20T02:25:39.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff152ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 0x7ff1540aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff152ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 0x7ff1540aaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35014/0 (socket says 172.21.15.26:35014) 2022-07-20T02:25:39.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff152ffd700 1 -- 172.21.15.26:0/2165452375 learned_addr learned my addr 172.21.15.26:0/2165452375 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:39.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 --2- 172.21.15.26:0/2165452375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 0x7ff154143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff14a7fc700 1 --2- 172.21.15.26:0/2165452375 >> v2:172.21.15.120:3300/0 conn(0x7ff1540b1cb0 0x7ff154142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 -- 172.21.15.26:0/2165452375 >> v2:172.21.15.120:3300/0 conn(0x7ff1540b1cb0 msgr2=0x7ff154142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:39.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 --2- 172.21.15.26:0/2165452375 >> v2:172.21.15.120:3300/0 conn(0x7ff1540b1cb0 0x7ff154142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 -- 172.21.15.26:0/2165452375 >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 msgr2=0x7ff1540aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:39.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 --2- 172.21.15.26:0/2165452375 >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 0x7ff1540aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.543+0000 7ff1537fe700 1 -- 172.21.15.26:0/2165452375 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540b1520 con 0x7ff154142d80 2022-07-20T02:25:39.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff1537fe700 1 --2- 172.21.15.26:0/2165452375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 0x7ff154143810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff14c0032f0 tx=0x7ff14c0091d0).ready entity=mon.2 client_cookie=1b3e98c1e843cbad server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff1527fc700 1 -- 172.21.15.26:0/2165452375 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c00b600 con 0x7ff154142d80 2022-07-20T02:25:39.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff1527fc700 1 -- 172.21.15.26:0/2165452375 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c00b760 con 0x7ff154142d80 2022-07-20T02:25:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff1527fc700 1 -- 172.21.15.26:0/2165452375 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c00b8d0 con 0x7ff154142d80 2022-07-20T02:25:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- 172.21.15.26:0/2165452375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 msgr2=0x7ff154143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 --2- 172.21.15.26:0/2165452375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 0x7ff154143810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff14c0032f0 tx=0x7ff14c0091d0).stop 2022-07-20T02:25:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- 172.21.15.26:0/2165452375 shutdown_connections 2022-07-20T02:25:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 --2- 172.21.15.26:0/2165452375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff154142d80 0x7ff154143810 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 --2- 172.21.15.26:0/2165452375 >> v2:172.21.15.120:3300/0 conn(0x7ff1540b1cb0 0x7ff154142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 --2- 172.21.15.26:0/2165452375 >> v2:172.21.15.26:3300/0 conn(0x7ff15405cfd0 0x7ff1540aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- 172.21.15.26:0/2165452375 >> 172.21.15.26:0/2165452375 conn(0x7ff1540b7090 msgr2=0x7ff15405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- 172.21.15.26:0/2165452375 shutdown_connections 2022-07-20T02:25:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- 172.21.15.26:0/2165452375 wait complete. 2022-07-20T02:25:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 Processor -- start 2022-07-20T02:25:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.547+0000 7ff153fff700 1 -- start start 2022-07-20T02:25:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff15405cfd0 0x7ff1540b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 0x7ff1540ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540adb00 con 0x7ff1540b1cb0 2022-07-20T02:25:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154064320 con 0x7ff154142d80 2022-07-20T02:25:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff154063800 con 0x7ff15405cfd0 2022-07-20T02:25:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff14a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 0x7ff1540ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:54714/0 (socket says 172.21.15.26:54714) 2022-07-20T02:25:39.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 -- 172.21.15.26:0/4292098845 learned_addr learned my addr 172.21.15.26:0/4292098845 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:39.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff152ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff15405cfd0 0x7ff1540b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff14a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 0x7ff1540ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35024/0 (socket says 172.21.15.26:35024) 2022-07-20T02:25:39.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 -- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff15405cfd0 msgr2=0x7ff1540b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff15405cfd0 0x7ff1540b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 -- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 msgr2=0x7ff1540ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 0x7ff1540ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 -- 172.21.15.26:0/4292098845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c009470 con 0x7ff154142d80 2022-07-20T02:25:39.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff1537fe700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff14c0032f0 tx=0x7ff14c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c00b600 con 0x7ff154142d80 2022-07-20T02:25:39.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c011ca0 con 0x7ff154142d80 2022-07-20T02:25:39.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540aedd0 con 0x7ff154142d80 2022-07-20T02:25:39.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c01f3f0 con 0x7ff154142d80 2022-07-20T02:25:39.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff154063a60 con 0x7ff154142d80 2022-07-20T02:25:39.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.551+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1540b17c0 con 0x7ff154142d80 2022-07-20T02:25:39.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.555+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff14c026030 con 0x7ff154142d80 2022-07-20T02:25:39.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.555+0000 7ff150ff9700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 0x7ff138055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.555+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff14c06caa0 con 0x7ff154142d80 2022-07-20T02:25:39.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.555+0000 7ff152ffd700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 0x7ff138055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.555+0000 7ff152ffd700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 0x7ff138055ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff144008000 tx=0x7ff144008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.563+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff14c0438e0 con 0x7ff154142d80 2022-07-20T02:25:39.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.687+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff1540574e0 con 0x7ff154142d80 2022-07-20T02:25:39.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff150ff9700 1 -- 172.21.15.26:0/4292098845 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff14c0150a0 con 0x7ff154142d80 2022-07-20T02:25:39.704 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:39.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 msgr2=0x7ff138055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 0x7ff138055ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff144008000 tx=0x7ff144008040).stop 2022-07-20T02:25:39.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 msgr2=0x7ff1540ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff14c0032f0 tx=0x7ff14c016000).stop 2022-07-20T02:25:39.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 shutdown_connections 2022-07-20T02:25:39.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1540b1cb0 0x7ff1540ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff138053a10 0x7ff138055ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff154142d80 0x7ff1540ae8e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 --2- 172.21.15.26:0/4292098845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff15405cfd0 0x7ff1540b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 >> 172.21.15.26:0/4292098845 conn(0x7ff1540b7090 msgr2=0x7ff15405c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 shutdown_connections 2022-07-20T02:25:39.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.691+0000 7ff153fff700 1 -- 172.21.15.26:0/4292098845 wait complete. 2022-07-20T02:25:39.739 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-07-20T02:25:39.740 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-07-20T02:25:39.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-07-20T02:25:39.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 Processor -- start 2022-07-20T02:25:39.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 -- start start 2022-07-20T02:25:39.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08ec0b06e0 0x7f08ec14a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0aa1e0 con 0x7f08ec0b2ab0 2022-07-20T02:25:39.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0ae110 con 0x7f08ec0add30 2022-07-20T02:25:39.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08eb59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0a9f40 con 0x7f08ec0b06e0 2022-07-20T02:25:39.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08ead9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08ec0b06e0 0x7f08ec14a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08e3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f08ec0b2ab0 0x7f08ec061060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34308/0 (socket says 172.21.15.26:34308) 2022-07-20T02:25:39.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08e3fff700 1 -- 172.21.15.26:0/1504162402 learned_addr learned my addr 172.21.15.26:0/1504162402 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:39.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08ead9d700 1 -- 172.21.15.26:0/1504162402 >> v1:172.21.15.120:6789/0 conn(0x7f08ec0add30 legacy=0x7f08ec060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08ead9d700 1 -- 172.21.15.26:0/1504162402 >> v1:172.21.15.26:6789/0 conn(0x7f08ec0b2ab0 legacy=0x7f08ec061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08e9d9b700 1 -- 172.21.15.26:0/1504162402 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029046849 0 0) 0x7f08ec0aa1e0 con 0x7f08ec0b2ab0 2022-07-20T02:25:39.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.847+0000 7f08ead9d700 1 -- 172.21.15.26:0/1504162402 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec0b1110 con 0x7f08ec0b06e0 2022-07-20T02:25:39.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08ead9d700 1 --2- 172.21.15.26:0/1504162402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec14a9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f08e40032f0 tx=0x7f08e40091d0).ready entity=mon.0 client_cookie=1a3abf0324ba5e12 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e9d9b700 1 -- 172.21.15.26:0/1504162402 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e4009ea0 con 0x7f08ec0b06e0 2022-07-20T02:25:39.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e9d9b700 1 -- 172.21.15.26:0/1504162402 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e400b8a0 con 0x7f08ec0b06e0 2022-07-20T02:25:39.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e9d9b700 1 -- 172.21.15.26:0/1504162402 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e400ba10 con 0x7f08ec0b06e0 2022-07-20T02:25:39.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- 172.21.15.26:0/1504162402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 msgr2=0x7f08ec14a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1504162402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec14a9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f08e40032f0 tx=0x7f08e40091d0).stop 2022-07-20T02:25:39.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- 172.21.15.26:0/1504162402 shutdown_connections 2022-07-20T02:25:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1504162402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec14a9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- 172.21.15.26:0/1504162402 >> 172.21.15.26:0/1504162402 conn(0x7f08ec0b3050 msgr2=0x7f08ec0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:39.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- 172.21.15.26:0/1504162402 shutdown_connections 2022-07-20T02:25:39.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- 172.21.15.26:0/1504162402 wait complete. 2022-07-20T02:25:39.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 Processor -- start 2022-07-20T02:25:39.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- start start 2022-07-20T02:25:39.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 0x7f08ec13dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2ab0 0x7f08ec09eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0abac0 con 0x7f08ec0b06e0 2022-07-20T02:25:39.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0a97e0 con 0x7f08ec0b2ab0 2022-07-20T02:25:39.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08eb59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0ab5e0 con 0x7f08ec0add30 2022-07-20T02:25:39.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35036/0 (socket says 172.21.15.26:35036) 2022-07-20T02:25:39.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08e3fff700 1 -- 172.21.15.26:0/1853779106 learned_addr learned my addr 172.21.15.26:0/1853779106 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:39.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.851+0000 7f08ea59c700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 0x7f08ec13dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08ead9d700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2ab0 0x7f08ec09eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 -- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 msgr2=0x7f08ec13dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 0x7f08ec13dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 -- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2ab0 msgr2=0x7f08ec09eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:39.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2ab0 0x7f08ec09eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:39.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 -- 172.21.15.26:0/1853779106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4009470 con 0x7f08ec0b06e0 2022-07-20T02:25:39.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08ea59c700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 0x7f08ec13dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:39.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e3fff700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08d4000bc0 tx=0x7f08d4007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08d400ce80 con 0x7f08ec0b06e0 2022-07-20T02:25:39.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d40072e0 con 0x7f08ec0b06e0 2022-07-20T02:25:39.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec0ac3c0 con 0x7f08ec0b06e0 2022-07-20T02:25:39.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08d4005800 con 0x7f08ec0b06e0 2022-07-20T02:25:39.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec0ad270 con 0x7f08ec0b06e0 2022-07-20T02:25:39.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f08d4010070 con 0x7f08ec0b06e0 2022-07-20T02:25:39.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec0a9950 con 0x7f08ec0b06e0 2022-07-20T02:25:39.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 0x7f08d0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:39.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f08d406b920 con 0x7f08ec0b06e0 2022-07-20T02:25:39.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08ea59c700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 0x7f08d0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:39.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.855+0000 7f08ea59c700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 0x7f08d0055f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f08dc00b000 tx=0x7f08dc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:39.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:39.863+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08ec0a9950 con 0x7f08ec0b06e0 2022-07-20T02:25:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.015+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f08ec0a9640 con 0x7f08ec0b06e0 2022-07-20T02:25:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.019+0000 7f08e37fe700 1 -- 172.21.15.26:0/1853779106 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+131775 (secure 0 0 0) 0x7f08ec0a9640 con 0x7f08ec0b06e0 2022-07-20T02:25:40.036 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:40.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 msgr2=0x7f08d0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 0x7f08d0055f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f08dc00b000 tx=0x7f08dc009040).stop 2022-07-20T02:25:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 msgr2=0x7f08ec09eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08d4000bc0 tx=0x7f08d4007bf0).stop 2022-07-20T02:25:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 shutdown_connections 2022-07-20T02:25:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec0b06e0 0x7f08ec09eb00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08d0053a70 0x7f08d0055f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2ab0 0x7f08ec09eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 --2- 172.21.15.26:0/1853779106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec0add30 0x7f08ec13dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 >> 172.21.15.26:0/1853779106 conn(0x7f08ec0b3050 msgr2=0x7f08ec13e930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:40.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 shutdown_connections 2022-07-20T02:25:40.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.023+0000 7f08eb59e700 1 -- 172.21.15.26:0/1853779106 wait complete. 2022-07-20T02:25:40.049 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":4,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":22865},{"type":"v1","addr":"172.21.15.26:6801","nonce":22865}]},"active_addr":"172.21.15.26:6801/22865","active_change":"2022-07-20T02:25:19.366072+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[{"gid":4114,"name":"x","mgr_features":4540138292840890367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":849642207}]},{"addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3717631609}]},{"addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":4106944390}]}]}} 2022-07-20T02:25:40.083 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-07-20T02:25:40.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-07-20T02:25:40.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:40.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.195+0000 7f5f8e340700 1 Processor -- start 2022-07-20T02:25:40.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.195+0000 7f5f8e340700 1 -- start start 2022-07-20T02:25:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5f880b30c0 0x7f5f88060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f8805f840 0x7f5f88066f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f881485b0 con 0x7f5f880952f0 2022-07-20T02:25:40.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f88095a50 con 0x7f5f880b30c0 2022-07-20T02:25:40.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8814b730 con 0x7f5f8805f840 2022-07-20T02:25:40.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5f880b30c0 0x7f5f88060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8cb3d700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f5f880952f0 0x7f5f88147ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:48534/0 (socket says 172.21.15.26:48534) 2022-07-20T02:25:40.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8cb3d700 1 -- 172.21.15.26:0/2028061416 learned_addr learned my addr 172.21.15.26:0/2028061416 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:40.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5f880b30c0 0x7f5f88060570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35044/0 (socket says 172.21.15.26:35044) 2022-07-20T02:25:40.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8db3f700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f8805f840 0x7f5f88066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f7bfff700 1 -- 172.21.15.26:0/2028061416 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231821256 0 0) 0x7f5f881485b0 con 0x7f5f880952f0 2022-07-20T02:25:40.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 -- 172.21.15.26:0/2028061416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f8805f840 msgr2=0x7f5f88066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f8805f840 0x7f5f88066f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 -- 172.21.15.26:0/2028061416 >> v1:172.21.15.120:6789/0 conn(0x7f5f880952f0 legacy=0x7f5f88147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 -- 172.21.15.26:0/2028061416 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8814b9d0 con 0x7f5f880b30c0 2022-07-20T02:25:40.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880b30c0 0x7f5f88060570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5f88147680 tx=0x7f5f84006df0).ready entity=mon.0 client_cookie=1340149adb119b34 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f7bfff700 1 -- 172.21.15.26:0/2028061416 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8400d570 con 0x7f5f880b30c0 2022-07-20T02:25:40.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f7bfff700 1 -- 172.21.15.26:0/2028061416 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84000e90 con 0x7f5f880b30c0 2022-07-20T02:25:40.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f7bfff700 1 -- 172.21.15.26:0/2028061416 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8400d8c0 con 0x7f5f880b30c0 2022-07-20T02:25:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- 172.21.15.26:0/2028061416 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880b30c0 msgr2=0x7f5f88060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880b30c0 0x7f5f88060570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5f88147680 tx=0x7f5f84006df0).stop 2022-07-20T02:25:40.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- 172.21.15.26:0/2028061416 shutdown_connections 2022-07-20T02:25:40.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f8805f840 0x7f5f88066f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 --2- 172.21.15.26:0/2028061416 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880b30c0 0x7f5f88060570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- 172.21.15.26:0/2028061416 >> 172.21.15.26:0/2028061416 conn(0x7f5f880b67f0 msgr2=0x7f5f880b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- 172.21.15.26:0/2028061416 shutdown_connections 2022-07-20T02:25:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- 172.21.15.26:0/2028061416 wait complete. 2022-07-20T02:25:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 Processor -- start 2022-07-20T02:25:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.199+0000 7f5f8e340700 1 -- start start 2022-07-20T02:25:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 0x7f5f8814e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f880b30c0 0x7f5f88146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88065650 con 0x7f5f880952f0 2022-07-20T02:25:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88063060 con 0x7f5f8805f840 2022-07-20T02:25:40.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88063300 con 0x7f5f880b30c0 2022-07-20T02:25:40.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35050/0 (socket says 172.21.15.26:35050) 2022-07-20T02:25:40.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 -- 172.21.15.26:0/1238796806 learned_addr learned my addr 172.21.15.26:0/1238796806 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8db3f700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f880b30c0 0x7f5f88146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 0x7f5f8814e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 -- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f880b30c0 msgr2=0x7f5f88146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f880b30c0 0x7f5f88146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 -- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 msgr2=0x7f5f8814e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 0x7f5f8814e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 -- 172.21.15.26:0/1238796806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8400b180 con 0x7f5f880952f0 2022-07-20T02:25:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 0x7f5f8814e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:40.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8cb3d700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5f7c000fe0 tx=0x7f5f7c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f7c0072e0 con 0x7f5f880952f0 2022-07-20T02:25:40.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c007440 con 0x7f5f880952f0 2022-07-20T02:25:40.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f880635a0 con 0x7f5f880952f0 2022-07-20T02:25:40.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f7c005800 con 0x7f5f880952f0 2022-07-20T02:25:40.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.203+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8809b890 con 0x7f5f880952f0 2022-07-20T02:25:40.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f7c01a060 con 0x7f5f880952f0 2022-07-20T02:25:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f88147760 con 0x7f5f880952f0 2022-07-20T02:25:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f7a7fc700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 0x7f5f70055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5f7c06b8d0 con 0x7f5f880952f0 2022-07-20T02:25:40.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 0x7f5f70055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.207+0000 7f5f8d33e700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 0x7f5f70055e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5f84018000 tx=0x7f5f84014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.211+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f7c042690 con 0x7f5f880952f0 2022-07-20T02:25:40.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.339+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f880574e0 con 0x7f5f880952f0 2022-07-20T02:25:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f7a7fc700 1 -- 172.21.15.26:0/1238796806 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f5f7c018070 con 0x7f5f880952f0 2022-07-20T02:25:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:40.347 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 msgr2=0x7f5f70055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 0x7f5f70055e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5f84018000 tx=0x7f5f84014000).stop 2022-07-20T02:25:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 msgr2=0x7f5f881469c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5f7c000fe0 tx=0x7f5f7c007b50).stop 2022-07-20T02:25:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 shutdown_connections 2022-07-20T02:25:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f880952f0 0x7f5f881469c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f70053980 0x7f5f70055e40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f880b30c0 0x7f5f88146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 --2- 172.21.15.26:0/1238796806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f8805f840 0x7f5f8814e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 >> 172.21.15.26:0/1238796806 conn(0x7f5f880b67f0 msgr2=0x7f5f880b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 shutdown_connections 2022-07-20T02:25:40.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.343+0000 7f5f8e340700 1 -- 172.21.15.26:0/1238796806 wait complete. 2022-07-20T02:25:40.367 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-07-20T02:25:40.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:40.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 Processor -- start 2022-07-20T02:25:40.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 -- start start 2022-07-20T02:25:40.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fe00aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00b62d0 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe014aac0 con 0x7f8fe00b5ef0 2022-07-20T02:25:40.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5b74700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe014ad60 con 0x7f8fe0147be0 2022-07-20T02:25:40.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5373700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8fe00a7fc0 0x7f8fe00af330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34338/0 (socket says 172.21.15.26:34338) 2022-07-20T02:25:40.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fe5373700 1 -- 172.21.15.26:0/1724155285 learned_addr learned my addr 172.21.15.26:0/1724155285 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:40.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdffff700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fe00aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726731977 0 0) 0x7f8fe00b62d0 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4003260 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453255362 0 0) 0x7f8fc4003260 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 msgr2=0x7f8fe00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.487+0000 7f8fdf7fe700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fe00aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 >> v1:172.21.15.120:6789/0 conn(0x7f8fe00b5ef0 legacy=0x7f8fe01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe014b000 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f8fd80019e0 con 0x7f8fe00a7fc0 2022-07-20T02:25:40.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 >> v1:172.21.15.26:6789/0 conn(0x7f8fe00a7fc0 legacy=0x7f8fe00af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc4003780 0x7f8fc4003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fc4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fc4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe014ad60 con 0x7f8fc40049f0 2022-07-20T02:25:40.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc400b780 con 0x7f8fc4003780 2022-07-20T02:25:40.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc400b940 con 0x7f8fe0147be0 2022-07-20T02:25:40.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fc4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe4b72700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc4003780 0x7f8fc4003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdffff700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fc4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 -- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 msgr2=0x7f8fc4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe0147be0 0x7f8fc4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 -- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc4003780 msgr2=0x7f8fc4003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc4003780 0x7f8fc4003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 -- 172.21.15.26:0/1724155285 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4003260 con 0x7f8fc40049f0 2022-07-20T02:25:40.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fc4004e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8fd80016f0 tx=0x7f8fe00ace20).ready entity=mon.0 client_cookie=c5718fe1af7756b3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fd800cca0 con 0x7f8fc40049f0 2022-07-20T02:25:40.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd800ce00 con 0x7f8fc40049f0 2022-07-20T02:25:40.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fdf7fe700 1 -- 172.21.15.26:0/1724155285 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fd8005b80 con 0x7f8fc40049f0 2022-07-20T02:25:40.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 -- 172.21.15.26:0/1724155285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 msgr2=0x7f8fc4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/1724155285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fc4004e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8fd80016f0 tx=0x7f8fe00ace20).stop 2022-07-20T02:25:40.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5373700 1 -- 172.21.15.26:0/1724155285 reap_dead start 2022-07-20T02:25:40.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 -- 172.21.15.26:0/1724155285 shutdown_connections 2022-07-20T02:25:40.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 -- 172.21.15.26:0/1724155285 >> 172.21.15.26:0/1724155285 conn(0x7f8fe0060310 msgr2=0x7f8fe00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:40.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 -- 172.21.15.26:0/1724155285 shutdown_connections 2022-07-20T02:25:40.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.491+0000 7f8fe5b74700 1 -- 172.21.15.26:0/1724155285 wait complete. 2022-07-20T02:25:40.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 Processor -- start 2022-07-20T02:25:40.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- start start 2022-07-20T02:25:40.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00aaf00 0x7f8fe01475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 0x7f8fe0156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0067780 con 0x7f8fc40049f0 2022-07-20T02:25:40.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00ab480 con 0x7f8fe0144a30 2022-07-20T02:25:40.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0156530 con 0x7f8fe00aaf00 2022-07-20T02:25:40.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35068/0 (socket says 172.21.15.26:35068) 2022-07-20T02:25:40.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 -- 172.21.15.26:0/4281023290 learned_addr learned my addr 172.21.15.26:0/4281023290 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:40.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5373700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 0x7f8fe0156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fdffff700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00aaf00 0x7f8fe01475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 -- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00aaf00 msgr2=0x7f8fe01475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00aaf00 0x7f8fe01475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 -- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 msgr2=0x7f8fe0156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 0x7f8fe0156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 -- 172.21.15.26:0/4281023290 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe01567d0 con 0x7f8fc40049f0 2022-07-20T02:25:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5373700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 0x7f8fe0156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe4b72700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8fcc00b000 tx=0x7f8fcc009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fcc0155f0 con 0x7f8fc40049f0 2022-07-20T02:25:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc010040 con 0x7f8fc40049f0 2022-07-20T02:25:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe0156a70 con 0x7f8fc40049f0 2022-07-20T02:25:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fcc015940 con 0x7f8fc40049f0 2022-07-20T02:25:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.495+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe0157400 con 0x7f8fc40049f0 2022-07-20T02:25:40.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0156c00 con 0x7f8fc40049f0 2022-07-20T02:25:40.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8fcc015aa0 con 0x7f8fc40049f0 2022-07-20T02:25:40.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fddffb700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 0x7f8fc0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8fcc06c280 con 0x7f8fc40049f0 2022-07-20T02:25:40.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fdffff700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 0x7f8fc0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.499+0000 7f8fdffff700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 0x7f8fc0055f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8fd400b000 tx=0x7f8fd4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.507+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fcc042ed0 con 0x7f8fc40049f0 2022-07-20T02:25:40.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.635+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe014b2a0 con 0x7f8fc40049f0 2022-07-20T02:25:40.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.635+0000 7f8fddffb700 1 -- 172.21.15.26:0/4281023290 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f8fcc0464f0 con 0x7f8fc40049f0 2022-07-20T02:25:40.692 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:40.692 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 msgr2=0x7f8fc0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 0x7f8fc0055f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8fd400b000 tx=0x7f8fd4008040).stop 2022-07-20T02:25:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 msgr2=0x7f8fe00abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8fcc00b000 tx=0x7f8fcc009040).stop 2022-07-20T02:25:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 shutdown_connections 2022-07-20T02:25:40.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc0053a70 0x7f8fc0055f30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0144a30 0x7f8fe0156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00aaf00 0x7f8fe01475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 --2- 172.21.15.26:0/4281023290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fc40049f0 0x7f8fe00abe80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 >> 172.21.15.26:0/4281023290 conn(0x7f8fe0060310 msgr2=0x7f8fe00aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 shutdown_connections 2022-07-20T02:25:40.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.639+0000 7f8fe5b74700 1 -- 172.21.15.26:0/4281023290 wait complete. 2022-07-20T02:25:40.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-07-20T02:25:40.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.863+0000 7fd82e5a6700 1 Processor -- start 2022-07-20T02:25:40.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.863+0000 7fd82e5a6700 1 -- start start 2022-07-20T02:25:40.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8280b1c60 0x7fd8280a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828098320 0x7fd828060020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd828060510 0x7fd828143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:40.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd828098800 con 0x7fd828060510 2022-07-20T02:25:40.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd828098aa0 con 0x7fd8280b1c60 2022-07-20T02:25:40.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82e5a6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82805f8e0 con 0x7fd828098320 2022-07-20T02:25:40.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd828060510 0x7fd828143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd828060510 0x7fd828143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35080/0 (socket says 172.21.15.26:35080) 2022-07-20T02:25:40.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 -- 172.21.15.26:0/95846899 learned_addr learned my addr 172.21.15.26:0/95846899 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:40.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd81ffff700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828098320 0x7fd828060020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd827fff700 1 --2- 172.21.15.26:0/95846899 >> v2:172.21.15.120:3300/0 conn(0x7fd8280b1c60 0x7fd8280a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:40.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 -- 172.21.15.26:0/95846899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828098320 msgr2=0x7fd828060020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828098320 0x7fd828060020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 -- 172.21.15.26:0/95846899 >> v2:172.21.15.120:3300/0 conn(0x7fd8280b1c60 msgr2=0x7fd8280a9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:40.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 --2- 172.21.15.26:0/95846899 >> v2:172.21.15.120:3300/0 conn(0x7fd8280b1c60 0x7fd8280a9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:40.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 -- 172.21.15.26:0/95846899 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82805fb80 con 0x7fd828060510 2022-07-20T02:25:40.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd82cb43700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828060510 0x7fd828143710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd828061770 tx=0x7fd82000b530).ready entity=mon.0 client_cookie=f0cbb934daad7174 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:40.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd8277fe700 1 -- 172.21.15.26:0/95846899 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd82000dd30 con 0x7fd828060510 2022-07-20T02:25:41.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd8277fe700 1 -- 172.21.15.26:0/95846899 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82000de90 con 0x7fd828060510 2022-07-20T02:25:41.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.867+0000 7fd8277fe700 1 -- 172.21.15.26:0/95846899 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd820010660 con 0x7fd828060510 2022-07-20T02:25:41.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- 172.21.15.26:0/95846899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828060510 msgr2=0x7fd828143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828060510 0x7fd828143710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd828061770 tx=0x7fd82000b530).stop 2022-07-20T02:25:41.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- 172.21.15.26:0/95846899 shutdown_connections 2022-07-20T02:25:41.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828060510 0x7fd828143710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/95846899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828098320 0x7fd828060020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/95846899 >> v2:172.21.15.120:3300/0 conn(0x7fd8280b1c60 0x7fd8280a9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- 172.21.15.26:0/95846899 >> 172.21.15.26:0/95846899 conn(0x7fd8280b5e70 msgr2=0x7fd8280b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- 172.21.15.26:0/95846899 shutdown_connections 2022-07-20T02:25:41.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- 172.21.15.26:0/95846899 wait complete. 2022-07-20T02:25:41.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 Processor -- start 2022-07-20T02:25:41.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.871+0000 7fd82e5a6700 1 -- start start 2022-07-20T02:25:41.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828060510 0x7fd828143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 0x7fd8280ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8280b1c60 0x7fd828065930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8280aca10 con 0x7fd828098320 2022-07-20T02:25:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8280aad30 con 0x7fd8280b1c60 2022-07-20T02:25:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8280aafd0 con 0x7fd828060510 2022-07-20T02:25:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 0x7fd8280ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 0x7fd8280ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35084/0 (socket says 172.21.15.26:35084) 2022-07-20T02:25:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 -- 172.21.15.26:0/1050289519 learned_addr learned my addr 172.21.15.26:0/1050289519 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd827fff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828060510 0x7fd828143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82cb43700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8280b1c60 0x7fd828065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 -- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828060510 msgr2=0x7fd828143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd828060510 0x7fd828143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 -- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8280b1c60 msgr2=0x7fd828065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8280b1c60 0x7fd828065930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82000b7d0 con 0x7fd828098320 2022-07-20T02:25:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd81ffff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 0x7fd8280ab710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd81000c000 tx=0x7fd810006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd810010780 con 0x7fd828098320 2022-07-20T02:25:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8100108e0 con 0x7fd828098320 2022-07-20T02:25:41.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8280ab270 con 0x7fd828098320 2022-07-20T02:25:41.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd810010c40 con 0x7fd828098320 2022-07-20T02:25:41.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8280abc00 con 0x7fd828098320 2022-07-20T02:25:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.875+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8280ab400 con 0x7fd828098320 2022-07-20T02:25:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd810026430 con 0x7fd828098320 2022-07-20T02:25:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd80c053b30 0x7fd80c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd81006ceb0 con 0x7fd828098320 2022-07-20T02:25:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd827fff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd80c053b30 0x7fd80c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fd80c05b4b0 0x7fd80c05d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd80c05df30 con 0x7fd80c05b4b0 2022-07-20T02:25:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd81006d290 con 0x7fd828098320 2022-07-20T02:25:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd82cb43700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fd80c05b4b0 0x7fd80c05d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd827fff700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd80c053b30 0x7fd80c055ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd81800b000 tx=0x7fd818006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.879+0000 7fd82cb43700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fd80c05b4b0 0x7fd80c05d950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.883+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== osd.0 v2:172.21.15.26:6802/23049 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fd80c05df30 con 0x7fd80c05b4b0 2022-07-20T02:25:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 --> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8280660f0 con 0x7fd80c05b4b0 2022-07-20T02:25:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd825ffb700 1 -- 172.21.15.26:0/1050289519 <== osd.0 v2:172.21.15.26:6802/23049 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8280660f0 con 0x7fd80c05b4b0 2022-07-20T02:25:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fd80c05b4b0 msgr2=0x7fd80c05d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fd80c05b4b0 0x7fd80c05d950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd80c053b30 msgr2=0x7fd80c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd80c053b30 0x7fd80c055ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd81800b000 tx=0x7fd818006df0).stop 2022-07-20T02:25:41.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 msgr2=0x7fd8280ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 --2- 172.21.15.26:0/1050289519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd828098320 0x7fd8280ab710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd81000c000 tx=0x7fd810006df0).stop 2022-07-20T02:25:41.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82cb43700 1 -- 172.21.15.26:0/1050289519 reap_dead start 2022-07-20T02:25:41.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 shutdown_connections 2022-07-20T02:25:41.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 >> 172.21.15.26:0/1050289519 conn(0x7fd8280b5e70 msgr2=0x7fd8280b3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 shutdown_connections 2022-07-20T02:25:41.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:40.899+0000 7fd82e5a6700 1 -- 172.21.15.26:0/1050289519 wait complete. 2022-07-20T02:25:41.053 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:41.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-07-20T02:25:41.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 Processor -- start 2022-07-20T02:25:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 -- start start 2022-07-20T02:25:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 0x7f530014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300064880 con 0x7f5300064450 2022-07-20T02:25:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53000b35a0 con 0x7f53000670c0 2022-07-20T02:25:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530599e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000b85f0 con 0x7f53001444a0 2022-07-20T02:25:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52fffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5300064450 0x7f5300064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34374/0 (socket says 172.21.15.26:34374) 2022-07-20T02:25:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52fffff700 1 -- 172.21.15.26:0/2004916348 learned_addr learned my addr 172.21.15.26:0/2004916348 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530519d700 1 --2- 172.21.15.26:0/2004916348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 0x7f530014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358444054 0 0) 0x7f5300064880 con 0x7f5300064450 2022-07-20T02:25:41.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0003260 con 0x7f5300064450 2022-07-20T02:25:41.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383634561 0 0) 0x7f53000b35a0 con 0x7f53000670c0 2022-07-20T02:25:41.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300064880 con 0x7f53000670c0 2022-07-20T02:25:41.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530519d700 1 -- 172.21.15.26:0/2004916348 >> v1:172.21.15.120:6789/0 conn(0x7f53000670c0 legacy=0x7f53000b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530519d700 1 -- 172.21.15.26:0/2004916348 >> v1:172.21.15.26:6789/0 conn(0x7f5300064450 legacy=0x7f5300064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530519d700 1 -- 172.21.15.26:0/2004916348 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53000b80b0 con 0x7f53001444a0 2022-07-20T02:25:41.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f530519d700 1 --2- 172.21.15.26:0/2004916348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 0x7f530014c850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f52f0002f70 tx=0x7f52f0006df0).ready entity=mon.2 client_cookie=7a3f2c22a9da70c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52f000d5c0 con 0x7f53001444a0 2022-07-20T02:25:41.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f0000e90 con 0x7f53001444a0 2022-07-20T02:25:41.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.171+0000 7f52ff7fe700 1 -- 172.21.15.26:0/2004916348 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52f000d910 con 0x7f53001444a0 2022-07-20T02:25:41.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- 172.21.15.26:0/2004916348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 msgr2=0x7f530014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 --2- 172.21.15.26:0/2004916348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 0x7f530014c850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f52f0002f70 tx=0x7f52f0006df0).stop 2022-07-20T02:25:41.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- 172.21.15.26:0/2004916348 shutdown_connections 2022-07-20T02:25:41.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 --2- 172.21.15.26:0/2004916348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53001444a0 0x7f530014c850 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- 172.21.15.26:0/2004916348 >> 172.21.15.26:0/2004916348 conn(0x7f530005f840 msgr2=0x7f530005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- 172.21.15.26:0/2004916348 shutdown_connections 2022-07-20T02:25:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- 172.21.15.26:0/2004916348 wait complete. 2022-07-20T02:25:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 Processor -- start 2022-07-20T02:25:41.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- start start 2022-07-20T02:25:41.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53000670c0 0x7f53000adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53001444a0 0x7f5300152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 0x7f53000a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5300062890 con 0x7f53001444a0 2022-07-20T02:25:41.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530014a820 con 0x7f53000670c0 2022-07-20T02:25:41.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530599e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000625f0 con 0x7f5300143260 2022-07-20T02:25:41.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530519d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 0x7f53000a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530519d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 0x7f53000a9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:56618/0 (socket says 172.21.15.26:56618) 2022-07-20T02:25:41.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f52fffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53001444a0 0x7f5300152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530519d700 1 -- 172.21.15.26:0/3898524876 learned_addr learned my addr 172.21.15.26:0/3898524876 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.175+0000 7f530499c700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53000670c0 0x7f53000adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 -- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53000670c0 msgr2=0x7f53000adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53000670c0 0x7f53000adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 -- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53001444a0 msgr2=0x7f5300152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53001444a0 0x7f5300152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f000b180 con 0x7f5300143260 2022-07-20T02:25:41.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f52fffff700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53001444a0 0x7f5300152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:41.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530519d700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 0x7f53000a9f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f52f0002f70 tx=0x7f52f0014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52f0016070 con 0x7f5300143260 2022-07-20T02:25:41.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f000de40 con 0x7f5300143260 2022-07-20T02:25:41.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53000ad7e0 con 0x7f5300143260 2022-07-20T02:25:41.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52f0013d60 con 0x7f5300143260 2022-07-20T02:25:41.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5300152700 con 0x7f5300143260 2022-07-20T02:25:41.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5300062450 con 0x7f5300143260 2022-07-20T02:25:41.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.179+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f52f0025030 con 0x7f5300143260 2022-07-20T02:25:41.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f52e4053a70 0x7f52e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f52f006ce70 con 0x7f5300143260 2022-07-20T02:25:41.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f530499c700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f52e4053a70 0x7f52e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f52e405b3f0 0x7f52e405d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52e405de70 con 0x7f52e405b3f0 2022-07-20T02:25:41.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52f006f490 con 0x7f5300143260 2022-07-20T02:25:41.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fffff700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f52e405b3f0 0x7f52e405d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f530499c700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f52e4053a70 0x7f52e4055f30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f52e800d000 tx=0x7f52e8007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fffff700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f52e405b3f0 0x7f52e405d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.183+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== osd.1 v2:172.21.15.26:6810/23050 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f52e405de70 con 0x7f52e405b3f0 2022-07-20T02:25:41.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.203+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 --> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f53000ad970 con 0x7f52e405b3f0 2022-07-20T02:25:41.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.203+0000 7f52fdffb700 1 -- 172.21.15.26:0/3898524876 <== osd.1 v2:172.21.15.26:6810/23050 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f53000ad970 con 0x7f52e405b3f0 2022-07-20T02:25:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.203+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f52e405b3f0 msgr2=0x7f52e405d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.203+0000 7f530599e700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f52e405b3f0 0x7f52e405d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f52e4053a70 msgr2=0x7f52e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f52e4053a70 0x7f52e4055f30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f52e800d000 tx=0x7f52e8007a60).stop 2022-07-20T02:25:41.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 msgr2=0x7f53000a9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 --2- 172.21.15.26:0/3898524876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5300143260 0x7f53000a9f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f52f0002f70 tx=0x7f52f0014000).stop 2022-07-20T02:25:41.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530519d700 1 -- 172.21.15.26:0/3898524876 reap_dead start 2022-07-20T02:25:41.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 shutdown_connections 2022-07-20T02:25:41.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 >> 172.21.15.26:0/3898524876 conn(0x7f530005f840 msgr2=0x7f5300065240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 shutdown_connections 2022-07-20T02:25:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.207+0000 7f530599e700 1 -- 172.21.15.26:0/3898524876 wait complete. 2022-07-20T02:25:41.334 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:41.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-07-20T02:25:41.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.459+0000 7f25a83b5700 1 Processor -- start 2022-07-20T02:25:41.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.459+0000 7f25a83b5700 1 -- start start 2022-07-20T02:25:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a00b1bc0 con 0x7f25a0061050 2022-07-20T02:25:41.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a0147700 con 0x7f25a00b2920 2022-07-20T02:25:41.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a014b040 con 0x7f25a01484a0 2022-07-20T02:25:41.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a6952700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25a0061050 0x7f25a0066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34392/0 (socket says 172.21.15.26:34392) 2022-07-20T02:25:41.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a6952700 1 -- 172.21.15.26:0/1564744475 learned_addr learned my addr 172.21.15.26:0/1564744475 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:41.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a5950700 1 --2- 172.21.15.26:0/1564744475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649472207 0 0) 0x7f25a00b1bc0 con 0x7f25a0061050 2022-07-20T02:25:41.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588003260 con 0x7f25a0061050 2022-07-20T02:25:41.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692377132 0 0) 0x7f25a0147700 con 0x7f25a00b2920 2022-07-20T02:25:41.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a00b1bc0 con 0x7f25a00b2920 2022-07-20T02:25:41.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a5950700 1 -- 172.21.15.26:0/1564744475 >> v1:172.21.15.120:6789/0 conn(0x7f25a00b2920 legacy=0x7f25a0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a5950700 1 -- 172.21.15.26:0/1564744475 >> v1:172.21.15.26:6789/0 conn(0x7f25a0061050 legacy=0x7f25a0066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a5950700 1 -- 172.21.15.26:0/1564744475 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0147930 con 0x7f25a01484a0 2022-07-20T02:25:41.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a5950700 1 --2- 172.21.15.26:0/1564744475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0066a80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f2590002f70 tx=0x7f2590006df0).ready entity=mon.2 client_cookie=f8926f62c957acba server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259000d5c0 con 0x7f25a01484a0 2022-07-20T02:25:41.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2590000e90 con 0x7f25a01484a0 2022-07-20T02:25:41.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a514f700 1 -- 172.21.15.26:0/1564744475 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259000d910 con 0x7f25a01484a0 2022-07-20T02:25:41.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- 172.21.15.26:0/1564744475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 msgr2=0x7f25a0066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 --2- 172.21.15.26:0/1564744475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0066a80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f2590002f70 tx=0x7f2590006df0).stop 2022-07-20T02:25:41.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- 172.21.15.26:0/1564744475 shutdown_connections 2022-07-20T02:25:41.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 --2- 172.21.15.26:0/1564744475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0066a80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- 172.21.15.26:0/1564744475 >> 172.21.15.26:0/1564744475 conn(0x7f25a00b60a0 msgr2=0x7f25a00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- 172.21.15.26:0/1564744475 shutdown_connections 2022-07-20T02:25:41.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.463+0000 7f25a83b5700 1 -- 172.21.15.26:0/1564744475 wait complete. 2022-07-20T02:25:41.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 Processor -- start 2022-07-20T02:25:41.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 -- start start 2022-07-20T02:25:41.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 0x7f25a00ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a00b2920 0x7f25a0156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a0064300 con 0x7f25a0061050 2022-07-20T02:25:41.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00667f0 con 0x7f25a00b2920 2022-07-20T02:25:41.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a83b5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a0064c80 con 0x7f25a01484a0 2022-07-20T02:25:41.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a6151700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 0x7f25a00ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a6151700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 0x7f25a00ab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35114/0 (socket says 172.21.15.26:35114) 2022-07-20T02:25:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.467+0000 7f25a6151700 1 -- 172.21.15.26:0/4071706010 learned_addr learned my addr 172.21.15.26:0/4071706010 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a5950700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a00b2920 0x7f25a0156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6952700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 -- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 msgr2=0x7f25a0156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25a01484a0 0x7f25a0156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 -- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a00b2920 msgr2=0x7f25a0156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25a00b2920 0x7f25a0156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259000b180 con 0x7f25a0061050 2022-07-20T02:25:41.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a6151700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 0x7f25a00ab300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2598001490 tx=0x7f2598007ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2598010510 con 0x7f25a0061050 2022-07-20T02:25:41.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a01570f0 con 0x7f25a0061050 2022-07-20T02:25:41.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2598010670 con 0x7f25a0061050 2022-07-20T02:25:41.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25980107e0 con 0x7f25a0061050 2022-07-20T02:25:41.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a00a99a0 con 0x7f25a0061050 2022-07-20T02:25:41.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25a0064e40 con 0x7f25a0061050 2022-07-20T02:25:41.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2598010070 con 0x7f25a0061050 2022-07-20T02:25:41.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2580053b30 0x7f2580055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f25a5950700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2580053b30 0x7f2580055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f259806be10 con 0x7f25a0061050 2022-07-20T02:25:41.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f258005b430 0x7f258005d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.471+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f258005deb0 con 0x7f258005b430 2022-07-20T02:25:41.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.475+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f259806e480 con 0x7f25a0061050 2022-07-20T02:25:41.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.475+0000 7f25a6952700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f258005b430 0x7f258005d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.475+0000 7f25a5950700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2580053b30 0x7f2580055ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2590018000 tx=0x7f2590014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.475+0000 7f25a6952700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f258005b430 0x7f258005d8d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.475+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== osd.2 v2:172.21.15.26:6818/23051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f258005deb0 con 0x7f258005b430 2022-07-20T02:25:41.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 --> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25a00abb10 con 0x7f258005b430 2022-07-20T02:25:41.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f258f7fe700 1 -- 172.21.15.26:0/4071706010 <== osd.2 v2:172.21.15.26:6818/23051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25a00abb10 con 0x7f258005b430 2022-07-20T02:25:41.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f258005b430 msgr2=0x7f258005d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f258005b430 0x7f258005d8d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2580053b30 msgr2=0x7f2580055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2580053b30 0x7f2580055ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2590018000 tx=0x7f2590014000).stop 2022-07-20T02:25:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 msgr2=0x7f25a00ab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 --2- 172.21.15.26:0/4071706010 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25a0061050 0x7f25a00ab300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2598001490 tx=0x7f2598007ee0).stop 2022-07-20T02:25:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a6952700 1 -- 172.21.15.26:0/4071706010 reap_dead start 2022-07-20T02:25:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 shutdown_connections 2022-07-20T02:25:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 >> 172.21.15.26:0/4071706010 conn(0x7f25a00b60a0 msgr2=0x7f25a0146ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 shutdown_connections 2022-07-20T02:25:41.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.495+0000 7f25a83b5700 1 -- 172.21.15.26:0/4071706010 wait complete. 2022-07-20T02:25:41.549 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:41.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-07-20T02:25:41.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.675+0000 7f13094e9700 1 Processor -- start 2022-07-20T02:25:41.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.675+0000 7f13094e9700 1 -- start start 2022-07-20T02:25:41.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13041465b0 0x7f1304066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040b1ae0 con 0x7f13040672a0 2022-07-20T02:25:41.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040b1d80 con 0x7f13040b24c0 2022-07-20T02:25:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040b2e20 con 0x7f13041465b0 2022-07-20T02:25:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1308ce8700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f13040672a0 0x7f1304146990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34408/0 (socket says 172.21.15.26:34408) 2022-07-20T02:25:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1308ce8700 1 -- 172.21.15.26:0/3344741725 learned_addr learned my addr 172.21.15.26:0/3344741725 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1303fff700 1 --2- 172.21.15.26:0/3344741725 >> v2:172.21.15.120:3300/0 conn(0x7f13041465b0 0x7f1304066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980004148 0 0) 0x7f13040b1ae0 con 0x7f13040672a0 2022-07-20T02:25:41.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e4003260 con 0x7f13040672a0 2022-07-20T02:25:41.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343427444 0 0) 0x7f13040b1d80 con 0x7f13040b24c0 2022-07-20T02:25:41.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13040b1ae0 con 0x7f13040b24c0 2022-07-20T02:25:41.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884147351 0 0) 0x7f12e4003260 con 0x7f13040672a0 2022-07-20T02:25:41.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 >> v2:172.21.15.120:3300/0 conn(0x7f13041465b0 msgr2=0x7f1304066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:41.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 --2- 172.21.15.26:0/3344741725 >> v2:172.21.15.120:3300/0 conn(0x7f13041465b0 0x7f1304066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 >> v1:172.21.15.120:6789/0 conn(0x7f13040b24c0 legacy=0x7f13040604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040b2020 con 0x7f13040672a0 2022-07-20T02:25:41.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f12fc0019e0 con 0x7f13040672a0 2022-07-20T02:25:41.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 >> v1:172.21.15.26:6789/0 conn(0x7f13040672a0 legacy=0x7f1304146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40038e0 0x7f12e4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 0x7f12e4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004e70 0x7f12e400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040b2e20 con 0x7f12e4004500 2022-07-20T02:25:41.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e400bcf0 con 0x7f12e40038e0 2022-07-20T02:25:41.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e400beb0 con 0x7f12e4004e70 2022-07-20T02:25:41.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 0x7f12e4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1303fff700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40038e0 0x7f12e4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1308ce8700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004e70 0x7f12e400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 -- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004e70 msgr2=0x7f12e400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:41.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004e70 0x7f12e400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 -- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40038e0 msgr2=0x7f12e4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:41.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40038e0 0x7f12e4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 -- 172.21.15.26:0/3344741725 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4003260 con 0x7f12e4004500 2022-07-20T02:25:41.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f1308ce8700 1 -- 172.21.15.26:0/3344741725 reap_dead start 2022-07-20T02:25:41.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f12fbfff700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 0x7f12e4004980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f12ec001110 tx=0x7f12ec007a00).ready entity=mon.0 client_cookie=74a53181c0179579 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec007d10 con 0x7f12e4004500 2022-07-20T02:25:41.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec007070 con 0x7f12e4004500 2022-07-20T02:25:41.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13037fe700 1 -- 172.21.15.26:0/3344741725 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec0073d0 con 0x7f12e4004500 2022-07-20T02:25:41.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 msgr2=0x7f12e4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 0x7f12e4004980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f12ec001110 tx=0x7f12ec007a00).stop 2022-07-20T02:25:41.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- 172.21.15.26:0/3344741725 shutdown_connections 2022-07-20T02:25:41.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 --2- 172.21.15.26:0/3344741725 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e4004500 0x7f12e4004980 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- 172.21.15.26:0/3344741725 >> 172.21.15.26:0/3344741725 conn(0x7f13040b3050 msgr2=0x7f13040b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- 172.21.15.26:0/3344741725 shutdown_connections 2022-07-20T02:25:41.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.679+0000 7f13094e9700 1 -- 172.21.15.26:0/3344741725 wait complete. 2022-07-20T02:25:41.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 Processor -- start 2022-07-20T02:25:41.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 -- start start 2022-07-20T02:25:41.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004500 0x7f13040639a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 0x7f13040ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13040ab840 0x7f1304141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040ac230 con 0x7f1304063e90 2022-07-20T02:25:41.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040acba0 con 0x7f13040ab840 2022-07-20T02:25:41.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f13094e9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040ace40 con 0x7f12e4004500 2022-07-20T02:25:41.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 0x7f13040ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 0x7f13040ab350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35138/0 (socket says 172.21.15.26:35138) 2022-07-20T02:25:41.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 -- 172.21.15.26:0/1622155329 learned_addr learned my addr 172.21.15.26:0/1622155329 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:41.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f1303fff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004500 0x7f13040639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f1308ce8700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13040ab840 0x7f1304141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 -- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004500 msgr2=0x7f13040639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e4004500 0x7f13040639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 -- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13040ab840 msgr2=0x7f1304141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13040ab840 0x7f1304141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304152f20 con 0x7f1304063e90 2022-07-20T02:25:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.683+0000 7f12fbfff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 0x7f13040ab350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12ec001110 tx=0x7f12ec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec007d10 con 0x7f1304063e90 2022-07-20T02:25:41.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec013450 con 0x7f1304063e90 2022-07-20T02:25:41.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13041531c0 con 0x7f1304063e90 2022-07-20T02:25:41.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec0137b0 con 0x7f1304063e90 2022-07-20T02:25:41.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13040ac750 con 0x7f1304063e90 2022-07-20T02:25:41.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1304153350 con 0x7f1304063e90 2022-07-20T02:25:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f12ec026070 con 0x7f1304063e90 2022-07-20T02:25:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12e8053aa0 0x7f12e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f12ec06c8b0 con 0x7f1304063e90 2022-07-20T02:25:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1303fff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12e8053aa0 0x7f12e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f12e805b420 0x7f12e805d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12e805dea0 con 0x7f12e805b420 2022-07-20T02:25:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.687+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12ec06cc90 con 0x7f1304063e90 2022-07-20T02:25:41.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.691+0000 7f1303fff700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12e8053aa0 0x7f12e8055f60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f12f400b000 tx=0x7f12f4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.691+0000 7f1308ce8700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f12e805b420 0x7f12e805d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.691+0000 7f1308ce8700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f12e805b420 0x7f12e805d8c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.691+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== osd.3 v2:172.21.15.26:6826/23052 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f12e805dea0 con 0x7f12e805b420 2022-07-20T02:25:41.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.707+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 --> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f13040b4770 con 0x7f12e805b420 2022-07-20T02:25:41.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f1301ffb700 1 -- 172.21.15.26:0/1622155329 <== osd.3 v2:172.21.15.26:6826/23052 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f13040b4770 con 0x7f12e805b420 2022-07-20T02:25:41.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f12e805b420 msgr2=0x7f12e805d8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f12e805b420 0x7f12e805d8c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12e8053aa0 msgr2=0x7f12e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12e8053aa0 0x7f12e8055f60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f12f400b000 tx=0x7f12f4008040).stop 2022-07-20T02:25:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 msgr2=0x7f13040ab350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 --2- 172.21.15.26:0/1622155329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1304063e90 0x7f13040ab350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12ec001110 tx=0x7f12ec015000).stop 2022-07-20T02:25:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f1308ce8700 1 -- 172.21.15.26:0/1622155329 reap_dead start 2022-07-20T02:25:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 shutdown_connections 2022-07-20T02:25:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 >> 172.21.15.26:0/1622155329 conn(0x7f13040b3050 msgr2=0x7f130413ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 shutdown_connections 2022-07-20T02:25:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.711+0000 7f13094e9700 1 -- 172.21.15.26:0/1622155329 wait complete. 2022-07-20T02:25:41.780 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:41.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-07-20T02:25:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.899+0000 7f915bfff700 1 Processor -- start 2022-07-20T02:25:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.899+0000 7f915bfff700 1 -- start start 2022-07-20T02:25:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f915c0b5ef0 0x7f915c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 0x7f915c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c0b62d0 con 0x7f915c0a7fc0 2022-07-20T02:25:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c14aac0 con 0x7f915c0b5ef0 2022-07-20T02:25:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c14ad60 con 0x7f915c0a8c40 2022-07-20T02:25:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 0x7f915c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f915c0b5ef0 0x7f915c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915a7fc700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f915c0a7fc0 0x7f915c0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:48640/0 (socket says 172.21.15.26:48640) 2022-07-20T02:25:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915a7fc700 1 -- 172.21.15.26:0/342354797 learned_addr learned my addr 172.21.15.26:0/342354797 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:41.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f9159ffb700 1 -- 172.21.15.26:0/342354797 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808490314 0 0) 0x7f915c0b62d0 con 0x7f915c0a7fc0 2022-07-20T02:25:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f9159ffb700 1 -- 172.21.15.26:0/342354797 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9140003260 con 0x7f915c0a7fc0 2022-07-20T02:25:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 -- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 msgr2=0x7f915c0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 0x7f915c0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 -- 172.21.15.26:0/342354797 >> v1:172.21.15.120:6789/0 conn(0x7f915c0a7fc0 legacy=0x7f915c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 -- 172.21.15.26:0/342354797 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c14b000 con 0x7f915c0b5ef0 2022-07-20T02:25:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915b7fe700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 0x7f915c0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915affd700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0b5ef0 0x7f915c1474d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f9150002f70 tx=0x7f9150006df0).ready entity=mon.1 client_cookie=f352cbfdb455bd26 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f9159ffb700 1 -- 172.21.15.26:0/342354797 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f915000d4a0 con 0x7f915c0b5ef0 2022-07-20T02:25:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f9159ffb700 1 -- 172.21.15.26:0/342354797 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9150000e90 con 0x7f915c0b5ef0 2022-07-20T02:25:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f9159ffb700 1 -- 172.21.15.26:0/342354797 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f915000d7f0 con 0x7f915c0b5ef0 2022-07-20T02:25:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0b5ef0 msgr2=0x7f915c1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0b5ef0 0x7f915c1474d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f9150002f70 tx=0x7f9150006df0).stop 2022-07-20T02:25:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- 172.21.15.26:0/342354797 shutdown_connections 2022-07-20T02:25:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a8c40 0x7f915c0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 --2- 172.21.15.26:0/342354797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0b5ef0 0x7f915c1474d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- 172.21.15.26:0/342354797 >> 172.21.15.26:0/342354797 conn(0x7f915c060310 msgr2=0x7f915c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- 172.21.15.26:0/342354797 shutdown_connections 2022-07-20T02:25:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.903+0000 7f915bfff700 1 -- 172.21.15.26:0/342354797 wait complete. 2022-07-20T02:25:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 Processor -- start 2022-07-20T02:25:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- start start 2022-07-20T02:25:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a7fc0 0x7f915c149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0a8c40 0x7f915c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 0x7f915c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0ab5a0 con 0x7f915c0b5ef0 2022-07-20T02:25:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0ab840 con 0x7f915c0a8c40 2022-07-20T02:25:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0abae0 con 0x7f915c0a7fc0 2022-07-20T02:25:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 0x7f915c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 0x7f915c156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35156/0 (socket says 172.21.15.26:35156) 2022-07-20T02:25:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 -- 172.21.15.26:0/4110656477 learned_addr learned my addr 172.21.15.26:0/4110656477 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915a7fc700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0a8c40 0x7f915c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915affd700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a7fc0 0x7f915c149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 -- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a7fc0 msgr2=0x7f915c149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f915c0a7fc0 0x7f915c149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 -- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0a8c40 msgr2=0x7f915c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0a8c40 0x7f915c1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915400b180 con 0x7f915c0b5ef0 2022-07-20T02:25:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915a7fc700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f915c0a8c40 0x7f915c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915b7fe700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 0x7f915c156da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f915400c000 tx=0x7f9154006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9154010780 con 0x7f915c0b5ef0 2022-07-20T02:25:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91540108e0 con 0x7f915c0b5ef0 2022-07-20T02:25:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915000b180 con 0x7f915c0b5ef0 2022-07-20T02:25:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9154010c40 con 0x7f915c0b5ef0 2022-07-20T02:25:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.907+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c157290 con 0x7f915c0b5ef0 2022-07-20T02:25:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f915c0ab420 con 0x7f915c0b5ef0 2022-07-20T02:25:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f915401c460 con 0x7f915c0b5ef0 2022-07-20T02:25:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9138053a70 0x7f9138055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f915406c580 con 0x7f915c0b5ef0 2022-07-20T02:25:41.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f915affd700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9138053a70 0x7f9138055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f913805b3f0 0x7f913805d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:41.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f913805de70 con 0x7f913805b3f0 2022-07-20T02:25:41.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f915406ec10 con 0x7f915c0b5ef0 2022-07-20T02:25:41.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f915a7fc700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f913805b3f0 0x7f913805d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:41.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f915affd700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9138053a70 0x7f9138055f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f915001a000 tx=0x7f9150014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.911+0000 7f915a7fc700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f913805b3f0 0x7f913805d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:41.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.915+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== osd.4 v2:172.21.15.120:6816/23448 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f913805de70 con 0x7f913805b3f0 2022-07-20T02:25:41.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.931+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 --> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f915c156610 con 0x7f913805b3f0 2022-07-20T02:25:41.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f914bfff700 1 -- 172.21.15.26:0/4110656477 <== osd.4 v2:172.21.15.120:6816/23448 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f914c001fc0 con 0x7f913805b3f0 2022-07-20T02:25:41.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f913805b3f0 msgr2=0x7f913805d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f913805b3f0 0x7f913805d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:41.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9138053a70 msgr2=0x7f9138055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9138053a70 0x7f9138055f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f915001a000 tx=0x7f9150014000).stop 2022-07-20T02:25:41.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 msgr2=0x7f915c156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:41.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 --2- 172.21.15.26:0/4110656477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f915c0b5ef0 0x7f915c156da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f915400c000 tx=0x7f9154006df0).stop 2022-07-20T02:25:41.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915b7fe700 1 -- 172.21.15.26:0/4110656477 reap_dead start 2022-07-20T02:25:41.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 shutdown_connections 2022-07-20T02:25:41.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 >> 172.21.15.26:0/4110656477 conn(0x7f915c060310 msgr2=0x7f915c148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:41.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 shutdown_connections 2022-07-20T02:25:41.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:41.935+0000 7f915bfff700 1 -- 172.21.15.26:0/4110656477 wait complete. 2022-07-20T02:25:41.968 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:41.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-07-20T02:25:42.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.087+0000 7f3dcf896700 1 Processor -- start 2022-07-20T02:25:42.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.087+0000 7f3dcf896700 1 -- start start 2022-07-20T02:25:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3dc80b2920 0x7f3dc8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80b1bc0 con 0x7f3dc8061050 2022-07-20T02:25:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc8147700 con 0x7f3dc8067190 2022-07-20T02:25:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc814b040 con 0x7f3dc80b2920 2022-07-20T02:25:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcce31700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3dc8061050 0x7f3dc8066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34442/0 (socket says 172.21.15.26:34442) 2022-07-20T02:25:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcce31700 1 -- 172.21.15.26:0/4236437529 learned_addr learned my addr 172.21.15.26:0/4236437529 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcd632700 1 --2- 172.21.15.26:0/4236437529 >> v2:172.21.15.120:3300/0 conn(0x7f3dc80b2920 0x7f3dc8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962829445 0 0) 0x7f3dc80b1bc0 con 0x7f3dc8061050 2022-07-20T02:25:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac003260 con 0x7f3dc8061050 2022-07-20T02:25:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453849192 0 0) 0x7f3dc8147700 con 0x7f3dc8067190 2022-07-20T02:25:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcd632700 1 -- 172.21.15.26:0/4236437529 >> v1:172.21.15.120:6789/0 conn(0x7f3dc8067190 legacy=0x7f3dc80ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcd632700 1 -- 172.21.15.26:0/4236437529 >> v1:172.21.15.26:6789/0 conn(0x7f3dc8061050 legacy=0x7f3dc8066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcd632700 1 -- 172.21.15.26:0/4236437529 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8147930 con 0x7f3dc80b2920 2022-07-20T02:25:42.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcd632700 1 --2- 172.21.15.26:0/4236437529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80b2920 0x7f3dc8147d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3db8002f70 tx=0x7f3db8006df0).ready entity=mon.1 client_cookie=92d0d858cf1676de server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db800d610 con 0x7f3dc80b2920 2022-07-20T02:25:42.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8000e90 con 0x7f3dc80b2920 2022-07-20T02:25:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dbffff700 1 -- 172.21.15.26:0/4236437529 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db800d960 con 0x7f3dc80b2920 2022-07-20T02:25:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- 172.21.15.26:0/4236437529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80b2920 msgr2=0x7f3dc8147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 --2- 172.21.15.26:0/4236437529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80b2920 0x7f3dc8147d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3db8002f70 tx=0x7f3db8006df0).stop 2022-07-20T02:25:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- 172.21.15.26:0/4236437529 shutdown_connections 2022-07-20T02:25:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 --2- 172.21.15.26:0/4236437529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc80b2920 0x7f3dc8147d90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- 172.21.15.26:0/4236437529 >> 172.21.15.26:0/4236437529 conn(0x7f3dc80b60a0 msgr2=0x7f3dc80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- 172.21.15.26:0/4236437529 shutdown_connections 2022-07-20T02:25:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.091+0000 7f3dcf896700 1 -- 172.21.15.26:0/4236437529 wait complete. 2022-07-20T02:25:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.095+0000 7f3dcf896700 1 Processor -- start 2022-07-20T02:25:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.095+0000 7f3dcf896700 1 -- start start 2022-07-20T02:25:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc8061050 0x7f3dc81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc8067190 0x7f3dc8065110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 0x7f3dc8146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8065a70 con 0x7f3dc80b2920 2022-07-20T02:25:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8064aa0 con 0x7f3dc8061050 2022-07-20T02:25:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8064560 con 0x7f3dc8067190 2022-07-20T02:25:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 0x7f3dc8146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 0x7f3dc8146c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35172/0 (socket says 172.21.15.26:35172) 2022-07-20T02:25:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 -- 172.21.15.26:0/3800913270 learned_addr learned my addr 172.21.15.26:0/3800913270 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcd632700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc8061050 0x7f3dc81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcce31700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc8067190 0x7f3dc8065110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 -- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc8067190 msgr2=0x7f3dc8065110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3dc8067190 0x7f3dc8065110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 -- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc8061050 msgr2=0x7f3dc81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc8061050 0x7f3dc81566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db800b180 con 0x7f3dc80b2920 2022-07-20T02:25:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcd632700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3dc8061050 0x7f3dc81566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcde33700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 0x7f3dc8146c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3dc0000bc0 tx=0x7f3dc0007b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc0012400 con 0x7f3dc80b2920 2022-07-20T02:25:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc0012560 con 0x7f3dc80b2920 2022-07-20T02:25:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc8064800 con 0x7f3dc80b2920 2022-07-20T02:25:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc00128c0 con 0x7f3dc80b2920 2022-07-20T02:25:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc80a9660 con 0x7f3dc80b2920 2022-07-20T02:25:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.099+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3dc8065c30 con 0x7f3dc80b2920 2022-07-20T02:25:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3dc0019030 con 0x7f3dc80b2920 2022-07-20T02:25:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3db0053a70 0x7f3db0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3dc006bbd0 con 0x7f3dc80b2920 2022-07-20T02:25:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dcd632700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3db0053a70 0x7f3db0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7f3db005b370 0x7f3db005d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3db005ddf0 con 0x7f3db005b370 2022-07-20T02:25:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dc006bea0 con 0x7f3dc80b2920 2022-07-20T02:25:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dcd632700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3db0053a70 0x7f3db0055f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3db8018000 tx=0x7f3db8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dcce31700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7f3db005b370 0x7f3db005d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.103+0000 7f3dcce31700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7f3db005b370 0x7f3db005d810 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.107+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== osd.5 v2:172.21.15.120:6800/23447 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f3db005ddf0 con 0x7f3db005b370 2022-07-20T02:25:42.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.123+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 --> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3dc8066be0 con 0x7f3db005b370 2022-07-20T02:25:42.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.123+0000 7f3dbe7fc700 1 -- 172.21.15.26:0/3800913270 <== osd.5 v2:172.21.15.120:6800/23447 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3dc8066be0 con 0x7f3db005b370 2022-07-20T02:25:42.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7f3db005b370 msgr2=0x7f3db005d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7f3db005b370 0x7f3db005d810 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3db0053a70 msgr2=0x7f3db0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3db0053a70 0x7f3db0055f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3db8018000 tx=0x7f3db8014000).stop 2022-07-20T02:25:42.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 msgr2=0x7f3dc8146c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 --2- 172.21.15.26:0/3800913270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc80b2920 0x7f3dc8146c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3dc0000bc0 tx=0x7f3dc0007b70).stop 2022-07-20T02:25:42.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcde33700 1 -- 172.21.15.26:0/3800913270 reap_dead start 2022-07-20T02:25:42.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 shutdown_connections 2022-07-20T02:25:42.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 >> 172.21.15.26:0/3800913270 conn(0x7f3dc80b60a0 msgr2=0x7f3dc80b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 shutdown_connections 2022-07-20T02:25:42.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.127+0000 7f3dcf896700 1 -- 172.21.15.26:0/3800913270 wait complete. 2022-07-20T02:25:42.150 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:42.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-07-20T02:25:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.259+0000 7f1940968700 1 Processor -- start 2022-07-20T02:25:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.259+0000 7f1940968700 1 -- start start 2022-07-20T02:25:42.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 0x7f193c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 0x7f193c1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c0b2e40 con 0x7f193c0b7d80 2022-07-20T02:25:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c0b4c20 con 0x7f193c064220 2022-07-20T02:25:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0617a0 con 0x7f193c0b3060 2022-07-20T02:25:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 0x7f193c1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 0x7f193c1434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35182/0 (socket says 172.21.15.26:35182) 2022-07-20T02:25:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193b59e700 1 -- 172.21.15.26:0/2836563510 learned_addr learned my addr 172.21.15.26:0/2836563510 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 --2- 172.21.15.26:0/2836563510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 0x7f193c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1939d9b700 1 -- 172.21.15.26:0/2836563510 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688613644 0 0) 0x7f193c0b2e40 con 0x7f193c0b7d80 2022-07-20T02:25:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1939d9b700 1 -- 172.21.15.26:0/2836563510 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c003260 con 0x7f193c0b7d80 2022-07-20T02:25:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 -- 172.21.15.26:0/2836563510 >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 msgr2=0x7f193c1434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 --2- 172.21.15.26:0/2836563510 >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 0x7f193c1434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 -- 172.21.15.26:0/2836563510 >> v1:172.21.15.120:6789/0 conn(0x7f193c0b7d80 legacy=0x7f193c0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 -- 172.21.15.26:0/2836563510 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c060b20 con 0x7f193c0b3060 2022-07-20T02:25:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f193a59c700 1 --2- 172.21.15.26:0/2836563510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 0x7f193c0b4730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1924002f70 tx=0x7f1924006df0).ready entity=mon.2 client_cookie=345c8c7f7ea6945d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1939d9b700 1 -- 172.21.15.26:0/2836563510 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f192400d4a0 con 0x7f193c0b3060 2022-07-20T02:25:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1939d9b700 1 -- 172.21.15.26:0/2836563510 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1924000e90 con 0x7f193c0b3060 2022-07-20T02:25:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1939d9b700 1 -- 172.21.15.26:0/2836563510 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f192400d7f0 con 0x7f193c0b3060 2022-07-20T02:25:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- 172.21.15.26:0/2836563510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 msgr2=0x7f193c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 --2- 172.21.15.26:0/2836563510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 0x7f193c0b4730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1924002f70 tx=0x7f1924006df0).stop 2022-07-20T02:25:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- 172.21.15.26:0/2836563510 shutdown_connections 2022-07-20T02:25:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 --2- 172.21.15.26:0/2836563510 >> v2:172.21.15.26:3300/0 conn(0x7f193c064220 0x7f193c1434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 --2- 172.21.15.26:0/2836563510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b3060 0x7f193c0b4730 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- 172.21.15.26:0/2836563510 >> 172.21.15.26:0/2836563510 conn(0x7f193c05f840 msgr2=0x7f193c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- 172.21.15.26:0/2836563510 shutdown_connections 2022-07-20T02:25:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.263+0000 7f1940968700 1 -- 172.21.15.26:0/2836563510 wait complete. 2022-07-20T02:25:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 Processor -- start 2022-07-20T02:25:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 -- start start 2022-07-20T02:25:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f193c064220 0x7f193c0a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 0x7f193c0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b7d80 0x7f193c152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0b07a0 con 0x7f193c0b3060 2022-07-20T02:25:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0b1590 con 0x7f193c064220 2022-07-20T02:25:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0a40c0 con 0x7f193c0b7d80 2022-07-20T02:25:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 0x7f193c0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 0x7f193c0a5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35186/0 (socket says 172.21.15.26:35186) 2022-07-20T02:25:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 -- 172.21.15.26:0/3459568468 learned_addr learned my addr 172.21.15.26:0/3459568468 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193b59e700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b7d80 0x7f193c152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193ad9d700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f193c064220 0x7f193c0a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 -- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b7d80 msgr2=0x7f193c152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b7d80 0x7f193c152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 -- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f193c064220 msgr2=0x7f193c0a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f193c064220 0x7f193c0a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192400b180 con 0x7f193c0b3060 2022-07-20T02:25:42.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193b59e700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f193c0b7d80 0x7f193c152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:42.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f193a59c700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 0x7f193c0a5880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1924002f70 tx=0x7f1924014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1924013ac0 con 0x7f193c0b3060 2022-07-20T02:25:42.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1924013c20 con 0x7f193c0b3060 2022-07-20T02:25:42.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c0b12f0 con 0x7f193c0b3060 2022-07-20T02:25:42.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.267+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f192401e400 con 0x7f193c0b3060 2022-07-20T02:25:42.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c0b0c00 con 0x7f193c0b3060 2022-07-20T02:25:42.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f193c0a60b0 con 0x7f193c0b3060 2022-07-20T02:25:42.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f192401c070 con 0x7f193c0b3060 2022-07-20T02:25:42.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1920053a70 0x7f1920055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f192406ccb0 con 0x7f193c0b3060 2022-07-20T02:25:42.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f193ad9d700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1920053a70 0x7f1920055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f192005b370 0x7f192005d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f192005ddf0 con 0x7f192005b370 2022-07-20T02:25:42.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f192406f2c0 con 0x7f193c0b3060 2022-07-20T02:25:42.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f193b59e700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f192005b370 0x7f192005d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f193ad9d700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1920053a70 0x7f1920055f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f192c00b000 tx=0x7f192c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.271+0000 7f193b59e700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f192005b370 0x7f192005d810 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.275+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== osd.6 v2:172.21.15.120:6808/23449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f192005ddf0 con 0x7f192005b370 2022-07-20T02:25:42.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 --> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f193c152830 con 0x7f192005b370 2022-07-20T02:25:42.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1933fff700 1 -- 172.21.15.26:0/3459568468 <== osd.6 v2:172.21.15.120:6808/23449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f193c152830 con 0x7f192005b370 2022-07-20T02:25:42.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f192005b370 msgr2=0x7f192005d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f192005b370 0x7f192005d810 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1920053a70 msgr2=0x7f1920055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1920053a70 0x7f1920055f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f192c00b000 tx=0x7f192c009040).stop 2022-07-20T02:25:42.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 msgr2=0x7f193c0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 --2- 172.21.15.26:0/3459568468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f193c0b3060 0x7f193c0a5880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1924002f70 tx=0x7f1924014000).stop 2022-07-20T02:25:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f193b59e700 1 -- 172.21.15.26:0/3459568468 reap_dead start 2022-07-20T02:25:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 shutdown_connections 2022-07-20T02:25:42.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 >> 172.21.15.26:0/3459568468 conn(0x7f193c05f840 msgr2=0x7f193c064ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 shutdown_connections 2022-07-20T02:25:42.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.295+0000 7f1940968700 1 -- 172.21.15.26:0/3459568468 wait complete. 2022-07-20T02:25:42.333 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:42.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-07-20T02:25:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 Processor -- start 2022-07-20T02:25:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 -- start start 2022-07-20T02:25:42.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09481430d0 0x7f09480b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09480b6230 0x7f094813f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b1060 0x7f094813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094813f270 con 0x7f09481430d0 2022-07-20T02:25:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094813cfb0 con 0x7f09480b6230 2022-07-20T02:25:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094eb8a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480b1570 con 0x7f09480b1060 2022-07-20T02:25:42.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09481430d0 0x7f09480b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09481430d0 0x7f09480b5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35194/0 (socket says 172.21.15.26:35194) 2022-07-20T02:25:42.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 -- 172.21.15.26:0/2449873031 learned_addr learned my addr 172.21.15.26:0/2449873031 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094d127700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b1060 0x7f094813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f0947fff700 1 --2- 172.21.15.26:0/2449873031 >> v2:172.21.15.120:3300/0 conn(0x7f09480b6230 0x7f094813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 -- 172.21.15.26:0/2449873031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b1060 msgr2=0x7f094813fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:42.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b1060 0x7f094813fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 -- 172.21.15.26:0/2449873031 >> v2:172.21.15.120:3300/0 conn(0x7f09480b6230 msgr2=0x7f094813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:42.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 --2- 172.21.15.26:0/2449873031 >> v2:172.21.15.120:3300/0 conn(0x7f09480b6230 0x7f094813f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 -- 172.21.15.26:0/2449873031 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480b1810 con 0x7f09481430d0 2022-07-20T02:25:42.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f094c926700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09481430d0 0x7f09480b5d40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f09480b3500 tx=0x7f0938006eb0).ready entity=mon.0 client_cookie=77069702c7313a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f09477fe700 1 -- 172.21.15.26:0/2449873031 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f093800e610 con 0x7f09481430d0 2022-07-20T02:25:42.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f09477fe700 1 -- 172.21.15.26:0/2449873031 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0938000e90 con 0x7f09481430d0 2022-07-20T02:25:42.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.447+0000 7f09477fe700 1 -- 172.21.15.26:0/2449873031 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f093800e9a0 con 0x7f09481430d0 2022-07-20T02:25:42.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- 172.21.15.26:0/2449873031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09481430d0 msgr2=0x7f09480b5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09481430d0 0x7f09480b5d40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f09480b3500 tx=0x7f0938006eb0).stop 2022-07-20T02:25:42.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- 172.21.15.26:0/2449873031 shutdown_connections 2022-07-20T02:25:42.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b1060 0x7f094813fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- 172.21.15.26:0/2449873031 >> v2:172.21.15.120:3300/0 conn(0x7f09480b6230 0x7f094813f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- 172.21.15.26:0/2449873031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09481430d0 0x7f09480b5d40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- 172.21.15.26:0/2449873031 >> 172.21.15.26:0/2449873031 conn(0x7f094805c580 msgr2=0x7f094805c960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- 172.21.15.26:0/2449873031 shutdown_connections 2022-07-20T02:25:42.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- 172.21.15.26:0/2449873031 wait complete. 2022-07-20T02:25:42.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 Processor -- start 2022-07-20T02:25:42.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- start start 2022-07-20T02:25:42.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09480b1060 0x7f0948132090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 0x7f09480ab240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f094814a7d0 0x7f09480ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094813eae0 con 0x7f094813e5d0 2022-07-20T02:25:42.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094813ed80 con 0x7f09480b1060 2022-07-20T02:25:42.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.451+0000 7f094eb8a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094813f020 con 0x7f094814a7d0 2022-07-20T02:25:42.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 0x7f09480ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 0x7f09480ab240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35202/0 (socket says 172.21.15.26:35202) 2022-07-20T02:25:42.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 -- 172.21.15.26:0/1318595941 learned_addr learned my addr 172.21.15.26:0/1318595941 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f094c926700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09480b1060 0x7f0948132090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f094d127700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f094814a7d0 0x7f09480ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 -- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f094814a7d0 msgr2=0x7f09480ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f094814a7d0 0x7f09480ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 -- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09480b1060 msgr2=0x7f0948132090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09480b1060 0x7f0948132090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093800b1e0 con 0x7f094813e5d0 2022-07-20T02:25:42.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0947fff700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 0x7f09480ab240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f093c00c000 tx=0x7f093c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f093c011750 con 0x7f094813e5d0 2022-07-20T02:25:42.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c0118b0 con 0x7f094813e5d0 2022-07-20T02:25:42.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09480aada0 con 0x7f094813e5d0 2022-07-20T02:25:42.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f093c011c10 con 0x7f094813e5d0 2022-07-20T02:25:42.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09480abc20 con 0x7f094813e5d0 2022-07-20T02:25:42.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09480abf30 con 0x7f094813e5d0 2022-07-20T02:25:42.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.455+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f093c011d70 con 0x7f094813e5d0 2022-07-20T02:25:42.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09280539e0 0x7f0928055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f093c06c310 con 0x7f094813e5d0 2022-07-20T02:25:42.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f094c926700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09280539e0 0x7f0928055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f092805b360 0x7f092805d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f092805de80 con 0x7f092805b360 2022-07-20T02:25:42.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f093c06e9a0 con 0x7f094813e5d0 2022-07-20T02:25:42.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f094d127700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f092805b360 0x7f092805d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f094c926700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09280539e0 0x7f0928055ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f093801b000 tx=0x7f0938015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f094d127700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f092805b360 0x7f092805d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.459+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== osd.7 v2:172.21.15.120:6824/23460 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f092805de80 con 0x7f092805b360 2022-07-20T02:25:42.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.479+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 --> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09480aaf30 con 0x7f092805b360 2022-07-20T02:25:42.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.479+0000 7f0945ffb700 1 -- 172.21.15.26:0/1318595941 <== osd.7 v2:172.21.15.120:6824/23460 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09480aaf30 con 0x7f092805b360 2022-07-20T02:25:42.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f092805b360 msgr2=0x7f092805d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f092805b360 0x7f092805d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09280539e0 msgr2=0x7f0928055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09280539e0 0x7f0928055ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f093801b000 tx=0x7f0938015000).stop 2022-07-20T02:25:42.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 msgr2=0x7f09480ab240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 --2- 172.21.15.26:0/1318595941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f094813e5d0 0x7f09480ab240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f093c00c000 tx=0x7f093c006df0).stop 2022-07-20T02:25:42.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094d127700 1 -- 172.21.15.26:0/1318595941 reap_dead start 2022-07-20T02:25:42.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 shutdown_connections 2022-07-20T02:25:42.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 >> 172.21.15.26:0/1318595941 conn(0x7f094805c580 msgr2=0x7f0948136db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 shutdown_connections 2022-07-20T02:25:42.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.483+0000 7f094eb8a700 1 -- 172.21.15.26:0/1318595941 wait complete. 2022-07-20T02:25:42.526 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:42.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-07-20T02:25:42.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 Processor -- start 2022-07-20T02:25:42.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 -- start start 2022-07-20T02:25:42.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29ec143070 0x7f29ec1434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec146750 con 0x7f29ec140320 2022-07-20T02:25:42.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec1469f0 con 0x7f29ec0b5d50 2022-07-20T02:25:42.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f3539700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ec067ba0 con 0x7f29ec143070 2022-07-20T02:25:42.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f2d38700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29ec140320 0x7f29ec1407d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34494/0 (socket says 172.21.15.26:34494) 2022-07-20T02:25:42.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f2d38700 1 -- 172.21.15.26:0/2445021813 learned_addr learned my addr 172.21.15.26:0/2445021813 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:42.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.655+0000 7f29f1d36700 1 --2- 172.21.15.26:0/2445021813 >> v2:172.21.15.120:3300/0 conn(0x7f29ec143070 0x7f29ec1434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1d36700 1 -- 172.21.15.26:0/2445021813 >> v1:172.21.15.120:6789/0 conn(0x7f29ec0b5d50 legacy=0x7f29ec067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1d36700 1 -- 172.21.15.26:0/2445021813 >> v1:172.21.15.26:6789/0 conn(0x7f29ec140320 legacy=0x7f29ec1407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1d36700 1 -- 172.21.15.26:0/2445021813 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec146c90 con 0x7f29ec143070 2022-07-20T02:25:42.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1d36700 1 --2- 172.21.15.26:0/2445021813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec143070 0x7f29ec1434c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f29ec066d30 tx=0x7f29e8006df0).ready entity=mon.1 client_cookie=1d8b99c6fa0bea3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1535700 1 -- 172.21.15.26:0/2445021813 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e800d610 con 0x7f29ec143070 2022-07-20T02:25:42.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1535700 1 -- 172.21.15.26:0/2445021813 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8000e90 con 0x7f29ec143070 2022-07-20T02:25:42.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f1535700 1 -- 172.21.15.26:0/2445021813 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e800d960 con 0x7f29ec143070 2022-07-20T02:25:42.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- 172.21.15.26:0/2445021813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec143070 msgr2=0x7f29ec1434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 --2- 172.21.15.26:0/2445021813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec143070 0x7f29ec1434c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f29ec066d30 tx=0x7f29e8006df0).stop 2022-07-20T02:25:42.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- 172.21.15.26:0/2445021813 shutdown_connections 2022-07-20T02:25:42.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 --2- 172.21.15.26:0/2445021813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec143070 0x7f29ec1434c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- 172.21.15.26:0/2445021813 >> 172.21.15.26:0/2445021813 conn(0x7f29ec060310 msgr2=0x7f29ec0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- 172.21.15.26:0/2445021813 shutdown_connections 2022-07-20T02:25:42.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- 172.21.15.26:0/2445021813 wait complete. 2022-07-20T02:25:42.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 Processor -- start 2022-07-20T02:25:42.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.659+0000 7f29f3539700 1 -- start start 2022-07-20T02:25:42.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 0x7f29ec0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 0x7f29ec13b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 0x7f29ec13b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec155fb0 con 0x7f29ec143070 2022-07-20T02:25:42.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec13bcb0 con 0x7f29ec0b5d50 2022-07-20T02:25:42.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec156ec0 con 0x7f29ec140320 2022-07-20T02:25:42.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 0x7f29ec13b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2537700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 0x7f29ec0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f1d36700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 0x7f29ec13b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2537700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 0x7f29ec0b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:54910/0 (socket says 172.21.15.26:54910) 2022-07-20T02:25:42.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2537700 1 -- 172.21.15.26:0/2737050881 learned_addr learned my addr 172.21.15.26:0/2737050881 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f1d36700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 0x7f29ec13b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:56732/0 (socket says 172.21.15.26:56732) 2022-07-20T02:25:42.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 -- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 msgr2=0x7f29ec13b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 0x7f29ec13b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 -- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 msgr2=0x7f29ec0b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 0x7f29ec0b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 -- 172.21.15.26:0/2737050881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e800b180 con 0x7f29ec143070 2022-07-20T02:25:42.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f2d38700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 0x7f29ec13b7c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f29ec068040 tx=0x7f29e4007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e4005ba0 con 0x7f29ec143070 2022-07-20T02:25:42.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e4005d00 con 0x7f29ec143070 2022-07-20T02:25:42.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e40072c0 con 0x7f29ec143070 2022-07-20T02:25:42.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec0a9ce0 con 0x7f29ec143070 2022-07-20T02:25:42.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec0a9f10 con 0x7f29ec143070 2022-07-20T02:25:42.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.663+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec146f30 con 0x7f29ec143070 2022-07-20T02:25:42.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.667+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29e4010070 con 0x7f29ec143070 2022-07-20T02:25:42.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.667+0000 7f29db7fe700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 0x7f29cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.667+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f29e406b980 con 0x7f29ec143070 2022-07-20T02:25:42.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.667+0000 7f29f2537700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 0x7f29cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.667+0000 7f29f2537700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 0x7f29cc055fc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f29dc00d000 tx=0x7f29dc007c50).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.675+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29e4042630 con 0x7f29ec143070 2022-07-20T02:25:42.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.795+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f29ec0574e0 con 0x7f29ec143070 2022-07-20T02:25:42.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.795+0000 7f29db7fe700 1 -- 172.21.15.26:0/2737050881 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29e4045c50 con 0x7f29ec143070 2022-07-20T02:25:42.801 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 msgr2=0x7f29cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 0x7f29cc055fc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f29dc00d000 tx=0x7f29dc007c50).stop 2022-07-20T02:25:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 msgr2=0x7f29ec13b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 0x7f29ec13b7c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f29ec068040 tx=0x7f29e4007e90).stop 2022-07-20T02:25:42.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 shutdown_connections 2022-07-20T02:25:42.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec143070 0x7f29ec13b7c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053b00 0x7f29cc055fc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec140320 0x7f29ec13b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 --2- 172.21.15.26:0/2737050881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec0b5d50 0x7f29ec0b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 >> 172.21.15.26:0/2737050881 conn(0x7f29ec060310 msgr2=0x7f29ec1407d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 shutdown_connections 2022-07-20T02:25:42.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.799+0000 7f29f3539700 1 -- 172.21.15.26:0/2737050881 wait complete. 2022-07-20T02:25:42.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2022-07-20T02:25:42.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-07-20T02:25:42.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.923+0000 7f5894416700 1 Processor -- start 2022-07-20T02:25:42.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- start start 2022-07-20T02:25:42.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f588c05f9e0 0x7f588c05fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 0x7f588c1436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c094fc0 con 0x7f588c061750 2022-07-20T02:25:42.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c0602b0 con 0x7f588c05f9e0 2022-07-20T02:25:42.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0b2120 con 0x7f588c0b0f90 2022-07-20T02:25:42.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58919b1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f588c061750 0x7f588c0b0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34508/0 (socket says 172.21.15.26:34508) 2022-07-20T02:25:42.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58919b1700 1 -- 172.21.15.26:0/1180592964 learned_addr learned my addr 172.21.15.26:0/1180592964 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:42.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 --2- 172.21.15.26:0/1180592964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 0x7f588c1436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58921b2700 1 --2- 172.21.15.26:0/1180592964 >> v2:172.21.15.120:3300/0 conn(0x7f588c05f9e0 0x7f588c05fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816624982 0 0) 0x7f588c094fc0 con 0x7f588c061750 2022-07-20T02:25:42.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5874003260 con 0x7f588c061750 2022-07-20T02:25:42.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 -- 172.21.15.26:0/1180592964 >> v2:172.21.15.120:3300/0 conn(0x7f588c05f9e0 msgr2=0x7f588c05fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:42.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 --2- 172.21.15.26:0/1180592964 >> v2:172.21.15.120:3300/0 conn(0x7f588c05f9e0 0x7f588c05fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 -- 172.21.15.26:0/1180592964 >> v1:172.21.15.26:6789/0 conn(0x7f588c061750 legacy=0x7f588c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934531350 0 0) 0x7f5874003260 con 0x7f588c061750 2022-07-20T02:25:42.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 -- 172.21.15.26:0/1180592964 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0b1e80 con 0x7f588c0b0f90 2022-07-20T02:25:42.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58929b3700 1 --2- 172.21.15.26:0/1180592964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 0x7f588c1436a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f587c001820 tx=0x7f58840091d0).ready entity=mon.2 client_cookie=6a9ad2210a495622 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f587c001680 con 0x7f588c0b0f90 2022-07-20T02:25:42.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400f510 con 0x7f588c0b0f90 2022-07-20T02:25:42.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f58911b0700 1 -- 172.21.15.26:0/1180592964 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588400f680 con 0x7f588c0b0f90 2022-07-20T02:25:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- 172.21.15.26:0/1180592964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 msgr2=0x7f588c1436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 --2- 172.21.15.26:0/1180592964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 0x7f588c1436a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f587c001820 tx=0x7f58840091d0).stop 2022-07-20T02:25:42.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- 172.21.15.26:0/1180592964 shutdown_connections 2022-07-20T02:25:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 --2- 172.21.15.26:0/1180592964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c0b0f90 0x7f588c1436a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 --2- 172.21.15.26:0/1180592964 >> v2:172.21.15.120:3300/0 conn(0x7f588c05f9e0 0x7f588c05fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.927+0000 7f5894416700 1 -- 172.21.15.26:0/1180592964 >> 172.21.15.26:0/1180592964 conn(0x7f588c0b5e70 msgr2=0x7f588c0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- 172.21.15.26:0/1180592964 shutdown_connections 2022-07-20T02:25:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- 172.21.15.26:0/1180592964 wait complete. 2022-07-20T02:25:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 Processor -- start 2022-07-20T02:25:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- start start 2022-07-20T02:25:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c05f9e0 0x7f588c155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c061750 0x7f588c148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c09b380 con 0x7f588c0b0f90 2022-07-20T02:25:42.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c067110 con 0x7f588c05f9e0 2022-07-20T02:25:42.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f5894416700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c066d20 con 0x7f588c061750 2022-07-20T02:25:42.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58919b1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c061750 0x7f588c148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35234/0 (socket says 172.21.15.26:35234) 2022-07-20T02:25:42.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 -- 172.21.15.26:0/2455904003 learned_addr learned my addr 172.21.15.26:0/2455904003 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:42.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58921b2700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c05f9e0 0x7f588c155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 -- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c061750 msgr2=0x7f588c148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c061750 0x7f588c148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 -- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c05f9e0 msgr2=0x7f588c155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:42.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c05f9e0 0x7f588c155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:42.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 -- 172.21.15.26:0/2455904003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884009470 con 0x7f588c0b0f90 2022-07-20T02:25:42.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.931+0000 7f58929b3700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f587c001820 tx=0x7f5884016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58840032f0 con 0x7f588c0b0f90 2022-07-20T02:25:42.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5884011c30 con 0x7f588c0b0f90 2022-07-20T02:25:42.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c1494a0 con 0x7f588c0b0f90 2022-07-20T02:25:42.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5884021480 con 0x7f588c0b0f90 2022-07-20T02:25:42.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c09a6e0 con 0x7f588c0b0f90 2022-07-20T02:25:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c066f70 con 0x7f588c0b0f90 2022-07-20T02:25:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f588400f070 con 0x7f588c0b0f90 2022-07-20T02:25:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 0x7f586c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:42.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f58921b2700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 0x7f586c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:42.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.935+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f588406cdf0 con 0x7f588c0b0f90 2022-07-20T02:25:42.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.939+0000 7f58921b2700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 0x7f586c055e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f588800b000 tx=0x7f5888006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:42.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:42.943+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f588c066f70 con 0x7f588c0b0f90 2022-07-20T02:25:43.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.067+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f588c066f70 con 0x7f588c0b0f90 2022-07-20T02:25:43.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.067+0000 7f587b7fe700 1 -- 172.21.15.26:0/2455904003 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f588c066f70 con 0x7f588c0b0f90 2022-07-20T02:25:43.073 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:43.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 msgr2=0x7f586c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 0x7f586c055e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f588800b000 tx=0x7f5888006df0).stop 2022-07-20T02:25:43.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 msgr2=0x7f588c0675b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f587c001820 tx=0x7f5884016000).stop 2022-07-20T02:25:43.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 shutdown_connections 2022-07-20T02:25:43.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f588c0b0f90 0x7f588c0675b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f586c053950 0x7f586c055e10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f588c061750 0x7f588c148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 --2- 172.21.15.26:0/2455904003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c05f9e0 0x7f588c155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 >> 172.21.15.26:0/2455904003 conn(0x7f588c0b5e70 msgr2=0x7f588c05c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:43.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 shutdown_connections 2022-07-20T02:25:43.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.071+0000 7f5894416700 1 -- 172.21.15.26:0/2455904003 wait complete. 2022-07-20T02:25:43.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2022-07-20T02:25:43.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-07-20T02:25:43.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 Processor -- start 2022-07-20T02:25:43.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 -- start start 2022-07-20T02:25:43.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:43.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0b1bc0 con 0x7f08ec061050 2022-07-20T02:25:43.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec147700 con 0x7f08ec0b2920 2022-07-20T02:25:43.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.199+0000 7f08f3d5b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec14b040 con 0x7f08ec1484a0 2022-07-20T02:25:43.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f22f8700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f08ec061050 0x7f08ec066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34522/0 (socket says 172.21.15.26:34522) 2022-07-20T02:25:43.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f22f8700 1 -- 172.21.15.26:0/2981752329 learned_addr learned my addr 172.21.15.26:0/2981752329 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:43.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f12f6700 1 --2- 172.21.15.26:0/2981752329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:43.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170280006 0 0) 0x7f08ec0b1bc0 con 0x7f08ec061050 2022-07-20T02:25:43.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d4003260 con 0x7f08ec061050 2022-07-20T02:25:43.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071233492 0 0) 0x7f08ec147700 con 0x7f08ec0b2920 2022-07-20T02:25:43.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ec0b1bc0 con 0x7f08ec0b2920 2022-07-20T02:25:43.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f12f6700 1 -- 172.21.15.26:0/2981752329 >> v1:172.21.15.120:6789/0 conn(0x7f08ec0b2920 legacy=0x7f08ec147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f12f6700 1 -- 172.21.15.26:0/2981752329 >> v1:172.21.15.26:6789/0 conn(0x7f08ec061050 legacy=0x7f08ec066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f12f6700 1 -- 172.21.15.26:0/2981752329 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec147930 con 0x7f08ec1484a0 2022-07-20T02:25:43.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f12f6700 1 --2- 172.21.15.26:0/2981752329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec066a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f08e8002f70 tx=0x7f08e8006df0).ready entity=mon.2 client_cookie=896d810442aeafaa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:43.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e800d5c0 con 0x7f08ec1484a0 2022-07-20T02:25:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e8000e90 con 0x7f08ec1484a0 2022-07-20T02:25:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f0af5700 1 -- 172.21.15.26:0/2981752329 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e800d910 con 0x7f08ec1484a0 2022-07-20T02:25:43.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2981752329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 msgr2=0x7f08ec066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2981752329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec066a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f08e8002f70 tx=0x7f08e8006df0).stop 2022-07-20T02:25:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2981752329 shutdown_connections 2022-07-20T02:25:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2981752329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec066a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2981752329 >> 172.21.15.26:0/2981752329 conn(0x7f08ec0b60a0 msgr2=0x7f08ec0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2981752329 shutdown_connections 2022-07-20T02:25:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.203+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2981752329 wait complete. 2022-07-20T02:25:43.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 Processor -- start 2022-07-20T02:25:43.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- start start 2022-07-20T02:25:43.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:43.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2920 0x7f08ec060090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:43.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec0666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:43.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0a9ea0 con 0x7f08ec061050 2022-07-20T02:25:43.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0662b0 con 0x7f08ec0b2920 2022-07-20T02:25:43.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec064b80 con 0x7f08ec1484a0 2022-07-20T02:25:43.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:43.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35244/0 (socket says 172.21.15.26:35244) 2022-07-20T02:25:43.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 -- 172.21.15.26:0/2602194659 learned_addr learned my addr 172.21.15.26:0/2602194659 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:43.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f12f6700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2920 0x7f08ec060090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:43.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f22f8700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec0666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:43.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 -- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 msgr2=0x7f08ec0666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec0666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 -- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2920 msgr2=0x7f08ec060090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2920 0x7f08ec060090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 -- 172.21.15.26:0/2602194659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e800b180 con 0x7f08ec061050 2022-07-20T02:25:43.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f1af7700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f08dc001110 tx=0x7f08dc007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:43.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08dc00ce50 con 0x7f08ec061050 2022-07-20T02:25:43.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc00f470 con 0x7f08ec061050 2022-07-20T02:25:43.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec064e20 con 0x7f08ec061050 2022-07-20T02:25:43.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08dc00f7d0 con 0x7f08ec061050 2022-07-20T02:25:43.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.207+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec0ab3f0 con 0x7f08ec061050 2022-07-20T02:25:43.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec14ada0 con 0x7f08ec061050 2022-07-20T02:25:43.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f08dc019030 con 0x7f08ec061050 2022-07-20T02:25:43.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08daffd700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 0x7f08cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:43.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08f12f6700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 0x7f08cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:43.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f08dc06b500 con 0x7f08ec061050 2022-07-20T02:25:43.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.211+0000 7f08f12f6700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 0x7f08cc055f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f08e801a000 tx=0x7f08e8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:43.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.219+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08dc0422c0 con 0x7f08ec061050 2022-07-20T02:25:43.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.343+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f08ec0574e0 con 0x7f08ec061050 2022-07-20T02:25:43.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08daffd700 1 -- 172.21.15.26:0/2602194659 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08dc0458e0 con 0x7f08ec061050 2022-07-20T02:25:43.349 INFO:teuthology.orchestra.run.smithi026.stdout:55834574852 2022-07-20T02:25:43.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 msgr2=0x7f08cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 0x7f08cc055f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f08e801a000 tx=0x7f08e8014000).stop 2022-07-20T02:25:43.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 msgr2=0x7f08ec156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:43.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f08dc001110 tx=0x7f08dc007b60).stop 2022-07-20T02:25:43.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 shutdown_connections 2022-07-20T02:25:43.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f08cc053aa0 0x7f08cc055f60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08ec1484a0 0x7f08ec0666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec0b2920 0x7f08ec060090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 --2- 172.21.15.26:0/2602194659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08ec061050 0x7f08ec156990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 >> 172.21.15.26:0/2602194659 conn(0x7f08ec0b60a0 msgr2=0x7f08ec146950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 shutdown_connections 2022-07-20T02:25:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:43.347+0000 7f08f3d5b700 1 -- 172.21.15.26:0/2602194659 wait complete. 2022-07-20T02:25:43.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2022-07-20T02:25:44.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-07-20T02:25:44.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.487+0000 7f2c141fd700 1 Processor -- start 2022-07-20T02:25:44.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- start start 2022-07-20T02:25:44.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2c0c0b2920 0x7f2c0c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2c0c061050 0x7f2c0c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0b1bc0 con 0x7f2c0c066a80 2022-07-20T02:25:44.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c14b040 con 0x7f2c0c061050 2022-07-20T02:25:44.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0a8860 con 0x7f2c0c0b2920 2022-07-20T02:25:44.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2c0c061050 0x7f2c0c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2c0c061050 0x7f2c0c148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35252/0 (socket says 172.21.15.26:35252) 2022-07-20T02:25:44.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 -- 172.21.15.26:0/2923905874 learned_addr learned my addr 172.21.15.26:0/2923905874 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:44.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11f99700 1 --2- 172.21.15.26:0/2923905874 >> v2:172.21.15.120:3300/0 conn(0x7f2c0c0b2920 0x7f2c0c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 -- 172.21.15.26:0/2923905874 >> v2:172.21.15.120:3300/0 conn(0x7f2c0c0b2920 msgr2=0x7f2c0c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:44.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 --2- 172.21.15.26:0/2923905874 >> v2:172.21.15.120:3300/0 conn(0x7f2c0c0b2920 0x7f2c0c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 -- 172.21.15.26:0/2923905874 >> v1:172.21.15.120:6789/0 conn(0x7f2c0c066a80 legacy=0x7f2c0c066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 -- 172.21.15.26:0/2923905874 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c147700 con 0x7f2c0c061050 2022-07-20T02:25:44.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c11798700 1 --2- 172.21.15.26:0/2923905874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c148280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2c08002f70 tx=0x7f2c08006eb0).ready entity=mon.0 client_cookie=84e073d8442b5847 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c10f97700 1 -- 172.21.15.26:0/2923905874 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c0800d5c0 con 0x7f2c0c061050 2022-07-20T02:25:44.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c10f97700 1 -- 172.21.15.26:0/2923905874 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08000e90 con 0x7f2c0c061050 2022-07-20T02:25:44.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c10f97700 1 -- 172.21.15.26:0/2923905874 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c0800d950 con 0x7f2c0c061050 2022-07-20T02:25:44.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- 172.21.15.26:0/2923905874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 msgr2=0x7f2c0c148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 --2- 172.21.15.26:0/2923905874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c148280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2c08002f70 tx=0x7f2c08006eb0).stop 2022-07-20T02:25:44.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- 172.21.15.26:0/2923905874 shutdown_connections 2022-07-20T02:25:44.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 --2- 172.21.15.26:0/2923905874 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c148280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 --2- 172.21.15.26:0/2923905874 >> v2:172.21.15.120:3300/0 conn(0x7f2c0c0b2920 0x7f2c0c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.491+0000 7f2c141fd700 1 -- 172.21.15.26:0/2923905874 >> 172.21.15.26:0/2923905874 conn(0x7f2c0c0b60a0 msgr2=0x7f2c0c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:44.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- 172.21.15.26:0/2923905874 shutdown_connections 2022-07-20T02:25:44.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- 172.21.15.26:0/2923905874 wait complete. 2022-07-20T02:25:44.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 Processor -- start 2022-07-20T02:25:44.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- start start 2022-07-20T02:25:44.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c0c066a80 0x7f2c0c15aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c0c0b2920 0x7f2c0c065cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0ab600 con 0x7f2c0c061050 2022-07-20T02:25:44.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0ab8a0 con 0x7f2c0c0b2920 2022-07-20T02:25:44.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0abb40 con 0x7f2c0c066a80 2022-07-20T02:25:44.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35258/0 (socket says 172.21.15.26:35258) 2022-07-20T02:25:44.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 -- 172.21.15.26:0/1031886484 learned_addr learned my addr 172.21.15.26:0/1031886484 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:44.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11798700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c0c066a80 0x7f2c0c15aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c1279a700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c0c0b2920 0x7f2c0c065cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 -- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c0c066a80 msgr2=0x7f2c0c15aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c0c066a80 0x7f2c0c15aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 -- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c0c0b2920 msgr2=0x7f2c0c065cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c0c0b2920 0x7f2c0c065cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 -- 172.21.15.26:0/1031886484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0800b1e0 con 0x7f2c0c061050 2022-07-20T02:25:44.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c11f99700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2bfc00c000 tx=0x7f2bfc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bfc010780 con 0x7f2c0c061050 2022-07-20T02:25:44.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc0108e0 con 0x7f2c0c061050 2022-07-20T02:25:44.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bfc010c40 con 0x7f2c0c061050 2022-07-20T02:25:44.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c0abde0 con 0x7f2c0c061050 2022-07-20T02:25:44.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.495+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c15af70 con 0x7f2c0c061050 2022-07-20T02:25:44.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c05dc30 con 0x7f2c0c061050 2022-07-20T02:25:44.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2bfc01c460 con 0x7f2c0c061050 2022-07-20T02:25:44.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2bfaffd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 0x7f2bec055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2bfc06c440 con 0x7f2c0c061050 2022-07-20T02:25:44.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2c11798700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 0x7f2bec055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.499+0000 7f2c11798700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 0x7f2bec055ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f2c0801c000 tx=0x7f2c08015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.507+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c0c05dc30 con 0x7f2c0c061050 2022-07-20T02:25:44.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2c0c05dc30 con 0x7f2c0c061050 2022-07-20T02:25:44.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2bfaffd700 1 -- 172.21.15.26:0/1031886484 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c0c05dc30 con 0x7f2c0c061050 2022-07-20T02:25:44.646 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:44.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 msgr2=0x7f2bec055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 0x7f2bec055ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f2c0801c000 tx=0x7f2c08015000).stop 2022-07-20T02:25:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 msgr2=0x7f2c0c0af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2bfc00c000 tx=0x7f2bfc006df0).stop 2022-07-20T02:25:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 shutdown_connections 2022-07-20T02:25:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2bec0539e0 0x7f2bec055ea0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c0c0b2920 0x7f2c0c065cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c0c066a80 0x7f2c0c15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 --2- 172.21.15.26:0/1031886484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c0c061050 0x7f2c0c0af380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 >> 172.21.15.26:0/1031886484 conn(0x7f2c0c0b60a0 msgr2=0x7f2c0c14b570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:44.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 shutdown_connections 2022-07-20T02:25:44.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.635+0000 7f2c141fd700 1 -- 172.21.15.26:0/1031886484 wait complete. 2022-07-20T02:25:44.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2022-07-20T02:25:44.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-07-20T02:25:44.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.767+0000 7fd8e5158700 1 Processor -- start 2022-07-20T02:25:44.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.767+0000 7fd8e5158700 1 -- start start 2022-07-20T02:25:44.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8e00b06d0 0x7fd8e013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e00b1090 con 0x7fd8e00a9520 2022-07-20T02:25:44.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00b0bd0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00b12c0 con 0x7fd8e00b06d0 2022-07-20T02:25:44.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8d7fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8e00b06d0 0x7fd8e013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35270/0 (socket says 172.21.15.26:35270) 2022-07-20T02:25:44.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 -- 172.21.15.26:0/804788274 learned_addr learned my addr 172.21.15.26:0/804788274 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:44.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 -- 172.21.15.26:0/804788274 >> v2:172.21.15.120:3300/0 conn(0x7fd8e00b06d0 msgr2=0x7fd8e013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:44.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 --2- 172.21.15.26:0/804788274 >> v2:172.21.15.120:3300/0 conn(0x7fd8e00b06d0 0x7fd8e013f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 -- 172.21.15.26:0/804788274 >> v1:172.21.15.120:6789/0 conn(0x7fd8e00a9520 legacy=0x7fd8e005c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 -- 172.21.15.26:0/804788274 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00b1560 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e4957700 1 --2- 172.21.15.26:0/804788274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd8d80032f0 tx=0x7fd8d8009290).ready entity=mon.0 client_cookie=9a18b92531e61ef3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8df7fe700 1 -- 172.21.15.26:0/804788274 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8d800b870 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8df7fe700 1 -- 172.21.15.26:0/804788274 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d800b9d0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8df7fe700 1 -- 172.21.15.26:0/804788274 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8d800bb40 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- 172.21.15.26:0/804788274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 msgr2=0x7fd8e013fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 --2- 172.21.15.26:0/804788274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd8d80032f0 tx=0x7fd8d8009290).stop 2022-07-20T02:25:44.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- 172.21.15.26:0/804788274 shutdown_connections 2022-07-20T02:25:44.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 --2- 172.21.15.26:0/804788274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e013fbd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 --2- 172.21.15.26:0/804788274 >> v2:172.21.15.120:3300/0 conn(0x7fd8e00b06d0 0x7fd8e013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- 172.21.15.26:0/804788274 >> 172.21.15.26:0/804788274 conn(0x7fd8e0060310 msgr2=0x7fd8e00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:44.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- 172.21.15.26:0/804788274 shutdown_connections 2022-07-20T02:25:44.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.771+0000 7fd8e5158700 1 -- 172.21.15.26:0/804788274 wait complete. 2022-07-20T02:25:44.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 Processor -- start 2022-07-20T02:25:44.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 -- start start 2022-07-20T02:25:44.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8e00a9520 0x7fd8e00b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8e00b06d0 0x7fd8e01500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e0153670 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e01538a0 con 0x7fd8e00a9520 2022-07-20T02:25:44.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.775+0000 7fd8e5158700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e0153b40 con 0x7fd8e00b06d0 2022-07-20T02:25:44.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8dffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8e00a9520 0x7fd8e00b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35274/0 (socket says 172.21.15.26:35274) 2022-07-20T02:25:44.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 -- 172.21.15.26:0/3737641281 learned_addr learned my addr 172.21.15.26:0/3737641281 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:44.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8e4957700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8e00b06d0 0x7fd8e01500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 -- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8e00b06d0 msgr2=0x7fd8e01500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8e00b06d0 0x7fd8e01500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 -- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8e00a9520 msgr2=0x7fd8e00b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8e00a9520 0x7fd8e00b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 -- 172.21.15.26:0/3737641281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d8009530 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8d7fff700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd8c8006a80 tx=0x7fd8c8006e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8c800c4a0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c8010630 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8c8010990 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e0153de0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e00ae1c0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0153f70 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.779+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd8c8021070 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.783+0000 7fd8ddffb700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 0x7fd8c4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:44.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.783+0000 7fd8dffff700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 0x7fd8c4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:44.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.783+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd8c806c1e0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.783+0000 7fd8dffff700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 0x7fd8c4055ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd8d000d000 tx=0x7fd8d0000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:44.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.791+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8c8042e90 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.915+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd8e00574e0 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8ddffb700 1 -- 172.21.15.26:0/3737641281 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd8c8015090 con 0x7fd8e00ac8e0 2022-07-20T02:25:44.925 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:44.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 msgr2=0x7fd8c4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 0x7fd8c4055ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd8d000d000 tx=0x7fd8d0000bc0).stop 2022-07-20T02:25:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 msgr2=0x7fd8e0063010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd8c8006a80 tx=0x7fd8c8006e80).stop 2022-07-20T02:25:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 shutdown_connections 2022-07-20T02:25:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8e00ac8e0 0x7fd8e0063010 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8c4053a10 0x7fd8c4055ed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8e00b06d0 0x7fd8e01500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 --2- 172.21.15.26:0/3737641281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8e00a9520 0x7fd8e00b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 >> 172.21.15.26:0/3737641281 conn(0x7fd8e0060310 msgr2=0x7fd8e00b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 shutdown_connections 2022-07-20T02:25:44.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:44.919+0000 7fd8e5158700 1 -- 172.21.15.26:0/3737641281 wait complete. 2022-07-20T02:25:44.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2022-07-20T02:25:44.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-07-20T02:25:45.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 Processor -- start 2022-07-20T02:25:45.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 -- start start 2022-07-20T02:25:45.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93540b2920 0x7f9354147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9354061050 0x7f9354148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93540b1bc0 con 0x7f9354066a80 2022-07-20T02:25:45.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935414b040 con 0x7f93540b2920 2022-07-20T02:25:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f935ae72700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93540a8860 con 0x7f9354061050 2022-07-20T02:25:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93540b2920 0x7f9354147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9353fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9354061050 0x7f9354148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93540b2920 0x7f9354147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35282/0 (socket says 172.21.15.26:35282) 2022-07-20T02:25:45.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 -- 172.21.15.26:0/3124369391 learned_addr learned my addr 172.21.15.26:0/3124369391 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 -- 172.21.15.26:0/3124369391 >> v2:172.21.15.120:3300/0 conn(0x7f9354061050 msgr2=0x7f9354148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:45.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 --2- 172.21.15.26:0/3124369391 >> v2:172.21.15.120:3300/0 conn(0x7f9354061050 0x7f9354148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 -- 172.21.15.26:0/3124369391 >> v1:172.21.15.120:6789/0 conn(0x7f9354066a80 legacy=0x7f9354066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 -- 172.21.15.26:0/3124369391 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9354147700 con 0x7f93540b2920 2022-07-20T02:25:45.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f93537fe700 1 -- 172.21.15.26:0/3124369391 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727226455 0 0) 0x7f93540b1bc0 con 0x7f9354066a80 2022-07-20T02:25:45.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.047+0000 7f9358c0e700 1 --2- 172.21.15.26:0/3124369391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354147d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9348002f70 tx=0x7f9348006eb0).ready entity=mon.0 client_cookie=fb28d9cf0147b699 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f93537fe700 1 -- 172.21.15.26:0/3124369391 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f934800d5c0 con 0x7f93540b2920 2022-07-20T02:25:45.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f93537fe700 1 -- 172.21.15.26:0/3124369391 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9348000e90 con 0x7f93540b2920 2022-07-20T02:25:45.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f93537fe700 1 -- 172.21.15.26:0/3124369391 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f934800d910 con 0x7f93540b2920 2022-07-20T02:25:45.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- 172.21.15.26:0/3124369391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 msgr2=0x7f9354147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- 172.21.15.26:0/3124369391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354147d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9348002f70 tx=0x7f9348006eb0).stop 2022-07-20T02:25:45.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- 172.21.15.26:0/3124369391 shutdown_connections 2022-07-20T02:25:45.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- 172.21.15.26:0/3124369391 >> v2:172.21.15.120:3300/0 conn(0x7f9354061050 0x7f9354148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- 172.21.15.26:0/3124369391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354147d90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- 172.21.15.26:0/3124369391 >> 172.21.15.26:0/3124369391 conn(0x7f93540b60a0 msgr2=0x7f93540b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- 172.21.15.26:0/3124369391 shutdown_connections 2022-07-20T02:25:45.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- 172.21.15.26:0/3124369391 wait complete. 2022-07-20T02:25:45.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 Processor -- start 2022-07-20T02:25:45.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- start start 2022-07-20T02:25:45.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9354061050 0x7f93540649c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9354066140 con 0x7f93540b2920 2022-07-20T02:25:45.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93540656c0 con 0x7f9354066a80 2022-07-20T02:25:45.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935ae72700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93540b3f60 con 0x7f9354061050 2022-07-20T02:25:45.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.051+0000 7f935940f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:54978/0 (socket says 172.21.15.26:54978) 2022-07-20T02:25:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 -- 172.21.15.26:0/1073548237 learned_addr learned my addr 172.21.15.26:0/1073548237 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9358c0e700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9354061050 0x7f93540649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 -- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9354061050 msgr2=0x7f93540649c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9354061050 0x7f93540649c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 -- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 msgr2=0x7f9354157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 -- 172.21.15.26:0/1073548237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934800b1e0 con 0x7f9354066a80 2022-07-20T02:25:45.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9353fff700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f934400c000 tx=0x7f9344006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9344011750 con 0x7f9354066a80 2022-07-20T02:25:45.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93440118b0 con 0x7f9354066a80 2022-07-20T02:25:45.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93540b4200 con 0x7f9354066a80 2022-07-20T02:25:45.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9344011c10 con 0x7f9354066a80 2022-07-20T02:25:45.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9354156a10 con 0x7f9354066a80 2022-07-20T02:25:45.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935414ada0 con 0x7f9354066a80 2022-07-20T02:25:45.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9344011d70 con 0x7f9354066a80 2022-07-20T02:25:45.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.055+0000 7f9351ffb700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 0x7f9334055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.059+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f934406c520 con 0x7f9354066a80 2022-07-20T02:25:45.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.059+0000 7f9358c0e700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 0x7f9334055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.059+0000 7f9358c0e700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 0x7f9334055fc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f934801c000 tx=0x7f9348015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.067+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9344043240 con 0x7f9354066a80 2022-07-20T02:25:45.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.191+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f93540574e0 con 0x7f9354066a80 2022-07-20T02:25:45.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.195+0000 7f9351ffb700 1 -- 172.21.15.26:0/1073548237 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9344046860 con 0x7f9354066a80 2022-07-20T02:25:45.200 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:45.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.195+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 msgr2=0x7f9334055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 0x7f9334055fc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f934801c000 tx=0x7f9348015000).stop 2022-07-20T02:25:45.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 msgr2=0x7f9354156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f934400c000 tx=0x7f9344006df0).stop 2022-07-20T02:25:45.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 shutdown_connections 2022-07-20T02:25:45.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93540b2920 0x7f9354157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9334053b00 0x7f9334055fc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354066a80 0x7f9354156e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 --2- 172.21.15.26:0/1073548237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9354061050 0x7f93540649c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 >> 172.21.15.26:0/1073548237 conn(0x7f93540b60a0 msgr2=0x7f935414a450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 shutdown_connections 2022-07-20T02:25:45.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.199+0000 7f935ae72700 1 -- 172.21.15.26:0/1073548237 wait complete. 2022-07-20T02:25:45.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2022-07-20T02:25:45.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-07-20T02:25:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 Processor -- start 2022-07-20T02:25:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 -- start start 2022-07-20T02:25:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efecc0621f0 0x7efecc067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc0b0f50 con 0x7efecc060250 2022-07-20T02:25:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc05f8e0 con 0x7efecc0aff40 2022-07-20T02:25:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecb59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc060dd0 con 0x7efecc0621f0 2022-07-20T02:25:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efecc060250 0x7efecc0606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34578/0 (socket says 172.21.15.26:34578) 2022-07-20T02:25:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efecad9d700 1 -- 172.21.15.26:0/1282573291 learned_addr learned my addr 172.21.15.26:0/1282573291 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efec9d9b700 1 --2- 172.21.15.26:0/1282573291 >> v2:172.21.15.120:3300/0 conn(0x7efecc0621f0 0x7efecc067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380678823 0 0) 0x7efecc0b0f50 con 0x7efecc060250 2022-07-20T02:25:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.315+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac003260 con 0x7efecc060250 2022-07-20T02:25:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133199025 0 0) 0x7efeac003260 con 0x7efecc060250 2022-07-20T02:25:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 >> v2:172.21.15.120:3300/0 conn(0x7efecc0621f0 msgr2=0x7efecc067240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 --2- 172.21.15.26:0/1282573291 >> v2:172.21.15.120:3300/0 conn(0x7efecc0621f0 0x7efecc067240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 >> v1:172.21.15.120:6789/0 conn(0x7efecc0aff40 legacy=0x7efecc066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc13e7f0 con 0x7efecc060250 2022-07-20T02:25:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7efec40019e0 con 0x7efecc060250 2022-07-20T02:25:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 >> v1:172.21.15.26:6789/0 conn(0x7efecc060250 legacy=0x7efecc0606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efeac003750 0x7efeac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efeac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeac004e70 0x7efeac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc060dd0 con 0x7efeac003750 2022-07-20T02:25:45.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac00bd20 con 0x7efeac004500 2022-07-20T02:25:45.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac00bee0 con 0x7efeac004e70 2022-07-20T02:25:45.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efeca59c700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efeac003750 0x7efeac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efeac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecad9d700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeac004e70 0x7efeac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 -- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeac004e70 msgr2=0x7efeac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:45.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeac004e70 0x7efeac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 -- 172.21.15.26:0/1282573291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efeac003750 msgr2=0x7efeac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:45.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efeac003750 0x7efeac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 -- 172.21.15.26:0/1282573291 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac003260 con 0x7efeac004500 2022-07-20T02:25:45.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecad9d700 1 -- 172.21.15.26:0/1282573291 reap_dead start 2022-07-20T02:25:45.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efeca59c700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efeac003750 0x7efeac003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:45.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec9d9b700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efeac004980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7efeb400c000 tx=0x7efeb4006df0).ready entity=mon.1 client_cookie=8c75b0d75385c06a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb400e830 con 0x7efeac004500 2022-07-20T02:25:45.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb400e990 con 0x7efeac004500 2022-07-20T02:25:45.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efec959a700 1 -- 172.21.15.26:0/1282573291 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb400ecf0 con 0x7efeac004500 2022-07-20T02:25:45.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 msgr2=0x7efeac004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efeac004980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7efeb400c000 tx=0x7efeb4006df0).stop 2022-07-20T02:25:45.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- 172.21.15.26:0/1282573291 shutdown_connections 2022-07-20T02:25:45.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 --2- 172.21.15.26:0/1282573291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efeac004980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- 172.21.15.26:0/1282573291 >> 172.21.15.26:0/1282573291 conn(0x7efecc0b3050 msgr2=0x7efecc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- 172.21.15.26:0/1282573291 shutdown_connections 2022-07-20T02:25:45.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- 172.21.15.26:0/1282573291 wait complete. 2022-07-20T02:25:45.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 Processor -- start 2022-07-20T02:25:45.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.319+0000 7efecb59e700 1 -- start start 2022-07-20T02:25:45.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efecc144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efecc146e60 0x7efecc13f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc1536c0 con 0x7efecc066100 2022-07-20T02:25:45.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0665a0 con 0x7efeac004500 2022-07-20T02:25:45.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc1538f0 con 0x7efecc146e60 2022-07-20T02:25:45.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35308/0 (socket says 172.21.15.26:35308) 2022-07-20T02:25:45.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 -- 172.21.15.26:0/443078704 learned_addr learned my addr 172.21.15.26:0/443078704 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efeca59c700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efecc144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efec9d9b700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efecc146e60 0x7efecc13f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 -- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efecc146e60 msgr2=0x7efecc13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efecc146e60 0x7efecc13f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 -- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 msgr2=0x7efecc144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efecc144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 -- 172.21.15.26:0/443078704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc153b90 con 0x7efecc066100 2022-07-20T02:25:45.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efeca59c700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efecc144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:45.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecad9d700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7efec4000b60 tx=0x7efec4002190).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec400f820 con 0x7efecc066100 2022-07-20T02:25:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec400f980 con 0x7efecc066100 2022-07-20T02:25:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc154520 con 0x7efecc066100 2022-07-20T02:25:45.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec400fce0 con 0x7efecc066100 2022-07-20T02:25:45.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc150310 con 0x7efecc066100 2022-07-20T02:25:45.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.323+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc05fb10 con 0x7efecc066100 2022-07-20T02:25:45.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.327+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efec401b030 con 0x7efecc066100 2022-07-20T02:25:45.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.327+0000 7efec37fe700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 0x7efeb0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.327+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efec406be00 con 0x7efecc066100 2022-07-20T02:25:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.327+0000 7efeca59c700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 0x7efeb0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.327+0000 7efeca59c700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 0x7efeb0055fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7efebc009000 tx=0x7efebc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.331+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efec4042ab0 con 0x7efecc066100 2022-07-20T02:25:45.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.455+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7efecc0574e0 con 0x7efecc066100 2022-07-20T02:25:45.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.455+0000 7efec37fe700 1 -- 172.21.15.26:0/443078704 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efec4019030 con 0x7efecc066100 2022-07-20T02:25:45.459 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 msgr2=0x7efeb0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 0x7efeb0055fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7efebc009000 tx=0x7efebc009040).stop 2022-07-20T02:25:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 msgr2=0x7efecc154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7efec4000b60 tx=0x7efec4002190).stop 2022-07-20T02:25:45.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 shutdown_connections 2022-07-20T02:25:45.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efecc066100 0x7efecc154030 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb0053b00 0x7efeb0055fc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efecc146e60 0x7efecc13f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 --2- 172.21.15.26:0/443078704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeac004500 0x7efecc144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 >> 172.21.15.26:0/443078704 conn(0x7efecc0b3050 msgr2=0x7efecc05cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 shutdown_connections 2022-07-20T02:25:45.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.459+0000 7efecb59e700 1 -- 172.21.15.26:0/443078704 wait complete. 2022-07-20T02:25:45.474 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2022-07-20T02:25:45.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-07-20T02:25:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 Processor -- start 2022-07-20T02:25:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 -- start start 2022-07-20T02:25:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb6ec05cfd0 0x7fb6ec143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ec1474e0 con 0x7fb6ec147000 2022-07-20T02:25:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ec0b4e40 con 0x7fb6ec060f00 2022-07-20T02:25:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f3139700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ec05d3b0 con 0x7fb6ec05cfd0 2022-07-20T02:25:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1936700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb6ec147000 0x7fb6ec0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34594/0 (socket says 172.21.15.26:34594) 2022-07-20T02:25:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1936700 1 -- 172.21.15.26:0/1089033905 learned_addr learned my addr 172.21.15.26:0/1089033905 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f2137700 1 --2- 172.21.15.26:0/1089033905 >> v2:172.21.15.120:3300/0 conn(0x7fb6ec05cfd0 0x7fb6ec143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211884419 0 0) 0x7fb6ec1474e0 con 0x7fb6ec147000 2022-07-20T02:25:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d8003260 con 0x7fb6ec147000 2022-07-20T02:25:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414462288 0 0) 0x7fb6ec0b4e40 con 0x7fb6ec060f00 2022-07-20T02:25:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ec1474e0 con 0x7fb6ec060f00 2022-07-20T02:25:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f2137700 1 -- 172.21.15.26:0/1089033905 >> v1:172.21.15.120:6789/0 conn(0x7fb6ec060f00 legacy=0x7fb6ec13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f2137700 1 -- 172.21.15.26:0/1089033905 >> v1:172.21.15.26:6789/0 conn(0x7fb6ec147000 legacy=0x7fb6ec0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251744051 0 0) 0x7fb6d8003260 con 0x7fb6ec147000 2022-07-20T02:25:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.583+0000 7fb6f2137700 1 -- 172.21.15.26:0/1089033905 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec1431e0 con 0x7fb6ec05cfd0 2022-07-20T02:25:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f2137700 1 --2- 172.21.15.26:0/1089033905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec05cfd0 0x7fb6ec143bc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb6dc0032f0 tx=0x7fb6dc0091d0).ready entity=mon.1 client_cookie=da275fad355ae0ea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc00b600 con 0x7fb6ec05cfd0 2022-07-20T02:25:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc00b760 con 0x7fb6ec05cfd0 2022-07-20T02:25:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f1135700 1 -- 172.21.15.26:0/1089033905 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc00b8d0 con 0x7fb6ec05cfd0 2022-07-20T02:25:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- 172.21.15.26:0/1089033905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec05cfd0 msgr2=0x7fb6ec143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 --2- 172.21.15.26:0/1089033905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec05cfd0 0x7fb6ec143bc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb6dc0032f0 tx=0x7fb6dc0091d0).stop 2022-07-20T02:25:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- 172.21.15.26:0/1089033905 shutdown_connections 2022-07-20T02:25:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 --2- 172.21.15.26:0/1089033905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec05cfd0 0x7fb6ec143bc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- 172.21.15.26:0/1089033905 >> 172.21.15.26:0/1089033905 conn(0x7fb6ec0620c0 msgr2=0x7fb6ec05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- 172.21.15.26:0/1089033905 shutdown_connections 2022-07-20T02:25:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- 172.21.15.26:0/1089033905 wait complete. 2022-07-20T02:25:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 Processor -- start 2022-07-20T02:25:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- start start 2022-07-20T02:25:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec147000 0x7fb6ec0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ec152230 0x7fb6ec145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec155800 con 0x7fb6ec060f00 2022-07-20T02:25:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec146580 con 0x7fb6ec147000 2022-07-20T02:25:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f3139700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec155a30 con 0x7fb6ec152230 2022-07-20T02:25:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f2137700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f2137700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35318/0 (socket says 172.21.15.26:35318) 2022-07-20T02:25:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f2137700 1 -- 172.21.15.26:0/4012341941 learned_addr learned my addr 172.21.15.26:0/4012341941 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f1936700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec147000 0x7fb6ec0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.587+0000 7fb6f2938700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ec152230 0x7fb6ec145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 -- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ec152230 msgr2=0x7fb6ec145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ec152230 0x7fb6ec145ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 -- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec147000 msgr2=0x7fb6ec0b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec147000 0x7fb6ec0b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 -- 172.21.15.26:0/4012341941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc009470 con 0x7fb6ec060f00 2022-07-20T02:25:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f2137700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb6dc0032f0 tx=0x7fb6dc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc00b600 con 0x7fb6ec060f00 2022-07-20T02:25:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc011e60 con 0x7fb6ec060f00 2022-07-20T02:25:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec155cd0 con 0x7fb6ec060f00 2022-07-20T02:25:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc01f5e0 con 0x7fb6ec060f00 2022-07-20T02:25:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec0b3290 con 0x7fb6ec060f00 2022-07-20T02:25:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec143480 con 0x7fb6ec060f00 2022-07-20T02:25:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb6dc026030 con 0x7fb6ec060f00 2022-07-20T02:25:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.591+0000 7fb6e37fe700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 0x7fb6cc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.595+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb6dc06cbc0 con 0x7fb6ec060f00 2022-07-20T02:25:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.595+0000 7fb6f1936700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 0x7fb6cc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.595+0000 7fb6f1936700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 0x7fb6cc055ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb6e800d000 tx=0x7fb6e8001760).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.599+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6dc043870 con 0x7fb6ec060f00 2022-07-20T02:25:45.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb6ec0574e0 con 0x7fb6ec060f00 2022-07-20T02:25:45.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6e37fe700 1 -- 172.21.15.26:0/4012341941 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6dc024090 con 0x7fb6ec060f00 2022-07-20T02:25:45.729 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:45.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 msgr2=0x7fb6cc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 0x7fb6cc055ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb6e800d000 tx=0x7fb6e8001760).stop 2022-07-20T02:25:45.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 msgr2=0x7fb6ec156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb6dc0032f0 tx=0x7fb6dc016000).stop 2022-07-20T02:25:45.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 shutdown_connections 2022-07-20T02:25:45.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6cc0539e0 0x7fb6cc055ea0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ec152230 0x7fb6ec145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ec147000 0x7fb6ec0b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 --2- 172.21.15.26:0/4012341941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ec060f00 0x7fb6ec156170 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 >> 172.21.15.26:0/4012341941 conn(0x7fb6ec05cfd0 msgr2=0x7fb6ec0b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 shutdown_connections 2022-07-20T02:25:45.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.727+0000 7fb6f3139700 1 -- 172.21.15.26:0/4012341941 wait complete. 2022-07-20T02:25:45.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2022-07-20T02:25:45.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-07-20T02:25:45.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:45.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.859+0000 7f8cf7fff700 1 Processor -- start 2022-07-20T02:25:45.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.859+0000 7f8cf7fff700 1 -- start start 2022-07-20T02:25:45.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cf80670c0 0x7f8cf80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf8064450 0x7f8cf8064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf8064880 con 0x7f8cf80b3380 2022-07-20T02:25:45.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf80b85f0 con 0x7f8cf80670c0 2022-07-20T02:25:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80b80b0 con 0x7f8cf8064450 2022-07-20T02:25:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cf80670c0 0x7f8cf80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cf80670c0 0x7f8cf80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35328/0 (socket says 172.21.15.26:35328) 2022-07-20T02:25:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 -- 172.21.15.26:0/305986760 learned_addr learned my addr 172.21.15.26:0/305986760 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf67fc700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf8064450 0x7f8cf8064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 -- 172.21.15.26:0/305986760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf8064450 msgr2=0x7f8cf8064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:45.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf8064450 0x7f8cf8064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 -- 172.21.15.26:0/305986760 >> v1:172.21.15.120:6789/0 conn(0x7f8cf80b3380 legacy=0x7f8cf80b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 -- 172.21.15.26:0/305986760 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf80b8350 con 0x7f8cf80670c0 2022-07-20T02:25:45.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80670c0 0x7f8cf80b2e90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8ce8002f70 tx=0x7f8ce8006eb0).ready entity=mon.0 client_cookie=f390880bdd694371 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf5ffb700 1 -- 172.21.15.26:0/305986760 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce800d570 con 0x7f8cf80670c0 2022-07-20T02:25:45.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf5ffb700 1 -- 172.21.15.26:0/305986760 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8000e90 con 0x7f8cf80670c0 2022-07-20T02:25:45.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf5ffb700 1 -- 172.21.15.26:0/305986760 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce800d900 con 0x7f8cf80670c0 2022-07-20T02:25:45.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- 172.21.15.26:0/305986760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80670c0 msgr2=0x7f8cf80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80670c0 0x7f8cf80b2e90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8ce8002f70 tx=0x7f8ce8006eb0).stop 2022-07-20T02:25:45.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- 172.21.15.26:0/305986760 shutdown_connections 2022-07-20T02:25:45.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf8064450 0x7f8cf8064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/305986760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80670c0 0x7f8cf80b2e90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- 172.21.15.26:0/305986760 >> 172.21.15.26:0/305986760 conn(0x7f8cf805f840 msgr2=0x7f8cf805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:45.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- 172.21.15.26:0/305986760 shutdown_connections 2022-07-20T02:25:45.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.863+0000 7f8cf7fff700 1 -- 172.21.15.26:0/305986760 wait complete. 2022-07-20T02:25:45.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 Processor -- start 2022-07-20T02:25:45.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- start start 2022-07-20T02:25:45.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 0x7f8cf80af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cf8155ef0 0x7f8cf81563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8155820 con 0x7f8cf80abef0 2022-07-20T02:25:45.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80ac600 con 0x7f8cf8155ef0 2022-07-20T02:25:45.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80ac8a0 con 0x7f8cf80670c0 2022-07-20T02:25:45.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35334/0 (socket says 172.21.15.26:35334) 2022-07-20T02:25:45.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 -- 172.21.15.26:0/3862764306 learned_addr learned my addr 172.21.15.26:0/3862764306 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:45.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 0x7f8cf80af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf77fe700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cf8155ef0 0x7f8cf81563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 -- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 msgr2=0x7f8cf80af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 0x7f8cf80af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 -- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cf8155ef0 msgr2=0x7f8cf81563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:45.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cf8155ef0 0x7f8cf81563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:45.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 -- 172.21.15.26:0/3862764306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce800b1e0 con 0x7f8cf80abef0 2022-07-20T02:25:45.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 0x7f8cf80af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:45.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf67fc700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8ce000c000 tx=0x7f8ce0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce0010780 con 0x7f8cf80abef0 2022-07-20T02:25:45.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce00108e0 con 0x7f8cf80abef0 2022-07-20T02:25:45.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf8155a50 con 0x7f8cf80abef0 2022-07-20T02:25:45.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce0010c40 con 0x7f8cf80abef0 2022-07-20T02:25:45.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.867+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf80b6b00 con 0x7f8cf80abef0 2022-07-20T02:25:45.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf8155be0 con 0x7f8cf80abef0 2022-07-20T02:25:45.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8ce001c460 con 0x7f8cf80abef0 2022-07-20T02:25:45.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8ceffff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 0x7f8cd0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:45.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8ce006c570 con 0x7f8cf80abef0 2022-07-20T02:25:45.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 0x7f8cd0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:45.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.871+0000 7f8cf6ffd700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 0x7f8cd0055e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8ce801a000 tx=0x7f8ce8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:45.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:45.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.879+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ce00432d0 con 0x7f8cf80abef0 2022-07-20T02:25:45.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:45.950 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:25:46.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.999+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8cf8064ab0 con 0x7f8cf80abef0 2022-07-20T02:25:46.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:45.999+0000 7f8ceffff700 1 -- 172.21.15.26:0/3862764306 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ce00468f0 con 0x7f8cf80abef0 2022-07-20T02:25:46.002 INFO:teuthology.orchestra.run.smithi026.stdout:55834574853 2022-07-20T02:25:46.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 msgr2=0x7f8cd0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 0x7f8cd0055e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8ce801a000 tx=0x7f8ce8015000).stop 2022-07-20T02:25:46.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 msgr2=0x7f8cf80b6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8ce000c000 tx=0x7f8ce0006df0).stop 2022-07-20T02:25:46.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 shutdown_connections 2022-07-20T02:25:46.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cf80abef0 0x7f8cf80b6610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cd0053980 0x7f8cd0055e40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cf8155ef0 0x7f8cf81563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 --2- 172.21.15.26:0/3862764306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cf80670c0 0x7f8cf80af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 >> 172.21.15.26:0/3862764306 conn(0x7f8cf805f840 msgr2=0x7f8cf80b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 shutdown_connections 2022-07-20T02:25:46.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.003+0000 7f8cf7fff700 1 -- 172.21.15.26:0/3862764306 wait complete. 2022-07-20T02:25:46.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2022-07-20T02:25:46.024 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-07-20T02:25:46.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T02:25:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 Processor -- start 2022-07-20T02:25:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 -- start start 2022-07-20T02:25:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6760067480 0x7f6760141710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6760144670 con 0x7f67600b49c0 2022-07-20T02:25:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67600b4f40 con 0x7f67600b7920 2022-07-20T02:25:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f6767d20700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67600b44a0 con 0x7f6760067480 2022-07-20T02:25:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f67662bd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6760067480 0x7f6760141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f67652bb700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67600b49c0 0x7f6760141000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34622/0 (socket says 172.21.15.26:34622) 2022-07-20T02:25:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.127+0000 7f67652bb700 1 -- 172.21.15.26:0/1471071758 learned_addr learned my addr 172.21.15.26:0/1471071758 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f67662bd700 1 -- 172.21.15.26:0/1471071758 >> v1:172.21.15.120:6789/0 conn(0x7f67600b7920 legacy=0x7f6760144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f67662bd700 1 -- 172.21.15.26:0/1471071758 >> v1:172.21.15.26:6789/0 conn(0x7f67600b49c0 legacy=0x7f6760141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f67662bd700 1 -- 172.21.15.26:0/1471071758 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67601448a0 con 0x7f6760067480 2022-07-20T02:25:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6764aba700 1 -- 172.21.15.26:0/1471071758 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407150461 0 0) 0x7f67600b4f40 con 0x7f67600b7920 2022-07-20T02:25:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f67662bd700 1 --2- 172.21.15.26:0/1471071758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f6760141710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6750002f70 tx=0x7f6750006df0).ready entity=mon.0 client_cookie=12c91f115909447b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6764aba700 1 -- 172.21.15.26:0/1471071758 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f675000d610 con 0x7f6760067480 2022-07-20T02:25:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6764aba700 1 -- 172.21.15.26:0/1471071758 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6750000e90 con 0x7f6760067480 2022-07-20T02:25:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6764aba700 1 -- 172.21.15.26:0/1471071758 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f675000d960 con 0x7f6760067480 2022-07-20T02:25:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- 172.21.15.26:0/1471071758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 msgr2=0x7f6760141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 --2- 172.21.15.26:0/1471071758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f6760141710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6750002f70 tx=0x7f6750006df0).stop 2022-07-20T02:25:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- 172.21.15.26:0/1471071758 shutdown_connections 2022-07-20T02:25:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 --2- 172.21.15.26:0/1471071758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f6760141710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- 172.21.15.26:0/1471071758 >> 172.21.15.26:0/1471071758 conn(0x7f6760064b10 msgr2=0x7f6760064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- 172.21.15.26:0/1471071758 shutdown_connections 2022-07-20T02:25:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- 172.21.15.26:0/1471071758 wait complete. 2022-07-20T02:25:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 Processor -- start 2022-07-20T02:25:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.131+0000 7f6767d20700 1 -- start start 2022-07-20T02:25:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67600b49c0 0x7f6760139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67600b7920 0x7f6760139fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600ad1c0 con 0x7f6760067480 2022-07-20T02:25:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676013a4a0 con 0x7f67600b7920 2022-07-20T02:25:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67600ad3f0 con 0x7f67600b49c0 2022-07-20T02:25:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35346/0 (socket says 172.21.15.26:35346) 2022-07-20T02:25:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 -- 172.21.15.26:0/2977199812 learned_addr learned my addr 172.21.15.26:0/2977199812 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f67662bd700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67600b7920 0x7f6760139fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f67652bb700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67600b49c0 0x7f6760139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 -- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67600b49c0 msgr2=0x7f6760139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67600b49c0 0x7f6760139ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 -- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67600b7920 msgr2=0x7f6760139fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67600b7920 0x7f6760139fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 -- 172.21.15.26:0/2977199812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675000b180 con 0x7f6760067480 2022-07-20T02:25:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6765abc700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f676013a630 tx=0x7f6758001490).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6758010510 con 0x7f6760067480 2022-07-20T02:25:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6758010670 con 0x7f6760067480 2022-07-20T02:25:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67600ae0a0 con 0x7f6760067480 2022-07-20T02:25:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67580107e0 con 0x7f6760067480 2022-07-20T02:25:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67600aa790 con 0x7f6760067480 2022-07-20T02:25:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6758010070 con 0x7f6760067480 2022-07-20T02:25:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600ae540 con 0x7f6760067480 2022-07-20T02:25:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.135+0000 7f675effd700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 0x7f6740055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.139+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f675806bbe0 con 0x7f6760067480 2022-07-20T02:25:46.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.139+0000 7f67652bb700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 0x7f6740055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.139+0000 7f67652bb700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 0x7f6740055f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f675400b000 tx=0x7f6754009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.143+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67600ae540 con 0x7f6760067480 2022-07-20T02:25:46.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.267+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67600ae540 con 0x7f6740053aa0 2022-07-20T02:25:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f675effd700 1 -- 172.21.15.26:0/2977199812 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82333 (secure 0 0 0) 0x7f67600ae540 con 0x7f6740053aa0 2022-07-20T02:25:46.280 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:46.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 msgr2=0x7f6740055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 0x7f6740055f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f675400b000 tx=0x7f6754009040).stop 2022-07-20T02:25:46.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 msgr2=0x7f67600ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f676013a630 tx=0x7f6758001490).stop 2022-07-20T02:25:46.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 shutdown_connections 2022-07-20T02:25:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6740053aa0 0x7f6740055f60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67600b7920 0x7f6760139fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67600b49c0 0x7f6760139ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 --2- 172.21.15.26:0/2977199812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6760067480 0x7f67600ac550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 >> 172.21.15.26:0/2977199812 conn(0x7f6760064b10 msgr2=0x7f676013db40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 shutdown_connections 2022-07-20T02:25:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.275+0000 7f6767d20700 1 -- 172.21.15.26:0/2977199812 wait complete. 2022-07-20T02:25:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2022-07-20T02:25:46.296 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2022-07-20T02:25:45.381579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":817128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.762943"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.789840+0000","last_change":"2022-07-20T02:25:35.784291+0000","last_active":"2022-07-20T02:25:35.789840+0000","last_peered":"2022-07-20T02:25:35.789840+0000","last_clean":"2022-07-20T02:25:35.789840+0000","last_became_active":"2022-07-20T02:25:35.783225+0000","last_became_peered":"2022-07-20T02:25:35.783225+0000","last_unstale":"2022-07-20T02:25:35.789840+0000","last_undegraded":"2022-07-20T02:25:35.789840+0000","last_fullsized":"2022-07-20T02:25:35.789840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779703+0000","last_change":"2022-07-20T02:25:25.353187+0000","last_active":"2022-07-20T02:25:35.779703+0000","last_peered":"2022-07-20T02:25:35.779703+0000","last_clean":"2022-07-20T02:25:35.779703+0000","last_became_active":"2022-07-20T02:25:25.351806+0000","last_became_peered":"2022-07-20T02:25:25.351806+0000","last_unstale":"2022-07-20T02:25:35.779703+0000","last_undegraded":"2022-07-20T02:25:35.779703+0000","last_fullsized":"2022-07-20T02:25:35.779703+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_clean_scrub_stamp":"2022-07-20T02:25:19.583498+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683447+0000","last_change":"2022-07-20T02:25:29.458996+0000","last_active":"2022-07-20T02:25:35.683447+0000","last_peered":"2022-07-20T02:25:35.683447+0000","last_clean":"2022-07-20T02:25:35.683447+0000","last_became_active":"2022-07-20T02:25:29.458580+0000","last_became_peered":"2022-07-20T02:25:29.458580+0000","last_unstale":"2022-07-20T02:25:35.683447+0000","last_undegraded":"2022-07-20T02:25:35.683447+0000","last_fullsized":"2022-07-20T02:25:35.683447+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.952348+0000","last_change":"2022-07-20T02:25:32.625166+0000","last_active":"2022-07-20T02:25:35.952348+0000","last_peered":"2022-07-20T02:25:35.952348+0000","last_clean":"2022-07-20T02:25:35.952348+0000","last_became_active":"2022-07-20T02:25:32.624603+0000","last_became_peered":"2022-07-20T02:25:32.624603+0000","last_unstale":"2022-07-20T02:25:35.952348+0000","last_undegraded":"2022-07-20T02:25:35.952348+0000","last_fullsized":"2022-07-20T02:25:35.952348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.823095+0000","last_change":"2022-07-20T02:25:35.821636+0000","last_active":"2022-07-20T02:25:35.823095+0000","last_peered":"2022-07-20T02:25:35.823095+0000","last_clean":"2022-07-20T02:25:35.823095+0000","last_became_active":"2022-07-20T02:25:35.820101+0000","last_became_peered":"2022-07-20T02:25:35.820101+0000","last_unstale":"2022-07-20T02:25:35.823095+0000","last_undegraded":"2022-07-20T02:25:35.823095+0000","last_fullsized":"2022-07-20T02:25:35.823095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779654+0000","last_change":"2022-07-20T02:25:29.455281+0000","last_active":"2022-07-20T02:25:35.779654+0000","last_peered":"2022-07-20T02:25:35.779654+0000","last_clean":"2022-07-20T02:25:35.779654+0000","last_became_active":"2022-07-20T02:25:29.454947+0000","last_became_peered":"2022-07-20T02:25:29.454947+0000","last_unstale":"2022-07-20T02:25:35.779654+0000","last_undegraded":"2022-07-20T02:25:35.779654+0000","last_fullsized":"2022-07-20T02:25:35.779654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.629581+0000","last_change":"2022-07-20T02:25:32.625168+0000","last_active":"2022-07-20T02:25:35.629581+0000","last_peered":"2022-07-20T02:25:35.629581+0000","last_clean":"2022-07-20T02:25:35.629581+0000","last_became_active":"2022-07-20T02:25:32.624937+0000","last_became_peered":"2022-07-20T02:25:32.624937+0000","last_unstale":"2022-07-20T02:25:35.629581+0000","last_undegraded":"2022-07-20T02:25:35.629581+0000","last_fullsized":"2022-07-20T02:25:35.629581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.684817+0000","last_change":"2022-07-20T02:25:35.683397+0000","last_active":"2022-07-20T02:25:35.684817+0000","last_peered":"2022-07-20T02:25:35.684817+0000","last_clean":"2022-07-20T02:25:35.684817+0000","last_became_active":"2022-07-20T02:25:35.681436+0000","last_became_peered":"2022-07-20T02:25:35.681436+0000","last_unstale":"2022-07-20T02:25:35.684817+0000","last_undegraded":"2022-07-20T02:25:35.684817+0000","last_fullsized":"2022-07-20T02:25:35.684817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.816736+0000","last_change":"2022-07-20T02:25:29.454317+0000","last_active":"2022-07-20T02:25:35.816736+0000","last_peered":"2022-07-20T02:25:35.816736+0000","last_clean":"2022-07-20T02:25:35.816736+0000","last_became_active":"2022-07-20T02:25:29.453725+0000","last_became_peered":"2022-07-20T02:25:29.453725+0000","last_unstale":"2022-07-20T02:25:35.816736+0000","last_undegraded":"2022-07-20T02:25:35.816736+0000","last_fullsized":"2022-07-20T02:25:35.816736+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.969323+0000","last_change":"2022-07-20T02:25:32.620050+0000","last_active":"2022-07-20T02:25:35.969323+0000","last_peered":"2022-07-20T02:25:35.969323+0000","last_clean":"2022-07-20T02:25:35.969323+0000","last_became_active":"2022-07-20T02:25:32.619629+0000","last_became_peered":"2022-07-20T02:25:32.619629+0000","last_unstale":"2022-07-20T02:25:35.969323+0000","last_undegraded":"2022-07-20T02:25:35.969323+0000","last_fullsized":"2022-07-20T02:25:35.969323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.822673+0000","last_change":"2022-07-20T02:25:35.820454+0000","last_active":"2022-07-20T02:25:35.822673+0000","last_peered":"2022-07-20T02:25:35.822673+0000","last_clean":"2022-07-20T02:25:35.822673+0000","last_became_active":"2022-07-20T02:25:35.819470+0000","last_became_peered":"2022-07-20T02:25:35.819470+0000","last_unstale":"2022-07-20T02:25:35.822673+0000","last_undegraded":"2022-07-20T02:25:35.822673+0000","last_fullsized":"2022-07-20T02:25:35.822673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683633+0000","last_change":"2022-07-20T02:25:29.457758+0000","last_active":"2022-07-20T02:25:35.683633+0000","last_peered":"2022-07-20T02:25:35.683633+0000","last_clean":"2022-07-20T02:25:35.683633+0000","last_became_active":"2022-07-20T02:25:29.457326+0000","last_became_peered":"2022-07-20T02:25:29.457326+0000","last_unstale":"2022-07-20T02:25:35.683633+0000","last_undegraded":"2022-07-20T02:25:35.683633+0000","last_fullsized":"2022-07-20T02:25:35.683633+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.974523+0000","last_change":"2022-07-20T02:25:32.628588+0000","last_active":"2022-07-20T02:25:35.974523+0000","last_peered":"2022-07-20T02:25:35.974523+0000","last_clean":"2022-07-20T02:25:35.974523+0000","last_became_active":"2022-07-20T02:25:32.627969+0000","last_became_peered":"2022-07-20T02:25:32.627969+0000","last_unstale":"2022-07-20T02:25:35.974523+0000","last_undegraded":"2022-07-20T02:25:35.974523+0000","last_fullsized":"2022-07-20T02:25:35.974523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683363+0000","last_change":"2022-07-20T02:25:35.681725+0000","last_active":"2022-07-20T02:25:35.683363+0000","last_peered":"2022-07-20T02:25:35.683363+0000","last_clean":"2022-07-20T02:25:35.683363+0000","last_became_active":"2022-07-20T02:25:35.681101+0000","last_became_peered":"2022-07-20T02:25:35.681101+0000","last_unstale":"2022-07-20T02:25:35.683363+0000","last_undegraded":"2022-07-20T02:25:35.683363+0000","last_fullsized":"2022-07-20T02:25:35.683363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.977689+0000","last_change":"2022-07-20T02:25:32.632438+0000","last_active":"2022-07-20T02:25:35.977689+0000","last_peered":"2022-07-20T02:25:35.977689+0000","last_clean":"2022-07-20T02:25:35.977689+0000","last_became_active":"2022-07-20T02:25:32.631973+0000","last_became_peered":"2022-07-20T02:25:32.631973+0000","last_unstale":"2022-07-20T02:25:35.977689+0000","last_undegraded":"2022-07-20T02:25:35.977689+0000","last_fullsized":"2022-07-20T02:25:35.977689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.625220+0000","last_change":"2022-07-20T02:25:29.902408+0000","last_active":"2022-07-20T02:25:35.625220+0000","last_peered":"2022-07-20T02:25:35.625220+0000","last_clean":"2022-07-20T02:25:35.625220+0000","last_became_active":"2022-07-20T02:25:29.901376+0000","last_became_peered":"2022-07-20T02:25:29.901376+0000","last_unstale":"2022-07-20T02:25:35.625220+0000","last_undegraded":"2022-07-20T02:25:35.625220+0000","last_fullsized":"2022-07-20T02:25:35.625220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.780880+0000","last_change":"2022-07-20T02:25:35.776638+0000","last_active":"2022-07-20T02:25:35.780880+0000","last_peered":"2022-07-20T02:25:35.780880+0000","last_clean":"2022-07-20T02:25:35.780880+0000","last_became_active":"2022-07-20T02:25:35.775969+0000","last_became_peered":"2022-07-20T02:25:35.775969+0000","last_unstale":"2022-07-20T02:25:35.780880+0000","last_undegraded":"2022-07-20T02:25:35.780880+0000","last_fullsized":"2022-07-20T02:25:35.780880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.970044+0000","last_change":"2022-07-20T02:25:32.620682+0000","last_active":"2022-07-20T02:25:35.970044+0000","last_peered":"2022-07-20T02:25:35.970044+0000","last_clean":"2022-07-20T02:25:35.970044+0000","last_became_active":"2022-07-20T02:25:32.620458+0000","last_became_peered":"2022-07-20T02:25:32.620458+0000","last_unstale":"2022-07-20T02:25:35.970044+0000","last_undegraded":"2022-07-20T02:25:35.970044+0000","last_fullsized":"2022-07-20T02:25:35.970044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779792+0000","last_change":"2022-07-20T02:25:29.909282+0000","last_active":"2022-07-20T02:25:35.779792+0000","last_peered":"2022-07-20T02:25:35.779792+0000","last_clean":"2022-07-20T02:25:35.779792+0000","last_became_active":"2022-07-20T02:25:29.908425+0000","last_became_peered":"2022-07-20T02:25:29.908425+0000","last_unstale":"2022-07-20T02:25:35.779792+0000","last_undegraded":"2022-07-20T02:25:35.779792+0000","last_fullsized":"2022-07-20T02:25:35.779792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.777403+0000","last_change":"2022-07-20T02:25:35.776114+0000","last_active":"2022-07-20T02:25:35.777403+0000","last_peered":"2022-07-20T02:25:35.777403+0000","last_clean":"2022-07-20T02:25:35.777403+0000","last_became_active":"2022-07-20T02:25:35.775573+0000","last_became_peered":"2022-07-20T02:25:35.775573+0000","last_unstale":"2022-07-20T02:25:35.777403+0000","last_undegraded":"2022-07-20T02:25:35.777403+0000","last_fullsized":"2022-07-20T02:25:35.777403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'7","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.968696+0000","last_change":"2022-07-20T02:25:32.743382+0000","last_active":"2022-07-20T02:25:35.968696+0000","last_peered":"2022-07-20T02:25:35.968696+0000","last_clean":"2022-07-20T02:25:35.968696+0000","last_became_active":"2022-07-20T02:25:32.742780+0000","last_became_peered":"2022-07-20T02:25:32.742780+0000","last_unstale":"2022-07-20T02:25:35.968696+0000","last_undegraded":"2022-07-20T02:25:35.968696+0000","last_fullsized":"2022-07-20T02:25:35.968696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.625135+0000","last_change":"2022-07-20T02:25:29.452052+0000","last_active":"2022-07-20T02:25:35.625135+0000","last_peered":"2022-07-20T02:25:35.625135+0000","last_clean":"2022-07-20T02:25:35.625135+0000","last_became_active":"2022-07-20T02:25:29.451566+0000","last_became_peered":"2022-07-20T02:25:29.451566+0000","last_unstale":"2022-07-20T02:25:35.625135+0000","last_undegraded":"2022-07-20T02:25:35.625135+0000","last_fullsized":"2022-07-20T02:25:35.625135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.785236+0000","last_change":"2022-07-20T02:25:35.783868+0000","last_active":"2022-07-20T02:25:35.785236+0000","last_peered":"2022-07-20T02:25:35.785236+0000","last_clean":"2022-07-20T02:25:35.785236+0000","last_became_active":"2022-07-20T02:25:35.783229+0000","last_became_peered":"2022-07-20T02:25:35.783229+0000","last_unstale":"2022-07-20T02:25:35.785236+0000","last_undegraded":"2022-07-20T02:25:35.785236+0000","last_fullsized":"2022-07-20T02:25:35.785236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"24'4","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.969262+0000","last_change":"2022-07-20T02:25:32.622438+0000","last_active":"2022-07-20T02:25:35.969262+0000","last_peered":"2022-07-20T02:25:35.969262+0000","last_clean":"2022-07-20T02:25:35.969262+0000","last_became_active":"2022-07-20T02:25:32.622028+0000","last_became_peered":"2022-07-20T02:25:32.622028+0000","last_unstale":"2022-07-20T02:25:35.969262+0000","last_undegraded":"2022-07-20T02:25:35.969262+0000","last_fullsized":"2022-07-20T02:25:35.969262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.628836+0000","last_change":"2022-07-20T02:25:29.458394+0000","last_active":"2022-07-20T02:25:35.628836+0000","last_peered":"2022-07-20T02:25:35.628836+0000","last_clean":"2022-07-20T02:25:35.628836+0000","last_became_active":"2022-07-20T02:25:29.458000+0000","last_became_peered":"2022-07-20T02:25:29.458000+0000","last_unstale":"2022-07-20T02:25:35.628836+0000","last_undegraded":"2022-07-20T02:25:35.628836+0000","last_fullsized":"2022-07-20T02:25:35.628836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":102140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":101565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":101599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":102300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":103100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":103100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":101599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]}]}} 2022-07-20T02:25:46.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T02:25:46.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 Processor -- start 2022-07-20T02:25:46.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- start start 2022-07-20T02:25:46.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb40c0b5ef0 0x7fb40c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb40c0a7fc0 0x7fb40c1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40c0b62d0 con 0x7fb40c0a8c40 2022-07-20T02:25:46.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40c147eb0 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40c14aac0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb40c0b5ef0 0x7fb40c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb40c0b5ef0 0x7fb40c1474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35356/0 (socket says 172.21.15.26:35356) 2022-07-20T02:25:46.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 -- 172.21.15.26:0/1589173825 learned_addr learned my addr 172.21.15.26:0/1589173825 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:46.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb409d9b700 1 --2- 172.21.15.26:0/1589173825 >> v2:172.21.15.120:3300/0 conn(0x7fb40c0a7fc0 0x7fb40c1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 -- 172.21.15.26:0/1589173825 >> v2:172.21.15.120:3300/0 conn(0x7fb40c0a7fc0 msgr2=0x7fb40c1479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:46.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 --2- 172.21.15.26:0/1589173825 >> v2:172.21.15.120:3300/0 conn(0x7fb40c0a7fc0 0x7fb40c1479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 -- 172.21.15.26:0/1589173825 >> v1:172.21.15.120:6789/0 conn(0x7fb40c0a8c40 legacy=0x7fb40c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 -- 172.21.15.26:0/1589173825 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c14ad60 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40a59c700 1 --2- 172.21.15.26:0/1589173825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c1474d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3f4002f70 tx=0x7fb3f4006eb0).ready entity=mon.0 client_cookie=e92f72a32f644d71 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40959a700 1 -- 172.21.15.26:0/1589173825 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3f400d5c0 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40959a700 1 -- 172.21.15.26:0/1589173825 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4000e90 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40959a700 1 -- 172.21.15.26:0/1589173825 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3f400d950 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- 172.21.15.26:0/1589173825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0b5ef0 msgr2=0x7fb40c1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1589173825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c1474d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3f4002f70 tx=0x7fb3f4006eb0).stop 2022-07-20T02:25:46.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- 172.21.15.26:0/1589173825 shutdown_connections 2022-07-20T02:25:46.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1589173825 >> v2:172.21.15.120:3300/0 conn(0x7fb40c0a7fc0 0x7fb40c1479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1589173825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c1474d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- 172.21.15.26:0/1589173825 >> 172.21.15.26:0/1589173825 conn(0x7fb40c060310 msgr2=0x7fb40c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- 172.21.15.26:0/1589173825 shutdown_connections 2022-07-20T02:25:46.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.399+0000 7fb40b59e700 1 -- 172.21.15.26:0/1589173825 wait complete. 2022-07-20T02:25:46.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 Processor -- start 2022-07-20T02:25:46.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- start start 2022-07-20T02:25:46.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb40c0a8c40 0x7fb40c062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c062f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0640c0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c065440 con 0x7fb40c0b5ef0 2022-07-20T02:25:46.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0ac240 con 0x7fb40c0a8c40 2022-07-20T02:25:46.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35360/0 (socket says 172.21.15.26:35360) 2022-07-20T02:25:46.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 -- 172.21.15.26:0/1964256808 learned_addr learned my addr 172.21.15.26:0/1964256808 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:46.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40ad9d700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c062f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb409d9b700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb40c0a8c40 0x7fb40c062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 -- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb40c0a8c40 msgr2=0x7fb40c062a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb40c0a8c40 0x7fb40c062a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 -- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb40c0b5ef0 msgr2=0x7fb40c062f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c062f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 -- 172.21.15.26:0/1964256808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f400b1e0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40a59c700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb3f4002f70 tx=0x7fb3f4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3f400d940 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4011ce0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c0aaed0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3f401e650 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.403+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c063450 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb3f4025030 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c0ab060 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb4037fe700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 0x7fb3f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb3f406d110 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb409d9b700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 0x7fb3f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.407+0000 7fb409d9b700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 0x7fb3f0055fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb40400b000 tx=0x7fb404008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.411+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3f4043ed0 con 0x7fb40c0a7fc0 2022-07-20T02:25:46.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.535+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb40c0ab490 con 0x7fb3f0053b00 2022-07-20T02:25:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.539+0000 7fb4037fe700 1 -- 172.21.15.26:0/1964256808 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82333 (secure 0 0 0) 0x7fb404009990 con 0x7fb3f0053b00 2022-07-20T02:25:46.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 msgr2=0x7fb3f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 0x7fb3f0055fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb40400b000 tx=0x7fb404008040).stop 2022-07-20T02:25:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 msgr2=0x7fb40c0b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb3f4002f70 tx=0x7fb3f4015000).stop 2022-07-20T02:25:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 shutdown_connections 2022-07-20T02:25:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb3f0053b00 0x7fb3f0055fc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb40c0b5ef0 0x7fb40c062f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb40c0a8c40 0x7fb40c062a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 --2- 172.21.15.26:0/1964256808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb40c0a7fc0 0x7fb40c0b4560 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 >> 172.21.15.26:0/1964256808 conn(0x7fb40c060310 msgr2=0x7fb40c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 shutdown_connections 2022-07-20T02:25:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.543+0000 7fb40b59e700 1 -- 172.21.15.26:0/1964256808 wait complete. 2022-07-20T02:25:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2022-07-20T02:25:46.587 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2022-07-20T02:25:45.381579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":817128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.762943"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.789840+0000","last_change":"2022-07-20T02:25:35.784291+0000","last_active":"2022-07-20T02:25:35.789840+0000","last_peered":"2022-07-20T02:25:35.789840+0000","last_clean":"2022-07-20T02:25:35.789840+0000","last_became_active":"2022-07-20T02:25:35.783225+0000","last_became_peered":"2022-07-20T02:25:35.783225+0000","last_unstale":"2022-07-20T02:25:35.789840+0000","last_undegraded":"2022-07-20T02:25:35.789840+0000","last_fullsized":"2022-07-20T02:25:35.789840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779703+0000","last_change":"2022-07-20T02:25:25.353187+0000","last_active":"2022-07-20T02:25:35.779703+0000","last_peered":"2022-07-20T02:25:35.779703+0000","last_clean":"2022-07-20T02:25:35.779703+0000","last_became_active":"2022-07-20T02:25:25.351806+0000","last_became_peered":"2022-07-20T02:25:25.351806+0000","last_unstale":"2022-07-20T02:25:35.779703+0000","last_undegraded":"2022-07-20T02:25:35.779703+0000","last_fullsized":"2022-07-20T02:25:35.779703+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_clean_scrub_stamp":"2022-07-20T02:25:19.583498+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683447+0000","last_change":"2022-07-20T02:25:29.458996+0000","last_active":"2022-07-20T02:25:35.683447+0000","last_peered":"2022-07-20T02:25:35.683447+0000","last_clean":"2022-07-20T02:25:35.683447+0000","last_became_active":"2022-07-20T02:25:29.458580+0000","last_became_peered":"2022-07-20T02:25:29.458580+0000","last_unstale":"2022-07-20T02:25:35.683447+0000","last_undegraded":"2022-07-20T02:25:35.683447+0000","last_fullsized":"2022-07-20T02:25:35.683447+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.952348+0000","last_change":"2022-07-20T02:25:32.625166+0000","last_active":"2022-07-20T02:25:35.952348+0000","last_peered":"2022-07-20T02:25:35.952348+0000","last_clean":"2022-07-20T02:25:35.952348+0000","last_became_active":"2022-07-20T02:25:32.624603+0000","last_became_peered":"2022-07-20T02:25:32.624603+0000","last_unstale":"2022-07-20T02:25:35.952348+0000","last_undegraded":"2022-07-20T02:25:35.952348+0000","last_fullsized":"2022-07-20T02:25:35.952348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.823095+0000","last_change":"2022-07-20T02:25:35.821636+0000","last_active":"2022-07-20T02:25:35.823095+0000","last_peered":"2022-07-20T02:25:35.823095+0000","last_clean":"2022-07-20T02:25:35.823095+0000","last_became_active":"2022-07-20T02:25:35.820101+0000","last_became_peered":"2022-07-20T02:25:35.820101+0000","last_unstale":"2022-07-20T02:25:35.823095+0000","last_undegraded":"2022-07-20T02:25:35.823095+0000","last_fullsized":"2022-07-20T02:25:35.823095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779654+0000","last_change":"2022-07-20T02:25:29.455281+0000","last_active":"2022-07-20T02:25:35.779654+0000","last_peered":"2022-07-20T02:25:35.779654+0000","last_clean":"2022-07-20T02:25:35.779654+0000","last_became_active":"2022-07-20T02:25:29.454947+0000","last_became_peered":"2022-07-20T02:25:29.454947+0000","last_unstale":"2022-07-20T02:25:35.779654+0000","last_undegraded":"2022-07-20T02:25:35.779654+0000","last_fullsized":"2022-07-20T02:25:35.779654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.629581+0000","last_change":"2022-07-20T02:25:32.625168+0000","last_active":"2022-07-20T02:25:35.629581+0000","last_peered":"2022-07-20T02:25:35.629581+0000","last_clean":"2022-07-20T02:25:35.629581+0000","last_became_active":"2022-07-20T02:25:32.624937+0000","last_became_peered":"2022-07-20T02:25:32.624937+0000","last_unstale":"2022-07-20T02:25:35.629581+0000","last_undegraded":"2022-07-20T02:25:35.629581+0000","last_fullsized":"2022-07-20T02:25:35.629581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.684817+0000","last_change":"2022-07-20T02:25:35.683397+0000","last_active":"2022-07-20T02:25:35.684817+0000","last_peered":"2022-07-20T02:25:35.684817+0000","last_clean":"2022-07-20T02:25:35.684817+0000","last_became_active":"2022-07-20T02:25:35.681436+0000","last_became_peered":"2022-07-20T02:25:35.681436+0000","last_unstale":"2022-07-20T02:25:35.684817+0000","last_undegraded":"2022-07-20T02:25:35.684817+0000","last_fullsized":"2022-07-20T02:25:35.684817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.816736+0000","last_change":"2022-07-20T02:25:29.454317+0000","last_active":"2022-07-20T02:25:35.816736+0000","last_peered":"2022-07-20T02:25:35.816736+0000","last_clean":"2022-07-20T02:25:35.816736+0000","last_became_active":"2022-07-20T02:25:29.453725+0000","last_became_peered":"2022-07-20T02:25:29.453725+0000","last_unstale":"2022-07-20T02:25:35.816736+0000","last_undegraded":"2022-07-20T02:25:35.816736+0000","last_fullsized":"2022-07-20T02:25:35.816736+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.969323+0000","last_change":"2022-07-20T02:25:32.620050+0000","last_active":"2022-07-20T02:25:35.969323+0000","last_peered":"2022-07-20T02:25:35.969323+0000","last_clean":"2022-07-20T02:25:35.969323+0000","last_became_active":"2022-07-20T02:25:32.619629+0000","last_became_peered":"2022-07-20T02:25:32.619629+0000","last_unstale":"2022-07-20T02:25:35.969323+0000","last_undegraded":"2022-07-20T02:25:35.969323+0000","last_fullsized":"2022-07-20T02:25:35.969323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.822673+0000","last_change":"2022-07-20T02:25:35.820454+0000","last_active":"2022-07-20T02:25:35.822673+0000","last_peered":"2022-07-20T02:25:35.822673+0000","last_clean":"2022-07-20T02:25:35.822673+0000","last_became_active":"2022-07-20T02:25:35.819470+0000","last_became_peered":"2022-07-20T02:25:35.819470+0000","last_unstale":"2022-07-20T02:25:35.822673+0000","last_undegraded":"2022-07-20T02:25:35.822673+0000","last_fullsized":"2022-07-20T02:25:35.822673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683633+0000","last_change":"2022-07-20T02:25:29.457758+0000","last_active":"2022-07-20T02:25:35.683633+0000","last_peered":"2022-07-20T02:25:35.683633+0000","last_clean":"2022-07-20T02:25:35.683633+0000","last_became_active":"2022-07-20T02:25:29.457326+0000","last_became_peered":"2022-07-20T02:25:29.457326+0000","last_unstale":"2022-07-20T02:25:35.683633+0000","last_undegraded":"2022-07-20T02:25:35.683633+0000","last_fullsized":"2022-07-20T02:25:35.683633+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.974523+0000","last_change":"2022-07-20T02:25:32.628588+0000","last_active":"2022-07-20T02:25:35.974523+0000","last_peered":"2022-07-20T02:25:35.974523+0000","last_clean":"2022-07-20T02:25:35.974523+0000","last_became_active":"2022-07-20T02:25:32.627969+0000","last_became_peered":"2022-07-20T02:25:32.627969+0000","last_unstale":"2022-07-20T02:25:35.974523+0000","last_undegraded":"2022-07-20T02:25:35.974523+0000","last_fullsized":"2022-07-20T02:25:35.974523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.683363+0000","last_change":"2022-07-20T02:25:35.681725+0000","last_active":"2022-07-20T02:25:35.683363+0000","last_peered":"2022-07-20T02:25:35.683363+0000","last_clean":"2022-07-20T02:25:35.683363+0000","last_became_active":"2022-07-20T02:25:35.681101+0000","last_became_peered":"2022-07-20T02:25:35.681101+0000","last_unstale":"2022-07-20T02:25:35.683363+0000","last_undegraded":"2022-07-20T02:25:35.683363+0000","last_fullsized":"2022-07-20T02:25:35.683363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.977689+0000","last_change":"2022-07-20T02:25:32.632438+0000","last_active":"2022-07-20T02:25:35.977689+0000","last_peered":"2022-07-20T02:25:35.977689+0000","last_clean":"2022-07-20T02:25:35.977689+0000","last_became_active":"2022-07-20T02:25:32.631973+0000","last_became_peered":"2022-07-20T02:25:32.631973+0000","last_unstale":"2022-07-20T02:25:35.977689+0000","last_undegraded":"2022-07-20T02:25:35.977689+0000","last_fullsized":"2022-07-20T02:25:35.977689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.625220+0000","last_change":"2022-07-20T02:25:29.902408+0000","last_active":"2022-07-20T02:25:35.625220+0000","last_peered":"2022-07-20T02:25:35.625220+0000","last_clean":"2022-07-20T02:25:35.625220+0000","last_became_active":"2022-07-20T02:25:29.901376+0000","last_became_peered":"2022-07-20T02:25:29.901376+0000","last_unstale":"2022-07-20T02:25:35.625220+0000","last_undegraded":"2022-07-20T02:25:35.625220+0000","last_fullsized":"2022-07-20T02:25:35.625220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.780880+0000","last_change":"2022-07-20T02:25:35.776638+0000","last_active":"2022-07-20T02:25:35.780880+0000","last_peered":"2022-07-20T02:25:35.780880+0000","last_clean":"2022-07-20T02:25:35.780880+0000","last_became_active":"2022-07-20T02:25:35.775969+0000","last_became_peered":"2022-07-20T02:25:35.775969+0000","last_unstale":"2022-07-20T02:25:35.780880+0000","last_undegraded":"2022-07-20T02:25:35.780880+0000","last_fullsized":"2022-07-20T02:25:35.780880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.970044+0000","last_change":"2022-07-20T02:25:32.620682+0000","last_active":"2022-07-20T02:25:35.970044+0000","last_peered":"2022-07-20T02:25:35.970044+0000","last_clean":"2022-07-20T02:25:35.970044+0000","last_became_active":"2022-07-20T02:25:32.620458+0000","last_became_peered":"2022-07-20T02:25:32.620458+0000","last_unstale":"2022-07-20T02:25:35.970044+0000","last_undegraded":"2022-07-20T02:25:35.970044+0000","last_fullsized":"2022-07-20T02:25:35.970044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.779792+0000","last_change":"2022-07-20T02:25:29.909282+0000","last_active":"2022-07-20T02:25:35.779792+0000","last_peered":"2022-07-20T02:25:35.779792+0000","last_clean":"2022-07-20T02:25:35.779792+0000","last_became_active":"2022-07-20T02:25:29.908425+0000","last_became_peered":"2022-07-20T02:25:29.908425+0000","last_unstale":"2022-07-20T02:25:35.779792+0000","last_undegraded":"2022-07-20T02:25:35.779792+0000","last_fullsized":"2022-07-20T02:25:35.779792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.777403+0000","last_change":"2022-07-20T02:25:35.776114+0000","last_active":"2022-07-20T02:25:35.777403+0000","last_peered":"2022-07-20T02:25:35.777403+0000","last_clean":"2022-07-20T02:25:35.777403+0000","last_became_active":"2022-07-20T02:25:35.775573+0000","last_became_peered":"2022-07-20T02:25:35.775573+0000","last_unstale":"2022-07-20T02:25:35.777403+0000","last_undegraded":"2022-07-20T02:25:35.777403+0000","last_fullsized":"2022-07-20T02:25:35.777403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'7","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.968696+0000","last_change":"2022-07-20T02:25:32.743382+0000","last_active":"2022-07-20T02:25:35.968696+0000","last_peered":"2022-07-20T02:25:35.968696+0000","last_clean":"2022-07-20T02:25:35.968696+0000","last_became_active":"2022-07-20T02:25:32.742780+0000","last_became_peered":"2022-07-20T02:25:32.742780+0000","last_unstale":"2022-07-20T02:25:35.968696+0000","last_undegraded":"2022-07-20T02:25:35.968696+0000","last_fullsized":"2022-07-20T02:25:35.968696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.625135+0000","last_change":"2022-07-20T02:25:29.452052+0000","last_active":"2022-07-20T02:25:35.625135+0000","last_peered":"2022-07-20T02:25:35.625135+0000","last_clean":"2022-07-20T02:25:35.625135+0000","last_became_active":"2022-07-20T02:25:29.451566+0000","last_became_peered":"2022-07-20T02:25:29.451566+0000","last_unstale":"2022-07-20T02:25:35.625135+0000","last_undegraded":"2022-07-20T02:25:35.625135+0000","last_fullsized":"2022-07-20T02:25:35.625135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.785236+0000","last_change":"2022-07-20T02:25:35.783868+0000","last_active":"2022-07-20T02:25:35.785236+0000","last_peered":"2022-07-20T02:25:35.785236+0000","last_clean":"2022-07-20T02:25:35.785236+0000","last_became_active":"2022-07-20T02:25:35.783229+0000","last_became_peered":"2022-07-20T02:25:35.783229+0000","last_unstale":"2022-07-20T02:25:35.785236+0000","last_undegraded":"2022-07-20T02:25:35.785236+0000","last_fullsized":"2022-07-20T02:25:35.785236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:34.610368+0000","last_clean_scrub_stamp":"2022-07-20T02:25:34.610368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"24'4","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.969262+0000","last_change":"2022-07-20T02:25:32.622438+0000","last_active":"2022-07-20T02:25:35.969262+0000","last_peered":"2022-07-20T02:25:35.969262+0000","last_clean":"2022-07-20T02:25:35.969262+0000","last_became_active":"2022-07-20T02:25:32.622028+0000","last_became_peered":"2022-07-20T02:25:32.622028+0000","last_unstale":"2022-07-20T02:25:35.969262+0000","last_undegraded":"2022-07-20T02:25:35.969262+0000","last_fullsized":"2022-07-20T02:25:35.969262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:31.601731+0000","last_clean_scrub_stamp":"2022-07-20T02:25:31.601731+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-07-20T02:25:35.628836+0000","last_change":"2022-07-20T02:25:29.458394+0000","last_active":"2022-07-20T02:25:35.628836+0000","last_peered":"2022-07-20T02:25:35.628836+0000","last_clean":"2022-07-20T02:25:35.628836+0000","last_became_active":"2022-07-20T02:25:29.458000+0000","last_became_peered":"2022-07-20T02:25:29.458000+0000","last_unstale":"2022-07-20T02:25:35.628836+0000","last_undegraded":"2022-07-20T02:25:35.628836+0000","last_fullsized":"2022-07-20T02:25:35.628836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":102140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":101565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":101599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":102300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":103100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":103100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":101599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]}]}} 2022-07-20T02:25:46.591 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-07-20T02:25:46.591 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-07-20T02:25:46.592 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-07-20T02:25:46.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-07-20T02:25:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 Processor -- start 2022-07-20T02:25:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 -- start start 2022-07-20T02:25:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 0x7f4abc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4abc0a7fc0 0x7f4abc1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0b62d0 con 0x7f4abc0a8c40 2022-07-20T02:25:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc147eb0 con 0x7f4abc0a7fc0 2022-07-20T02:25:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc14aac0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1642700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4abc0a8c40 0x7f4abc0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34652/0 (socket says 172.21.15.26:34652) 2022-07-20T02:25:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1642700 1 -- 172.21.15.26:0/3769440089 learned_addr learned my addr 172.21.15.26:0/3769440089 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3769440089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 0x7f4abc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab3fff700 1 --2- 172.21.15.26:0/3769440089 >> v2:172.21.15.120:3300/0 conn(0x7f4abc0a7fc0 0x7f4abc1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab37fe700 1 -- 172.21.15.26:0/3769440089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393444825 0 0) 0x7f4abc0b62d0 con 0x7f4abc0a8c40 2022-07-20T02:25:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab37fe700 1 -- 172.21.15.26:0/3769440089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa8003260 con 0x7f4abc0a8c40 2022-07-20T02:25:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 -- 172.21.15.26:0/3769440089 >> v2:172.21.15.120:3300/0 conn(0x7f4abc0a7fc0 msgr2=0x7f4abc1479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3769440089 >> v2:172.21.15.120:3300/0 conn(0x7f4abc0a7fc0 0x7f4abc1479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 -- 172.21.15.26:0/3769440089 >> v1:172.21.15.26:6789/0 conn(0x7f4abc0a8c40 legacy=0x7f4abc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 -- 172.21.15.26:0/3769440089 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc14ad60 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3769440089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 0x7f4abc1474d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4ab8002f70 tx=0x7f4ab8006df0).ready entity=mon.2 client_cookie=f14105530288cc98 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab37fe700 1 -- 172.21.15.26:0/3769440089 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab800d4a0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab37fe700 1 -- 172.21.15.26:0/3769440089 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab8000e90 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ab37fe700 1 -- 172.21.15.26:0/3769440089 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab800d7f0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3769440089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 msgr2=0x7f4abc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.699+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3769440089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 0x7f4abc1474d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4ab8002f70 tx=0x7f4ab8006df0).stop 2022-07-20T02:25:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3769440089 shutdown_connections 2022-07-20T02:25:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3769440089 >> v2:172.21.15.120:3300/0 conn(0x7f4abc0a7fc0 0x7f4abc1479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3769440089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0b5ef0 0x7f4abc1474d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3769440089 >> 172.21.15.26:0/3769440089 conn(0x7f4abc060310 msgr2=0x7f4abc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3769440089 shutdown_connections 2022-07-20T02:25:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3769440089 wait complete. 2022-07-20T02:25:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 Processor -- start 2022-07-20T02:25:46.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.703+0000 7f4ac1e43700 1 -- start start 2022-07-20T02:25:46.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0a8c40 0x7f4abc1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 0x7f4abc146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc064160 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc1470c0 con 0x7f4abc0a7fc0 2022-07-20T02:25:46.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0abd90 con 0x7f4abc0a8c40 2022-07-20T02:25:46.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 0x7f4abc146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac0e41700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac0e41700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:55066/0 (socket says 172.21.15.26:55066) 2022-07-20T02:25:46.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac0e41700 1 -- 172.21.15.26:0/3504824701 learned_addr learned my addr 172.21.15.26:0/3504824701 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:46.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ab3fff700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0a8c40 0x7f4abc1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 -- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0a8c40 msgr2=0x7f4abc1466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0a8c40 0x7f4abc1466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 -- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 msgr2=0x7f4abc062a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 -- 172.21.15.26:0/3504824701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab800b180 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:46.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1642700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 0x7f4abc146bd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4abc063240 tx=0x7f4ab4007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4005c70 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab4005dd0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc064350 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab40072c0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.707+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc0a9460 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc0647f0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ab4010070 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ab1ffb700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:46.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4ab406ba40 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:46.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.711+0000 7f4ac0e41700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4ab801a000 tx=0x7f4ab8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:46.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.715+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4abc0647f0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4abc0647f0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ab1ffb700 1 -- 172.21.15.26:0/3504824701 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f4abc0647f0 con 0x7f4abc0b5ef0 2022-07-20T02:25:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 msgr2=0x7f4a9c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4ab801a000 tx=0x7f4ab8014000).stop 2022-07-20T02:25:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 msgr2=0x7f4abc146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 0x7f4abc146bd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4abc063240 tx=0x7f4ab4007dc0).stop 2022-07-20T02:25:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 shutdown_connections 2022-07-20T02:25:46.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4abc0b5ef0 0x7f4abc146bd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4abc0a8c40 0x7f4abc1466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 --2- 172.21.15.26:0/3504824701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4abc0a7fc0 0x7f4abc062a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:46.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 >> 172.21.15.26:0/3504824701 conn(0x7f4abc060310 msgr2=0x7f4abc149d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:46.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 shutdown_connections 2022-07-20T02:25:46.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:46.887+0000 7f4ac1e43700 1 -- 172.21.15.26:0/3504824701 wait complete. 2022-07-20T02:25:46.903 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-20T02:25:46.904 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-07-20T02:25:46.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 Processor -- start 2022-07-20T02:25:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 -- start start 2022-07-20T02:25:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d40add30 0x7f29d40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29d4060950 0x7f29d4060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29d4061280 0x7f29d4143450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40aa1e0 con 0x7f29d4060950 2022-07-20T02:25:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40ae110 con 0x7f29d4061280 2022-07-20T02:25:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40a9f40 con 0x7f29d40add30 2022-07-20T02:25:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29cbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29d4060950 0x7f29d4060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29cbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29d4060950 0x7f29d4060d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35382/0 (socket says 172.21.15.26:35382) 2022-07-20T02:25:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29cbfff700 1 -- 172.21.15.26:0/3909815333 learned_addr learned my addr 172.21.15.26:0/3909815333 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.003+0000 7f29d2d9d700 1 --2- 172.21.15.26:0/3909815333 >> v2:172.21.15.120:3300/0 conn(0x7f29d4061280 0x7f29d4143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d259c700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d40add30 0x7f29d40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 -- 172.21.15.26:0/3909815333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d40add30 msgr2=0x7f29d40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d40add30 0x7f29d40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 -- 172.21.15.26:0/3909815333 >> v2:172.21.15.120:3300/0 conn(0x7f29d4061280 msgr2=0x7f29d4143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 --2- 172.21.15.26:0/3909815333 >> v2:172.21.15.120:3300/0 conn(0x7f29d4061280 0x7f29d4143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 -- 172.21.15.26:0/3909815333 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40b1110 con 0x7f29d4060950 2022-07-20T02:25:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29cbfff700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4060950 0x7f29d4060d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f29bc002f70 tx=0x7f29bc006f40).ready entity=mon.0 client_cookie=e719b28873cd3425 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d1d9b700 1 -- 172.21.15.26:0/3909815333 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc00d9a0 con 0x7f29d4060950 2022-07-20T02:25:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d1d9b700 1 -- 172.21.15.26:0/3909815333 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc000e90 con 0x7f29d4060950 2022-07-20T02:25:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d1d9b700 1 -- 172.21.15.26:0/3909815333 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc00dd30 con 0x7f29d4060950 2022-07-20T02:25:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 -- 172.21.15.26:0/3909815333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4060950 msgr2=0x7f29d4060d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4060950 0x7f29d4060d90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f29bc002f70 tx=0x7f29bc006f40).stop 2022-07-20T02:25:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 -- 172.21.15.26:0/3909815333 shutdown_connections 2022-07-20T02:25:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 --2- 172.21.15.26:0/3909815333 >> v2:172.21.15.120:3300/0 conn(0x7f29d4061280 0x7f29d4143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4060950 0x7f29d4060d90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 --2- 172.21.15.26:0/3909815333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d40add30 0x7f29d40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 -- 172.21.15.26:0/3909815333 >> 172.21.15.26:0/3909815333 conn(0x7f29d40b3050 msgr2=0x7f29d40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 -- 172.21.15.26:0/3909815333 shutdown_connections 2022-07-20T02:25:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.007+0000 7f29d359e700 1 -- 172.21.15.26:0/3909815333 wait complete. 2022-07-20T02:25:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 Processor -- start 2022-07-20T02:25:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 -- start start 2022-07-20T02:25:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 0x7f29d413e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 0x7f29d413f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29d40add30 0x7f29d413f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d413d8c0 con 0x7f29d4061280 2022-07-20T02:25:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d41406b0 con 0x7f29d40add30 2022-07-20T02:25:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d41408e0 con 0x7f29d4060950 2022-07-20T02:25:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 0x7f29d413f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 0x7f29d413e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d2d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29d40add30 0x7f29d413f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 0x7f29d413e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:56904/0 (socket says 172.21.15.26:56904) 2022-07-20T02:25:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29d259c700 1 -- 172.21.15.26:0/697633640 learned_addr learned my addr 172.21.15.26:0/697633640 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 -- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 msgr2=0x7f29d413e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 0x7f29d413e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 -- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29d40add30 msgr2=0x7f29d413f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29d40add30 0x7f29d413f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 -- 172.21.15.26:0/697633640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc00b460 con 0x7f29d4061280 2022-07-20T02:25:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cbfff700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 0x7f29d413f380 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29d413cdc0 tx=0x7f29d40671e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc024450 con 0x7f29d4061280 2022-07-20T02:25:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.011+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc0245b0 con 0x7f29d4061280 2022-07-20T02:25:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d4140b80 con 0x7f29d4061280 2022-07-20T02:25:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc024910 con 0x7f29d4061280 2022-07-20T02:25:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d413eee0 con 0x7f29d4061280 2022-07-20T02:25:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d40b13b0 con 0x7f29d4061280 2022-07-20T02:25:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29bc021070 con 0x7f29d4061280 2022-07-20T02:25:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29cb7fe700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 0x7f29b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f29bc06f880 con 0x7f29d4061280 2022-07-20T02:25:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29d259c700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 0x7f29b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.015+0000 7f29d259c700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 0x7f29b8055ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29c400b000 tx=0x7f29c4006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.023+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29bc0466b0 con 0x7f29d4061280 2022-07-20T02:25:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.143+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29d40574e0 con 0x7f29d4061280 2022-07-20T02:25:47.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29cb7fe700 1 -- 172.21.15.26:0/697633640 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f29bc049cd0 con 0x7f29d4061280 2022-07-20T02:25:47.152 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:47.152 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 msgr2=0x7f29b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 0x7f29b8055ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29c400b000 tx=0x7f29c4006df0).stop 2022-07-20T02:25:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 msgr2=0x7f29d413f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 0x7f29d413f380 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29d413cdc0 tx=0x7f29d40671e0).stop 2022-07-20T02:25:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 shutdown_connections 2022-07-20T02:25:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29d4061280 0x7f29d413f380 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29b80539e0 0x7f29b8055ea0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29d40add30 0x7f29d413f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 --2- 172.21.15.26:0/697633640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29d4060950 0x7f29d413e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 >> 172.21.15.26:0/697633640 conn(0x7f29d40b3050 msgr2=0x7f29d40b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 shutdown_connections 2022-07-20T02:25:47.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.147+0000 7f29d359e700 1 -- 172.21.15.26:0/697633640 wait complete. 2022-07-20T02:25:47.163 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:47.164 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:47.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 Processor -- start 2022-07-20T02:25:47.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 -- start start 2022-07-20T02:25:47.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4304060c00 0x7f43041445b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43040b3640 con 0x7f4304097c50 2022-07-20T02:25:47.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4304095300 con 0x7f43040b3260 2022-07-20T02:25:47.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4303fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43040955a0 con 0x7f4304060c00 2022-07-20T02:25:47.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43027fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4304060c00 0x7f43041445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43037fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4304097c50 0x7f4304144aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34680/0 (socket says 172.21.15.26:34680) 2022-07-20T02:25:47.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43037fe700 1 -- 172.21.15.26:0/2964212558 learned_addr learned my addr 172.21.15.26:0/2964212558 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:47.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048684018 0 0) 0x7f4304095300 con 0x7f43040b3260 2022-07-20T02:25:47.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e4003260 con 0x7f43040b3260 2022-07-20T02:25:47.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43027fc700 1 -- 172.21.15.26:0/2964212558 >> v1:172.21.15.120:6789/0 conn(0x7f43040b3260 legacy=0x7f43040604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499528046 0 0) 0x7f43040b3640 con 0x7f4304097c50 2022-07-20T02:25:47.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43027fc700 1 -- 172.21.15.26:0/2964212558 >> v1:172.21.15.26:6789/0 conn(0x7f4304097c50 legacy=0x7f4304144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43027fc700 1 -- 172.21.15.26:0/2964212558 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304095840 con 0x7f4304060c00 2022-07-20T02:25:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.275+0000 7f43027fc700 1 --2- 172.21.15.26:0/2964212558 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304060c00 0x7f43041445b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f42ec002f70 tx=0x7f42ec006df0).ready entity=mon.0 client_cookie=5e954cff3d9067f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42ec00d610 con 0x7f4304060c00 2022-07-20T02:25:47.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec000e90 con 0x7f4304060c00 2022-07-20T02:25:47.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4301ffb700 1 -- 172.21.15.26:0/2964212558 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42ec00d960 con 0x7f4304060c00 2022-07-20T02:25:47.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- 172.21.15.26:0/2964212558 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304060c00 msgr2=0x7f43041445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 --2- 172.21.15.26:0/2964212558 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304060c00 0x7f43041445b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f42ec002f70 tx=0x7f42ec006df0).stop 2022-07-20T02:25:47.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- 172.21.15.26:0/2964212558 shutdown_connections 2022-07-20T02:25:47.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 --2- 172.21.15.26:0/2964212558 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304060c00 0x7f43041445b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- 172.21.15.26:0/2964212558 >> 172.21.15.26:0/2964212558 conn(0x7f43040b67f0 msgr2=0x7f43040b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- 172.21.15.26:0/2964212558 shutdown_connections 2022-07-20T02:25:47.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- 172.21.15.26:0/2964212558 wait complete. 2022-07-20T02:25:47.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 Processor -- start 2022-07-20T02:25:47.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.279+0000 7f4303fff700 1 -- start start 2022-07-20T02:25:47.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304060c00 0x7f43040aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43040b3260 0x7f43040abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304151190 con 0x7f4304097c50 2022-07-20T02:25:47.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040ade30 con 0x7f43040b3260 2022-07-20T02:25:47.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43040ae060 con 0x7f4304060c00 2022-07-20T02:25:47.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35404/0 (socket says 172.21.15.26:35404) 2022-07-20T02:25:47.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 -- 172.21.15.26:0/3453312166 learned_addr learned my addr 172.21.15.26:0/3453312166 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4302ffd700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304060c00 0x7f43040aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43037fe700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43040b3260 0x7f43040abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 -- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304060c00 msgr2=0x7f43040aa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304060c00 0x7f43040aa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 -- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43040b3260 msgr2=0x7f43040abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43040b3260 0x7f43040abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 -- 172.21.15.26:0/3453312166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec00b180 con 0x7f4304097c50 2022-07-20T02:25:47.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f43027fc700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f42ec002f70 tx=0x7f42ec014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42ec016070 con 0x7f4304097c50 2022-07-20T02:25:47.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec00de90 con 0x7f4304097c50 2022-07-20T02:25:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43040ae300 con 0x7f4304097c50 2022-07-20T02:25:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42ec013d60 con 0x7f4304097c50 2022-07-20T02:25:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43040ae840 con 0x7f4304097c50 2022-07-20T02:25:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.283+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43040aeb30 con 0x7f4304097c50 2022-07-20T02:25:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.287+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f42ec025030 con 0x7f4304097c50 2022-07-20T02:25:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.287+0000 7f42fbfff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 0x7f42e8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.287+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f42ec06d030 con 0x7f4304097c50 2022-07-20T02:25:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.287+0000 7f4302ffd700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 0x7f42e8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.287+0000 7f4302ffd700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 0x7f42e8055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f42f400d000 tx=0x7f42f4007da0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.295+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42ec043df0 con 0x7f4304097c50 2022-07-20T02:25:47.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.419+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f43040993f0 con 0x7f4304097c50 2022-07-20T02:25:47.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f42fbfff700 1 -- 172.21.15.26:0/3453312166 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f42ec023070 con 0x7f4304097c50 2022-07-20T02:25:47.425 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:47.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 msgr2=0x7f42e8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 0x7f42e8055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f42f400d000 tx=0x7f42f4007da0).stop 2022-07-20T02:25:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 msgr2=0x7f4304150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f42ec002f70 tx=0x7f42ec014000).stop 2022-07-20T02:25:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 shutdown_connections 2022-07-20T02:25:47.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4304097c50 0x7f4304150ca0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42e8053b00 0x7f42e8055fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43040b3260 0x7f43040abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 --2- 172.21.15.26:0/3453312166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4304060c00 0x7f43040aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 >> 172.21.15.26:0/3453312166 conn(0x7f43040b67f0 msgr2=0x7f430409abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 shutdown_connections 2022-07-20T02:25:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.423+0000 7f4303fff700 1 -- 172.21.15.26:0/3453312166 wait complete. 2022-07-20T02:25:47.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:47.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 Processor -- start 2022-07-20T02:25:47.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- start start 2022-07-20T02:25:47.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d8c0a7620 0x7f5d8c066850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0b2ea0 con 0x7f5d8c0b2ac0 2022-07-20T02:25:47.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c147900 con 0x7f5d8c148580 2022-07-20T02:25:47.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8c147ba0 con 0x7f5d8c0a7620 2022-07-20T02:25:47.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d8c0a7620 0x7f5d8c066850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d8c0a7620 0x7f5d8c066850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35410/0 (socket says 172.21.15.26:35410) 2022-07-20T02:25:47.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 -- 172.21.15.26:0/1793460087 learned_addr learned my addr 172.21.15.26:0/1793460087 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8b7fe700 1 --1- 172.21.15.26:0/1793460087 >> v1:172.21.15.26:6789/0 conn(0x7f5d8c0b2ac0 0x7f5d8c0aff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34690/0 (socket says 172.21.15.26:34690) 2022-07-20T02:25:47.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 -- 172.21.15.26:0/1793460087 >> v1:172.21.15.120:6789/0 conn(0x7f5d8c148580 legacy=0x7f5d8c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 -- 172.21.15.26:0/1793460087 >> v1:172.21.15.26:6789/0 conn(0x7f5d8c0b2ac0 legacy=0x7f5d8c0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 -- 172.21.15.26:0/1793460087 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c147e40 con 0x7f5d8c0a7620 2022-07-20T02:25:47.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8affd700 1 --2- 172.21.15.26:0/1793460087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c066850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5d8c0a88a0 tx=0x7f5d7c006eb0).ready entity=mon.0 client_cookie=2c7f517a03823cc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8a7fc700 1 -- 172.21.15.26:0/1793460087 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7c00d660 con 0x7f5d8c0a7620 2022-07-20T02:25:47.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8a7fc700 1 -- 172.21.15.26:0/1793460087 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7c000e90 con 0x7f5d8c0a7620 2022-07-20T02:25:47.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d8a7fc700 1 -- 172.21.15.26:0/1793460087 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7c00d9f0 con 0x7f5d8c0a7620 2022-07-20T02:25:47.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1793460087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c0a7620 msgr2=0x7f5d8c066850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1793460087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c066850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5d8c0a88a0 tx=0x7f5d7c006eb0).stop 2022-07-20T02:25:47.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1793460087 shutdown_connections 2022-07-20T02:25:47.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1793460087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c066850 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1793460087 >> 172.21.15.26:0/1793460087 conn(0x7f5d8c0b60a0 msgr2=0x7f5d8c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1793460087 shutdown_connections 2022-07-20T02:25:47.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.543+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1793460087 wait complete. 2022-07-20T02:25:47.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 Processor -- start 2022-07-20T02:25:47.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- start start 2022-07-20T02:25:47.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8c0b2ac0 0x7f5d8c0a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c0a9de0 con 0x7f5d8c148580 2022-07-20T02:25:47.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1576f0 con 0x7f5d8c0a7620 2022-07-20T02:25:47.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c157960 con 0x7f5d8c0b2ac0 2022-07-20T02:25:47.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35420/0 (socket says 172.21.15.26:35420) 2022-07-20T02:25:47.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 -- 172.21.15.26:0/1175720445 learned_addr learned my addr 172.21.15.26:0/1175720445 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8affd700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8c0b2ac0 0x7f5d8c0a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8b7fe700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 -- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8c0b2ac0 msgr2=0x7f5d8c0a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8c0b2ac0 0x7f5d8c0a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 -- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 msgr2=0x7f5d8c0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c0b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 -- 172.21.15.26:0/1175720445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c00b1e0 con 0x7f5d8c148580 2022-07-20T02:25:47.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8b7fe700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:47.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d8bfff700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5d8c157880 tx=0x7f5d84007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d84006aa0 con 0x7f5d8c148580 2022-07-20T02:25:47.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d84006c00 con 0x7f5d8c148580 2022-07-20T02:25:47.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c157c00 con 0x7f5d8c148580 2022-07-20T02:25:47.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d84007030 con 0x7f5d8c148580 2022-07-20T02:25:47.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.547+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c0660a0 con 0x7f5d8c148580 2022-07-20T02:25:47.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c1480e0 con 0x7f5d8c148580 2022-07-20T02:25:47.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d84010030 con 0x7f5d8c148580 2022-07-20T02:25:47.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d88ff9700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 0x7f5d6c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5d8406be30 con 0x7f5d8c148580 2022-07-20T02:25:47.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d8b7fe700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 0x7f5d6c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.551+0000 7f5d8b7fe700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 0x7f5d6c055ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5d8000b000 tx=0x7f5d800070e0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.555+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d84042bf0 con 0x7f5d8c148580 2022-07-20T02:25:47.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.683+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5d8c0574e0 con 0x7f5d8c148580 2022-07-20T02:25:47.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.683+0000 7f5d88ff9700 1 -- 172.21.15.26:0/1175720445 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5d84046210 con 0x7f5d8c148580 2022-07-20T02:25:47.686 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:47.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 msgr2=0x7f5d6c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 0x7f5d6c055ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5d8000b000 tx=0x7f5d800070e0).stop 2022-07-20T02:25:47.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 msgr2=0x7f5d8c0a98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5d8c157880 tx=0x7f5d84007f70).stop 2022-07-20T02:25:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 shutdown_connections 2022-07-20T02:25:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d8c148580 0x7f5d8c0a98f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d6c053a10 0x7f5d6c055ed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d8c0b2ac0 0x7f5d8c0a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 --2- 172.21.15.26:0/1175720445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d8c0a7620 0x7f5d8c0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 >> 172.21.15.26:0/1175720445 conn(0x7f5d8c0b60a0 msgr2=0x7f5d8c148c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 shutdown_connections 2022-07-20T02:25:47.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.687+0000 7f5d91ed2700 1 -- 172.21.15.26:0/1175720445 wait complete. 2022-07-20T02:25:47.705 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:47.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 Processor -- start 2022-07-20T02:25:47.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 -- start start 2022-07-20T02:25:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 0x7fc9000a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b2f10 con 0x7fc9000b2b30 2022-07-20T02:25:47.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000a7500 con 0x7fc900060f10 2022-07-20T02:25:47.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc907682700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000a8e30 con 0x7fc900147d70 2022-07-20T02:25:47.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc90541e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc9000b2b30 0x7fc900147660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34704/0 (socket says 172.21.15.26:34704) 2022-07-20T02:25:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc90541e700 1 -- 172.21.15.26:0/1152781414 learned_addr learned my addr 172.21.15.26:0/1152781414 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc904c1d700 1 --2- 172.21.15.26:0/1152781414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 0x7fc9000a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596109319 0 0) 0x7fc9000b2f10 con 0x7fc9000b2b30 2022-07-20T02:25:47.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4003260 con 0x7fc9000b2b30 2022-07-20T02:25:47.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241465352 0 0) 0x7fc9000a7500 con 0x7fc900060f10 2022-07-20T02:25:47.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9000b2f10 con 0x7fc900060f10 2022-07-20T02:25:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc904c1d700 1 -- 172.21.15.26:0/1152781414 >> v1:172.21.15.120:6789/0 conn(0x7fc900060f10 legacy=0x7fc900066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc904c1d700 1 -- 172.21.15.26:0/1152781414 >> v1:172.21.15.26:6789/0 conn(0x7fc9000b2b30 legacy=0x7fc900147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc904c1d700 1 -- 172.21.15.26:0/1152781414 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000a77a0 con 0x7fc900147d70 2022-07-20T02:25:47.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.815+0000 7fc904c1d700 1 --2- 172.21.15.26:0/1152781414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 0x7fc9000a7ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc8f0002f70 tx=0x7fc8f0006df0).ready entity=mon.2 client_cookie=654d8907f58a6478 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f000d5c0 con 0x7fc900147d70 2022-07-20T02:25:47.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0000e90 con 0x7fc900147d70 2022-07-20T02:25:47.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc8f7fff700 1 -- 172.21.15.26:0/1152781414 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f000d910 con 0x7fc900147d70 2022-07-20T02:25:47.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- 172.21.15.26:0/1152781414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 msgr2=0x7fc9000a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 --2- 172.21.15.26:0/1152781414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 0x7fc9000a7ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc8f0002f70 tx=0x7fc8f0006df0).stop 2022-07-20T02:25:47.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- 172.21.15.26:0/1152781414 shutdown_connections 2022-07-20T02:25:47.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 --2- 172.21.15.26:0/1152781414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900147d70 0x7fc9000a7ee0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- 172.21.15.26:0/1152781414 >> 172.21.15.26:0/1152781414 conn(0x7fc9000b6110 msgr2=0x7fc9000b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- 172.21.15.26:0/1152781414 shutdown_connections 2022-07-20T02:25:47.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- 172.21.15.26:0/1152781414 wait complete. 2022-07-20T02:25:47.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 Processor -- start 2022-07-20T02:25:47.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.819+0000 7fc907682700 1 -- start start 2022-07-20T02:25:47.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900060f10 0x7fc9000adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 0x7fc900159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000ae3f0 con 0x7fc9000b2b30 2022-07-20T02:25:47.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000ad060 con 0x7fc900147d70 2022-07-20T02:25:47.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000ae150 con 0x7fc900060f10 2022-07-20T02:25:47.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35434/0 (socket says 172.21.15.26:35434) 2022-07-20T02:25:47.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 -- 172.21.15.26:0/980710119 learned_addr learned my addr 172.21.15.26:0/980710119 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:47.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc90541e700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900060f10 0x7fc9000adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc905c1f700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 0x7fc900159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 -- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900060f10 msgr2=0x7fc9000adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900060f10 0x7fc9000adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 -- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 msgr2=0x7fc900159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 0x7fc900159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 -- 172.21.15.26:0/980710119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f000b180 con 0x7fc9000b2b30 2022-07-20T02:25:47.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc905c1f700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 0x7fc900159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:47.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc904c1d700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc8f0002f70 tx=0x7fc8f0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f0013de0 con 0x7fc9000b2b30 2022-07-20T02:25:47.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f001e400 con 0x7fc9000b2b30 2022-07-20T02:25:47.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f001e760 con 0x7fc9000b2b30 2022-07-20T02:25:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90014ba80 con 0x7fc9000b2b30 2022-07-20T02:25:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc900062810 con 0x7fc9000b2b30 2022-07-20T02:25:47.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.823+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9000adf90 con 0x7fc9000b2b30 2022-07-20T02:25:47.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.827+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc8f001c070 con 0x7fc9000b2b30 2022-07-20T02:25:47.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.827+0000 7fc8f67fc700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 0x7fc8e8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:47.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.827+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc8f006ca80 con 0x7fc9000b2b30 2022-07-20T02:25:47.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.827+0000 7fc90541e700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 0x7fc8e8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:47.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.827+0000 7fc90541e700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 0x7fc8e8055fc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc8fc00d000 tx=0x7fc8fc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:47.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.835+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc8f0043730 con 0x7fc9000b2b30 2022-07-20T02:25:47.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9000ad220 con 0x7fc9000b2b30 2022-07-20T02:25:47.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc8f67fc700 1 -- 172.21.15.26:0/980710119 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc8f0023020 con 0x7fc9000b2b30 2022-07-20T02:25:47.967 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:47.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 msgr2=0x7fc8e8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 0x7fc8e8055fc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc8fc00d000 tx=0x7fc8fc009040).stop 2022-07-20T02:25:47.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 msgr2=0x7fc90014b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:47.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc8f0002f70 tx=0x7fc8f0014000).stop 2022-07-20T02:25:47.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 shutdown_connections 2022-07-20T02:25:47.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9000b2b30 0x7fc90014b590 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8e8053b00 0x7fc8e8055fc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc900147d70 0x7fc900159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 --2- 172.21.15.26:0/980710119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc900060f10 0x7fc9000adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:47.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 >> 172.21.15.26:0/980710119 conn(0x7fc9000b6110 msgr2=0x7fc900147660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:47.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 shutdown_connections 2022-07-20T02:25:47.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:47.963+0000 7fc907682700 1 -- 172.21.15.26:0/980710119 wait complete. 2022-07-20T02:25:47.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:48.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 Processor -- start 2022-07-20T02:25:48.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- start start 2022-07-20T02:25:48.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780afe40 0x7f0a780abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0a780b1200 0x7f0a780ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7805d4d0 con 0x7f0a7805c720 2022-07-20T02:25:48.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a780a9ff0 con 0x7f0a780b1200 2022-07-20T02:25:48.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78067e60 con 0x7f0a780afe40 2022-07-20T02:25:48.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0a780b1200 0x7f0a780ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7ca65700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780afe40 0x7f0a780abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7d266700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0a7805c720 0x7f0a78147550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:48930/0 (socket says 172.21.15.26:48930) 2022-07-20T02:25:48.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7d266700 1 -- 172.21.15.26:0/1638417039 learned_addr learned my addr 172.21.15.26:0/1638417039 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:48.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 -- 172.21.15.26:0/1638417039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780afe40 msgr2=0x7f0a780abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:48.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 --2- 172.21.15.26:0/1638417039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780afe40 0x7f0a780abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 -- 172.21.15.26:0/1638417039 >> v1:172.21.15.120:6789/0 conn(0x7f0a7805c720 legacy=0x7f0a78147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 -- 172.21.15.26:0/1638417039 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78067bc0 con 0x7f0a780b1200 2022-07-20T02:25:48.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7da67700 1 --2- 172.21.15.26:0/1638417039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a780b1200 0x7f0a780ac0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0a700032f0 tx=0x7f0a700091d0).ready entity=mon.0 client_cookie=ab8526b5374e8c72 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a6ffff700 1 -- 172.21.15.26:0/1638417039 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a7000b6c0 con 0x7f0a780b1200 2022-07-20T02:25:48.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a6ffff700 1 -- 172.21.15.26:0/1638417039 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7000b820 con 0x7f0a780b1200 2022-07-20T02:25:48.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a6ffff700 1 -- 172.21.15.26:0/1638417039 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a7000b990 con 0x7f0a780b1200 2022-07-20T02:25:48.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- 172.21.15.26:0/1638417039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a780b1200 msgr2=0x7f0a780ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 --2- 172.21.15.26:0/1638417039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a780b1200 0x7f0a780ac0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0a700032f0 tx=0x7f0a700091d0).stop 2022-07-20T02:25:48.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- 172.21.15.26:0/1638417039 shutdown_connections 2022-07-20T02:25:48.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 --2- 172.21.15.26:0/1638417039 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a780b1200 0x7f0a780ac0e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 --2- 172.21.15.26:0/1638417039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780afe40 0x7f0a780abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- 172.21.15.26:0/1638417039 >> 172.21.15.26:0/1638417039 conn(0x7f0a78060310 msgr2=0x7f0a780b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:48.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- 172.21.15.26:0/1638417039 shutdown_connections 2022-07-20T02:25:48.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.091+0000 7f0a7e268700 1 -- 172.21.15.26:0/1638417039 wait complete. 2022-07-20T02:25:48.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 Processor -- start 2022-07-20T02:25:48.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- start start 2022-07-20T02:25:48.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a780afe40 0x7f0a780adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 0x7f0a780ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781568e0 con 0x7f0a7805c720 2022-07-20T02:25:48.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780acf10 con 0x7f0a780afe40 2022-07-20T02:25:48.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780ac9e0 con 0x7f0a780b1200 2022-07-20T02:25:48.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35444/0 (socket says 172.21.15.26:35444) 2022-07-20T02:25:48.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 -- 172.21.15.26:0/712468028 learned_addr learned my addr 172.21.15.26:0/712468028 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:48.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7ca65700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a780afe40 0x7f0a780adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7da67700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 0x7f0a780ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 -- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 msgr2=0x7f0a780ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 0x7f0a780ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 -- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a780afe40 msgr2=0x7f0a780adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a780afe40 0x7f0a780adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 -- 172.21.15.26:0/712468028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a70009470 con 0x7f0a7805c720 2022-07-20T02:25:48.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7da67700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 0x7f0a780ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:25:48.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7d266700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0a780acb70 tx=0x7f0a68007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6800ce20 con 0x7f0a7805c720 2022-07-20T02:25:48.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a680072e0 con 0x7f0a7805c720 2022-07-20T02:25:48.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a780ac740 con 0x7f0a7805c720 2022-07-20T02:25:48.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a68005800 con 0x7f0a7805c720 2022-07-20T02:25:48.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.095+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a780ae520 con 0x7f0a7805c720 2022-07-20T02:25:48.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a780aa250 con 0x7f0a7805c720 2022-07-20T02:25:48.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0a68010070 con 0x7f0a7805c720 2022-07-20T02:25:48.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a6e7fc700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 0x7f0a60055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0a6806b630 con 0x7f0a7805c720 2022-07-20T02:25:48.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a7ca65700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 0x7f0a60055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.099+0000 7f0a7ca65700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 0x7f0a60055fc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0a7400b000 tx=0x7f0a74008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.103+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0a680423f0 con 0x7f0a7805c720 2022-07-20T02:25:48.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a780574e0 con 0x7f0a7805c720 2022-07-20T02:25:48.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a6e7fc700 1 -- 172.21.15.26:0/712468028 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0a68045a10 con 0x7f0a7805c720 2022-07-20T02:25:48.232 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 msgr2=0x7f0a60055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 0x7f0a60055fc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0a7400b000 tx=0x7f0a74008040).stop 2022-07-20T02:25:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 msgr2=0x7f0a780b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.227+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0a780acb70 tx=0x7f0a68007ca0).stop 2022-07-20T02:25:48.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 shutdown_connections 2022-07-20T02:25:48.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a60053b00 0x7f0a60055fc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0a780b1200 0x7f0a780ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a780afe40 0x7f0a780adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 --2- 172.21.15.26:0/712468028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a7805c720 0x7f0a780b24c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 >> 172.21.15.26:0/712468028 conn(0x7f0a78060310 msgr2=0x7f0a780ab100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:48.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 shutdown_connections 2022-07-20T02:25:48.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.231+0000 7f0a7e268700 1 -- 172.21.15.26:0/712468028 wait complete. 2022-07-20T02:25:48.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:25:48.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.351+0000 7f3c9d9a9700 1 Processor -- start 2022-07-20T02:25:48.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.351+0000 7f3c9d9a9700 1 -- start start 2022-07-20T02:25:48.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980670c0 0x7f3c980b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c980b3380 0x7f3c9814b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c98064880 con 0x7f3c98064450 2022-07-20T02:25:48.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980b85f0 con 0x7f3c980b3380 2022-07-20T02:25:48.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b80b0 con 0x7f3c980670c0 2022-07-20T02:25:48.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c980b3380 0x7f3c9814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c980b3380 0x7f3c9814b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35456/0 (socket says 172.21.15.26:35456) 2022-07-20T02:25:48.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 -- 172.21.15.26:0/3980992991 learned_addr learned my addr 172.21.15.26:0/3980992991 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:48.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c8ffff700 1 --1- 172.21.15.26:0/3980992991 >> v1:172.21.15.26:6789/0 conn(0x7f3c98064450 0x7f3c98064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34732/0 (socket says 172.21.15.26:34732) 2022-07-20T02:25:48.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9c9a7700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980670c0 0x7f3c980b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 -- 172.21.15.26:0/3980992991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980670c0 msgr2=0x7f3c980b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:48.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980670c0 0x7f3c980b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 -- 172.21.15.26:0/3980992991 >> v1:172.21.15.26:6789/0 conn(0x7f3c98064450 legacy=0x7f3c98064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 -- 172.21.15.26:0/3980992991 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980b8350 con 0x7f3c980b3380 2022-07-20T02:25:48.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c8f7fe700 1 -- 172.21.15.26:0/3980992991 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279550932 0 0) 0x7f3c98064880 con 0x7f3c98064450 2022-07-20T02:25:48.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d1a8700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980b3380 0x7f3c9814b840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3c900032f0 tx=0x7f3c90009290).ready entity=mon.0 client_cookie=e808bf640c7b3f0d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c8f7fe700 1 -- 172.21.15.26:0/3980992991 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c9000b8a0 con 0x7f3c980b3380 2022-07-20T02:25:48.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c8f7fe700 1 -- 172.21.15.26:0/3980992991 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9000ba00 con 0x7f3c980b3380 2022-07-20T02:25:48.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c8f7fe700 1 -- 172.21.15.26:0/3980992991 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c9000bb70 con 0x7f3c980b3380 2022-07-20T02:25:48.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3980992991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980b3380 msgr2=0x7f3c9814b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980b3380 0x7f3c9814b840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3c900032f0 tx=0x7f3c90009290).stop 2022-07-20T02:25:48.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3980992991 shutdown_connections 2022-07-20T02:25:48.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980b3380 0x7f3c9814b840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3980992991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980670c0 0x7f3c980b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3980992991 >> 172.21.15.26:0/3980992991 conn(0x7f3c9805f840 msgr2=0x7f3c9805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:48.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3980992991 shutdown_connections 2022-07-20T02:25:48.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3980992991 wait complete. 2022-07-20T02:25:48.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 Processor -- start 2022-07-20T02:25:48.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.355+0000 7f3c9d9a9700 1 -- start start 2022-07-20T02:25:48.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980afb90 0x7f3c98140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c980ab940 0x7f3c980acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b0ed0 con 0x7f3c980670c0 2022-07-20T02:25:48.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980ab510 con 0x7f3c980ab940 2022-07-20T02:25:48.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b1100 con 0x7f3c980afb90 2022-07-20T02:25:48.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35458/0 (socket says 172.21.15.26:35458) 2022-07-20T02:25:48.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 -- 172.21.15.26:0/3384179618 learned_addr learned my addr 172.21.15.26:0/3384179618 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:48.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d1a8700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c980ab940 0x7f3c980acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c8ffff700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980afb90 0x7f3c98140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 -- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980afb90 msgr2=0x7f3c98140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980afb90 0x7f3c98140740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 -- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c980ab940 msgr2=0x7f3c980acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c980ab940 0x7f3c980acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 -- 172.21.15.26:0/3384179618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90009530 con 0x7f3c980670c0 2022-07-20T02:25:48.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9c9a7700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3c9400c000 tx=0x7f3c94006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c94010780 con 0x7f3c980670c0 2022-07-20T02:25:48.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c940108e0 con 0x7f3c980670c0 2022-07-20T02:25:48.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c980b13a0 con 0x7f3c980670c0 2022-07-20T02:25:48.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c94010c40 con 0x7f3c980670c0 2022-07-20T02:25:48.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.359+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c980ad290 con 0x7f3c980670c0 2022-07-20T02:25:48.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c980ab680 con 0x7f3c980670c0 2022-07-20T02:25:48.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3c9401c460 con 0x7f3c980670c0 2022-07-20T02:25:48.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c8dffb700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 0x7f3c78055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:48.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3c9406c430 con 0x7f3c980670c0 2022-07-20T02:25:48.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c8ffff700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 0x7f3c78055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:48.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.363+0000 7f3c8ffff700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 0x7f3c78055e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f3c8800d000 tx=0x7f3c88009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:48.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.371+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c980ab680 con 0x7f3c980670c0 2022-07-20T02:25:48.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c980ab680 con 0x7f3c980670c0 2022-07-20T02:25:48.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c8dffb700 1 -- 172.21.15.26:0/3384179618 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2489 (secure 0 0 0) 0x7f3c980ab680 con 0x7f3c980670c0 2022-07-20T02:25:48.538 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:48.539 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:25:37.669039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/4183261198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4183261198},{"type":"v1","addr":"172.21.15.26:6837","nonce":4183261198}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:25:48.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 msgr2=0x7f3c78055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 0x7f3c78055e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f3c8800d000 tx=0x7f3c88009040).stop 2022-07-20T02:25:48.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 msgr2=0x7f3c980b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:48.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3c9400c000 tx=0x7f3c94006df0).stop 2022-07-20T02:25:48.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 shutdown_connections 2022-07-20T02:25:48.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3c78053950 0x7f3c78055e10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c980ab940 0x7f3c980acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c980afb90 0x7f3c98140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 --2- 172.21.15.26:0/3384179618 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c980670c0 0x7f3c980b02d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:48.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 >> 172.21.15.26:0/3384179618 conn(0x7f3c9805f840 msgr2=0x7f3c9813dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:48.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 shutdown_connections 2022-07-20T02:25:48.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:48.535+0000 7f3c9d9a9700 1 -- 172.21.15.26:0/3384179618 wait complete. 2022-07-20T02:25:48.549 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2022-07-20T02:25:48.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:25:35.946073+0000', 'modified': '2022-07-20T02:25:37.669039+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/4183261198', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 4183261198}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 4183261198}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:25:48.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:25:48.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:25:48.979 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":4266,"want_state":"up:active","state":"up:active","rank_uptime":13.020967031,"mdsmap_epoch":7,"osdmap_epoch":24,"osdmap_epoch_barrier":24,"uptime":19.387823610000002} 2022-07-20T02:25:48.981 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 4266, "mdsmap_epoch": 7, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "rank_uptime": 13.020967031, "state": "up:active", "uptime": 19.38782361, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:25:48.982 INFO:teuthology.run_tasks:Running task ceph-fuse... 2022-07-20T02:25:49.013 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2022-07-20T02:25:49.014 INFO:tasks.ceph_fuse:config is None 2022-07-20T02:25:49.014 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None} 2022-07-20T02:25:49.015 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2022-07-20T02:25:49.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 Processor -- start 2022-07-20T02:25:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- start start 2022-07-20T02:25:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb805cfd0 0x7fbbb8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbbb8147000 0x7fbbb80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb81474e0 con 0x7fbbb8060f00 2022-07-20T02:25:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb805d3b0 con 0x7fbbb8147000 2022-07-20T02:25:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb81431e0 con 0x7fbbb805cfd0 2022-07-20T02:25:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbbb8147000 0x7fbbb80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbbb8147000 0x7fbbb80b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35468/0 (socket says 172.21.15.26:35468) 2022-07-20T02:25:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 -- 172.21.15.26:0/2625810891 learned_addr learned my addr 172.21.15.26:0/2625810891 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb6ffd700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb805cfd0 0x7fbbb8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 -- 172.21.15.26:0/2625810891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb805cfd0 msgr2=0x7fbbb8143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb805cfd0 0x7fbbb8143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 -- 172.21.15.26:0/2625810891 >> v1:172.21.15.120:6789/0 conn(0x7fbbb8060f00 legacy=0x7fbbb813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 -- 172.21.15.26:0/2625810891 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8143480 con 0x7fbbb8147000 2022-07-20T02:25:49.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb67fc700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8147000 0x7fbbb80b4730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fbbb00032f0 tx=0x7fbbb0009290).ready entity=mon.0 client_cookie=43846513c3ff7dfd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb5ffb700 1 -- 172.21.15.26:0/2625810891 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbb000b8a0 con 0x7fbbb8147000 2022-07-20T02:25:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb5ffb700 1 -- 172.21.15.26:0/2625810891 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb000ba00 con 0x7fbbb8147000 2022-07-20T02:25:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb5ffb700 1 -- 172.21.15.26:0/2625810891 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbb000bb70 con 0x7fbbb8147000 2022-07-20T02:25:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- 172.21.15.26:0/2625810891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8147000 msgr2=0x7fbbb80b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8147000 0x7fbbb80b4730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fbbb00032f0 tx=0x7fbbb0009290).stop 2022-07-20T02:25:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- 172.21.15.26:0/2625810891 shutdown_connections 2022-07-20T02:25:49.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8147000 0x7fbbb80b4730 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/2625810891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb805cfd0 0x7fbbb8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.131+0000 7fbbb7fff700 1 -- 172.21.15.26:0/2625810891 >> 172.21.15.26:0/2625810891 conn(0x7fbbb80620c0 msgr2=0x7fbbb805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.135+0000 7fbbb7fff700 1 -- 172.21.15.26:0/2625810891 shutdown_connections 2022-07-20T02:25:49.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.135+0000 7fbbb7fff700 1 -- 172.21.15.26:0/2625810891 wait complete. 2022-07-20T02:25:49.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.135+0000 7fbbb7fff700 1 Processor -- start 2022-07-20T02:25:49.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.135+0000 7fbbb7fff700 1 -- start start 2022-07-20T02:25:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb8147000 0x7fbbb813c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbb8140460 0x7fbbb8138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb813cc70 con 0x7fbbb8060f00 2022-07-20T02:25:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80b90f0 con 0x7fbbb8140460 2022-07-20T02:25:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8156290 con 0x7fbbb8147000 2022-07-20T02:25:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35472/0 (socket says 172.21.15.26:35472) 2022-07-20T02:25:49.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 -- 172.21.15.26:0/60226229 learned_addr learned my addr 172.21.15.26:0/60226229 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:49.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb67fc700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb8147000 0x7fbbb813c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb77fe700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbb8140460 0x7fbbb8138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 -- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb8147000 msgr2=0x7fbbb813c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb8147000 0x7fbbb813c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 -- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbb8140460 msgr2=0x7fbbb8138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbb8140460 0x7fbbb8138100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 -- 172.21.15.26:0/60226229 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0009530 con 0x7fbbb8060f00 2022-07-20T02:25:49.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb6ffd700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fbba000c000 tx=0x7fbba0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbba0010780 con 0x7fbbb8060f00 2022-07-20T02:25:49.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba00108e0 con 0x7fbbb8060f00 2022-07-20T02:25:49.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb80b8590 con 0x7fbbb8060f00 2022-07-20T02:25:49.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbba0010c40 con 0x7fbbb8060f00 2022-07-20T02:25:49.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb80adc30 con 0x7fbbb8060f00 2022-07-20T02:25:49.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb8156450 con 0x7fbbb8060f00 2022-07-20T02:25:49.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.139+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbba001c460 con 0x7fbbb8060f00 2022-07-20T02:25:49.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.143+0000 7fbbaffff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 0x7fbb9c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.143+0000 7fbbb67fc700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 0x7fbb9c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.143+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbba006c6a0 con 0x7fbbb8060f00 2022-07-20T02:25:49.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.143+0000 7fbbb67fc700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 0x7fbb9c055e40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbbb0019000 tx=0x7fbbb0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.147+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbba00432f0 con 0x7fbbb8060f00 2022-07-20T02:25:49.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.271+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbb8143720 con 0x7fbbb8060f00 2022-07-20T02:25:49.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbaffff700 1 -- 172.21.15.26:0/60226229 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7fbba0046910 con 0x7fbbb8060f00 2022-07-20T02:25:49.278 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:49.278 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:49.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 msgr2=0x7fbb9c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 0x7fbb9c055e40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbbb0019000 tx=0x7fbbb0016000).stop 2022-07-20T02:25:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 msgr2=0x7fbbb81421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fbba000c000 tx=0x7fbba0006df0).stop 2022-07-20T02:25:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 shutdown_connections 2022-07-20T02:25:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbb9c053980 0x7fbb9c055e40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbb8140460 0x7fbbb8138100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbb8147000 0x7fbbb813c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 --2- 172.21.15.26:0/60226229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbbb8060f00 0x7fbbb81421d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 >> 172.21.15.26:0/60226229 conn(0x7fbbb805cfd0 msgr2=0x7fbbb813b140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 shutdown_connections 2022-07-20T02:25:49.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.275+0000 7fbbb7fff700 1 -- 172.21.15.26:0/60226229 wait complete. 2022-07-20T02:25:49.292 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:49.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:49.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 Processor -- start 2022-07-20T02:25:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 -- start start 2022-07-20T02:25:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdfcc060950 0x7fdfcc096220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc067dc0 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc0b3640 con 0x7fdfcc0b3260 2022-07-20T02:25:49.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.403+0000 7fdfd0e38700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfcc0615c0 con 0x7fdfcc060950 2022-07-20T02:25:49.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfc3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdfcc0a95a0 0x7fdfcc095b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34762/0 (socket says 172.21.15.26:34762) 2022-07-20T02:25:49.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfc3fff700 1 -- 172.21.15.26:0/885332732 learned_addr learned my addr 172.21.15.26:0/885332732 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:49.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcbfff700 1 --2- 172.21.15.26:0/885332732 >> v2:172.21.15.120:3300/0 conn(0x7fdfcc060950 0x7fdfcc096220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169201246 0 0) 0x7fdfcc067dc0 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfac003260 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457135515 0 0) 0x7fdfcc0b3640 con 0x7fdfcc0b3260 2022-07-20T02:25:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc067dc0 con 0x7fdfcc0b3260 2022-07-20T02:25:49.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcbfff700 1 -- 172.21.15.26:0/885332732 >> v1:172.21.15.120:6789/0 conn(0x7fdfcc0b3260 legacy=0x7fdfcc0676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268074598 0 0) 0x7fdfac003260 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcbfff700 1 -- 172.21.15.26:0/885332732 >> v1:172.21.15.26:6789/0 conn(0x7fdfcc0a95a0 legacy=0x7fdfcc095b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcbfff700 1 -- 172.21.15.26:0/885332732 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc061860 con 0x7fdfcc060950 2022-07-20T02:25:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcbfff700 1 --2- 172.21.15.26:0/885332732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc096220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdfb4001110 tx=0x7fdfc40091d0).ready entity=mon.1 client_cookie=5c8de726de4f2bcf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb4001660 con 0x7fdfcc060950 2022-07-20T02:25:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400f510 con 0x7fdfcc060950 2022-07-20T02:25:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfcaffd700 1 -- 172.21.15.26:0/885332732 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfc400f680 con 0x7fdfcc060950 2022-07-20T02:25:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 -- 172.21.15.26:0/885332732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 msgr2=0x7fdfcc096220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/885332732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc096220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdfb4001110 tx=0x7fdfc40091d0).stop 2022-07-20T02:25:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 -- 172.21.15.26:0/885332732 shutdown_connections 2022-07-20T02:25:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/885332732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc096220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 -- 172.21.15.26:0/885332732 >> 172.21.15.26:0/885332732 conn(0x7fdfcc0b67f0 msgr2=0x7fdfcc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 -- 172.21.15.26:0/885332732 shutdown_connections 2022-07-20T02:25:49.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.407+0000 7fdfd0e38700 1 -- 172.21.15.26:0/885332732 wait complete. 2022-07-20T02:25:49.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 Processor -- start 2022-07-20T02:25:49.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 -- start start 2022-07-20T02:25:49.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfcc0b3260 0x7fdfcc0634b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc145a30 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc066100 con 0x7fdfcc060950 2022-07-20T02:25:49.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfd0e38700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0654a0 con 0x7fdfcc0b3260 2022-07-20T02:25:49.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfc3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfc3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35488/0 (socket says 172.21.15.26:35488) 2022-07-20T02:25:49.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfcb7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfc3fff700 1 -- 172.21.15.26:0/4178687589 learned_addr learned my addr 172.21.15.26:0/4178687589 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:49.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.411+0000 7fdfcbfff700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfcc0b3260 0x7fdfcc0634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 -- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfcc0b3260 msgr2=0x7fdfcc0634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfcc0b3260 0x7fdfcc0634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 -- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 msgr2=0x7fdfcc0af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc0af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 -- 172.21.15.26:0/4178687589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4009470 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc3fff700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fdfb4000fe0 tx=0x7fdfb4001ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb40072e0 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb4007440 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc065740 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb4005800 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc065c80 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc061b00 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdfb4010070 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 0x7fdfb0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdfb406b8b0 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfcb7fe700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 0x7fdfb0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.415+0000 7fdfcb7fe700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 0x7fdfb0055ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdfbc00d000 tx=0x7fdfbc007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.423+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdfb4042670 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.551+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdfcc0574e0 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfc97fa700 1 -- 172.21.15.26:0/4178687589 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdfb4018070 con 0x7fdfcc0a95a0 2022-07-20T02:25:49.558 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:49.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 msgr2=0x7fdfb0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 0x7fdfb0055ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdfbc00d000 tx=0x7fdfbc007ad0).stop 2022-07-20T02:25:49.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 msgr2=0x7fdfcc062fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fdfb4000fe0 tx=0x7fdfb4001ab0).stop 2022-07-20T02:25:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 shutdown_connections 2022-07-20T02:25:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfcc0a95a0 0x7fdfcc062fc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfb00539e0 0x7fdfb0055ea0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfcc0b3260 0x7fdfcc0634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 --2- 172.21.15.26:0/4178687589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfcc060950 0x7fdfcc0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 >> 172.21.15.26:0/4178687589 conn(0x7fdfcc0b67f0 msgr2=0x7fdfcc0b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 shutdown_connections 2022-07-20T02:25:49.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.555+0000 7fdfd0e38700 1 -- 172.21.15.26:0/4178687589 wait complete. 2022-07-20T02:25:49.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:49.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 Processor -- start 2022-07-20T02:25:49.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 -- start start 2022-07-20T02:25:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd9b8143f80 0x7fd9b8147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8061c80 con 0x7fd9b80b22d0 2022-07-20T02:25:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b80b2810 con 0x7fd9b80618a0 2022-07-20T02:25:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be8fd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b80b2ab0 con 0x7fd9b8143f80 2022-07-20T02:25:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be0fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd9b80b22d0 0x7fd9b80b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34778/0 (socket says 172.21.15.26:34778) 2022-07-20T02:25:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9be0fc700 1 -- 172.21.15.26:0/2133722300 learned_addr learned my addr 172.21.15.26:0/2133722300 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bd0fa700 1 --2- 172.21.15.26:0/2133722300 >> v2:172.21.15.26:3300/0 conn(0x7fd9b8143f80 0x7fd9b8147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299776091 0 0) 0x7fd9b8061c80 con 0x7fd9b80b22d0 2022-07-20T02:25:49.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a0003260 con 0x7fd9b80b22d0 2022-07-20T02:25:49.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721300241 0 0) 0x7fd9b80b2810 con 0x7fd9b80618a0 2022-07-20T02:25:49.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/2133722300 >> v1:172.21.15.120:6789/0 conn(0x7fd9b80618a0 legacy=0x7fd9b8143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/2133722300 >> v1:172.21.15.26:6789/0 conn(0x7fd9b80b22d0 legacy=0x7fd9b80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/2133722300 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b80b2d50 con 0x7fd9b8143f80 2022-07-20T02:25:49.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bd0fa700 1 --2- 172.21.15.26:0/2133722300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b8143f80 0x7fd9b8147250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd9b0001710 tx=0x7fd9a8006df0).ready entity=mon.0 client_cookie=233ebcbacab6a4dc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a800d4f0 con 0x7fd9b8143f80 2022-07-20T02:25:49.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a8000e90 con 0x7fd9b8143f80 2022-07-20T02:25:49.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.683+0000 7fd9bc8f9700 1 -- 172.21.15.26:0/2133722300 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a800d840 con 0x7fd9b8143f80 2022-07-20T02:25:49.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- 172.21.15.26:0/2133722300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b8143f80 msgr2=0x7fd9b8147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/2133722300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b8143f80 0x7fd9b8147250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd9b0001710 tx=0x7fd9a8006df0).stop 2022-07-20T02:25:49.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- 172.21.15.26:0/2133722300 shutdown_connections 2022-07-20T02:25:49.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/2133722300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b8143f80 0x7fd9b8147250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- 172.21.15.26:0/2133722300 >> 172.21.15.26:0/2133722300 conn(0x7fd9b8066f70 msgr2=0x7fd9b8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- 172.21.15.26:0/2133722300 shutdown_connections 2022-07-20T02:25:49.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- 172.21.15.26:0/2133722300 wait complete. 2022-07-20T02:25:49.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 Processor -- start 2022-07-20T02:25:49.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- start start 2022-07-20T02:25:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9b80618a0 0x7fd9b80ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b8143f80 0x7fd9b81527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80ae3f0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80adeb0 con 0x7fd9b8143f80 2022-07-20T02:25:49.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be8fd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80ae150 con 0x7fd9b80618a0 2022-07-20T02:25:49.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9bd0fa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9bd0fa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35502/0 (socket says 172.21.15.26:35502) 2022-07-20T02:25:49.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/3553159461 learned_addr learned my addr 172.21.15.26:0/3553159461 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:49.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9be0fc700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b8143f80 0x7fd9b81527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.687+0000 7fd9bd8fb700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9b80618a0 0x7fd9b80ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9b80618a0 msgr2=0x7fd9b80ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9b80618a0 0x7fd9b80ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b8143f80 msgr2=0x7fd9b81527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b8143f80 0x7fd9b81527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 -- 172.21.15.26:0/3553159461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a800b180 con 0x7fd9b80b22d0 2022-07-20T02:25:49.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd0fa700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd9b0001710 tx=0x7fd9a8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a8013de0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a8002f50 con 0x7fd9b80b22d0 2022-07-20T02:25:49.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b813f250 con 0x7fd9b80b22d0 2022-07-20T02:25:49.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a801e5f0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b80aedf0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b80574e0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd9a8025030 con 0x7fd9b80b22d0 2022-07-20T02:25:49.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 0x7fd998055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd8fb700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 0x7fd998055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd9a806cc90 con 0x7fd9b80b22d0 2022-07-20T02:25:49.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.691+0000 7fd9bd8fb700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 0x7fd998055ff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9b400d000 tx=0x7fd9b4007a90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.699+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9a8043940 con 0x7fd9b80b22d0 2022-07-20T02:25:49.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd9b80b66d0 con 0x7fd9b80b22d0 2022-07-20T02:25:49.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9a6ffd700 1 -- 172.21.15.26:0/3553159461 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd9a8023020 con 0x7fd9b80b22d0 2022-07-20T02:25:49.831 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 msgr2=0x7fd998055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 0x7fd998055ff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9b400d000 tx=0x7fd9b4007a90).stop 2022-07-20T02:25:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 msgr2=0x7fd9b80b7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd9b0001710 tx=0x7fd9a8014000).stop 2022-07-20T02:25:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 shutdown_connections 2022-07-20T02:25:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9b80b22d0 0x7fd9b80b7aa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd998053b30 0x7fd998055ff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b8143f80 0x7fd9b81527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 --2- 172.21.15.26:0/3553159461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9b80618a0 0x7fd9b80ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 >> 172.21.15.26:0/3553159461 conn(0x7fd9b8066f70 msgr2=0x7fd9b805c670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.823+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 shutdown_connections 2022-07-20T02:25:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.827+0000 7fd9be8fd700 1 -- 172.21.15.26:0/3553159461 wait complete. 2022-07-20T02:25:49.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:49.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.947+0000 7efe5bfff700 1 Processor -- start 2022-07-20T02:25:49.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.947+0000 7efe5bfff700 1 -- start start 2022-07-20T02:25:49.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efe5c0670c0 0x7efe5c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe5c064450 0x7efe5c064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c064880 con 0x7efe5c0b3380 2022-07-20T02:25:49.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c0b85f0 con 0x7efe5c0670c0 2022-07-20T02:25:49.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c0b80b0 con 0x7efe5c064450 2022-07-20T02:25:49.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5b7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efe5c0b3380 0x7efe5c0b4090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34792/0 (socket says 172.21.15.26:34792) 2022-07-20T02:25:49.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5b7fe700 1 -- 172.21.15.26:0/2203988061 learned_addr learned my addr 172.21.15.26:0/2203988061 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:49.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 --2- 172.21.15.26:0/2203988061 >> v2:172.21.15.26:3300/0 conn(0x7efe5c0670c0 0x7efe5c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe527fc700 1 --2- 172.21.15.26:0/2203988061 >> v2:172.21.15.120:3300/0 conn(0x7efe5c064450 0x7efe5c064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5a7fc700 1 -- 172.21.15.26:0/2203988061 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404127343 0 0) 0x7efe5c064880 con 0x7efe5c0b3380 2022-07-20T02:25:49.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5a7fc700 1 -- 172.21.15.26:0/2203988061 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003260 con 0x7efe5c0b3380 2022-07-20T02:25:49.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 -- 172.21.15.26:0/2203988061 >> v2:172.21.15.120:3300/0 conn(0x7efe5c064450 msgr2=0x7efe5c064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:49.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 --2- 172.21.15.26:0/2203988061 >> v2:172.21.15.120:3300/0 conn(0x7efe5c064450 0x7efe5c064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 -- 172.21.15.26:0/2203988061 >> v1:172.21.15.26:6789/0 conn(0x7efe5c0b3380 legacy=0x7efe5c0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 -- 172.21.15.26:0/2203988061 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c0b8350 con 0x7efe5c0670c0 2022-07-20T02:25:49.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5affd700 1 --2- 172.21.15.26:0/2203988061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c0670c0 0x7efe5c0b2e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7efe4c002f70 tx=0x7efe4c006df0).ready entity=mon.0 client_cookie=bf8cd20e6e34f55d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5a7fc700 1 -- 172.21.15.26:0/2203988061 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe4c00d4f0 con 0x7efe5c0670c0 2022-07-20T02:25:49.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5a7fc700 1 -- 172.21.15.26:0/2203988061 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4c000e90 con 0x7efe5c0670c0 2022-07-20T02:25:49.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5a7fc700 1 -- 172.21.15.26:0/2203988061 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe4c00d840 con 0x7efe5c0670c0 2022-07-20T02:25:49.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- 172.21.15.26:0/2203988061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c0670c0 msgr2=0x7efe5c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 --2- 172.21.15.26:0/2203988061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c0670c0 0x7efe5c0b2e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7efe4c002f70 tx=0x7efe4c006df0).stop 2022-07-20T02:25:49.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- 172.21.15.26:0/2203988061 shutdown_connections 2022-07-20T02:25:49.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 --2- 172.21.15.26:0/2203988061 >> v2:172.21.15.120:3300/0 conn(0x7efe5c064450 0x7efe5c064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 --2- 172.21.15.26:0/2203988061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c0670c0 0x7efe5c0b2e90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- 172.21.15.26:0/2203988061 >> 172.21.15.26:0/2203988061 conn(0x7efe5c05f840 msgr2=0x7efe5c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:49.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- 172.21.15.26:0/2203988061 shutdown_connections 2022-07-20T02:25:49.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.951+0000 7efe5bfff700 1 -- 172.21.15.26:0/2203988061 wait complete. 2022-07-20T02:25:49.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 Processor -- start 2022-07-20T02:25:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 -- start start 2022-07-20T02:25:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5c0670c0 0x7efe5c142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe5c060f90 0x7efe5c144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c061650 con 0x7efe5c1447f0 2022-07-20T02:25:49.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c0618f0 con 0x7efe5c0670c0 2022-07-20T02:25:49.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c061b90 con 0x7efe5c060f90 2022-07-20T02:25:49.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35518/0 (socket says 172.21.15.26:35518) 2022-07-20T02:25:49.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5affd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5c0670c0 0x7efe5c142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe527fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe5c060f90 0x7efe5c144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 -- 172.21.15.26:0/1306266 learned_addr learned my addr 172.21.15.26:0/1306266 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:49.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 -- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe5c060f90 msgr2=0x7efe5c144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe5c060f90 0x7efe5c144300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.955+0000 7efe5b7fe700 1 -- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5c0670c0 msgr2=0x7efe5c142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5b7fe700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5c0670c0 0x7efe5c142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:49.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5b7fe700 1 -- 172.21.15.26:0/1306266 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c00b180 con 0x7efe5c1447f0 2022-07-20T02:25:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5b7fe700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efe54001490 tx=0x7efe54007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe54005ba0 con 0x7efe5c1447f0 2022-07-20T02:25:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe54005d00 con 0x7efe5c1447f0 2022-07-20T02:25:49.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c061e30 con 0x7efe5c1447f0 2022-07-20T02:25:49.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe540072c0 con 0x7efe5c1447f0 2022-07-20T02:25:49.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c060c30 con 0x7efe5c1447f0 2022-07-20T02:25:49.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c064ab0 con 0x7efe5c1447f0 2022-07-20T02:25:49.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efe54010070 con 0x7efe5c1447f0 2022-07-20T02:25:49.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 0x7efe40055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:49.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efe5406b9d0 con 0x7efe5c1447f0 2022-07-20T02:25:49.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5affd700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 0x7efe40055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:49.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.959+0000 7efe5affd700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 0x7efe40055f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efe4c01a000 tx=0x7efe4c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:49.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:49.963+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe54042790 con 0x7efe5c1447f0 2022-07-20T02:25:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.091+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe5c0574e0 con 0x7efe5c1447f0 2022-07-20T02:25:50.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.091+0000 7efe58ff9700 1 -- 172.21.15.26:0/1306266 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7efe54045db0 con 0x7efe5c1447f0 2022-07-20T02:25:50.095 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 msgr2=0x7efe40055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 0x7efe40055f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efe4c01a000 tx=0x7efe4c014000).stop 2022-07-20T02:25:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 msgr2=0x7efe5c155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efe54001490 tx=0x7efe54007e70).stop 2022-07-20T02:25:50.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 shutdown_connections 2022-07-20T02:25:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe5c1447f0 0x7efe5c155750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efe40053aa0 0x7efe40055f60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe5c060f90 0x7efe5c144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 --2- 172.21.15.26:0/1306266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5c0670c0 0x7efe5c142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 >> 172.21.15.26:0/1306266 conn(0x7efe5c05f840 msgr2=0x7efe5c05c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 shutdown_connections 2022-07-20T02:25:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.095+0000 7efe5bfff700 1 -- 172.21.15.26:0/1306266 wait complete. 2022-07-20T02:25:50.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 Processor -- start 2022-07-20T02:25:50.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 -- start start 2022-07-20T02:25:50.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc59805fcd0 0x7fc5980ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc5980aba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59805d3b0 con 0x7fc59805cfd0 2022-07-20T02:25:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc598061c00 con 0x7fc59805fcd0 2022-07-20T02:25:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc597fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc598061ea0 con 0x7fc5980602c0 2022-07-20T02:25:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc596ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc59805cfd0 0x7fc5980acba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34802/0 (socket says 172.21.15.26:34802) 2022-07-20T02:25:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.231+0000 7fc596ffd700 1 -- 172.21.15.26:0/2266450671 learned_addr learned my addr 172.21.15.26:0/2266450671 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5967fc700 1 --2- 172.21.15.26:0/2266450671 >> v2:172.21.15.120:3300/0 conn(0x7fc59805fcd0 0x7fc5980ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5977fe700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc5980aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040394032 0 0) 0x7fc59805d3b0 con 0x7fc59805cfd0 2022-07-20T02:25:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc578003260 con 0x7fc59805cfd0 2022-07-20T02:25:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004192204 0 0) 0x7fc578003260 con 0x7fc59805cfd0 2022-07-20T02:25:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 msgr2=0x7fc5980aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc5980aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 >> v2:172.21.15.120:3300/0 conn(0x7fc59805fcd0 msgr2=0x7fc5980ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 --2- 172.21.15.26:0/2266450671 >> v2:172.21.15.120:3300/0 conn(0x7fc59805fcd0 0x7fc5980ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598062140 con 0x7fc59805cfd0 2022-07-20T02:25:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fc580001680 con 0x7fc59805cfd0 2022-07-20T02:25:50.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 >> v1:172.21.15.26:6789/0 conn(0x7fc59805cfd0 legacy=0x7fc5980acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc578003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc578004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5780049f0 0x7fc578004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc598061c00 con 0x7fc578003750 2022-07-20T02:25:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc598061ea0 con 0x7fc5780049f0 2022-07-20T02:25:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57800b750 con 0x7fc5980602c0 2022-07-20T02:25:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc578003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5967fc700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc578004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5977fe700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5780049f0 0x7fc578004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 -- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 msgr2=0x7fc578004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc578004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 -- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5780049f0 msgr2=0x7fc578004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5780049f0 0x7fc578004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 -- 172.21.15.26:0/2266450671 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc578003260 con 0x7fc578003750 2022-07-20T02:25:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5967fc700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980602c0 0x7fc578004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc596ffd700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc578003b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc580001390 tx=0x7fc598060140).ready entity=mon.0 client_cookie=82f0c5938eb5e8fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc580007170 con 0x7fc578003750 2022-07-20T02:25:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5800072d0 con 0x7fc578003750 2022-07-20T02:25:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc595ffb700 1 -- 172.21.15.26:0/2266450671 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc58000c9c0 con 0x7fc578003750 2022-07-20T02:25:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 -- 172.21.15.26:0/2266450671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 msgr2=0x7fc578003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 --2- 172.21.15.26:0/2266450671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc578003b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc580001390 tx=0x7fc598060140).stop 2022-07-20T02:25:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc5977fe700 1 -- 172.21.15.26:0/2266450671 reap_dead start 2022-07-20T02:25:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 -- 172.21.15.26:0/2266450671 shutdown_connections 2022-07-20T02:25:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 -- 172.21.15.26:0/2266450671 >> 172.21.15.26:0/2266450671 conn(0x7fc5980b7090 msgr2=0x7fc59805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:50.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 -- 172.21.15.26:0/2266450671 shutdown_connections 2022-07-20T02:25:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.235+0000 7fc597fff700 1 -- 172.21.15.26:0/2266450671 wait complete. 2022-07-20T02:25:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 Processor -- start 2022-07-20T02:25:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 -- start start 2022-07-20T02:25:50.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598156fd0 0x7fc598157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 0x7fc5980a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc598156a50 con 0x7fc578003750 2022-07-20T02:25:50.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59809f020 con 0x7fc598156fd0 2022-07-20T02:25:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59809f250 con 0x7fc59809ff60 2022-07-20T02:25:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35532/0 (socket says 172.21.15.26:35532) 2022-07-20T02:25:50.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc5967fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598156fd0 0x7fc598157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 -- 172.21.15.26:0/500982785 learned_addr learned my addr 172.21.15.26:0/500982785 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:50.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc5977fe700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 0x7fc5980a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 -- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 msgr2=0x7fc5980a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 0x7fc5980a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 -- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598156fd0 msgr2=0x7fc598157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598156fd0 0x7fc598157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 -- 172.21.15.26:0/500982785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980a1280 con 0x7fc578003750 2022-07-20T02:25:50.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc5977fe700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 0x7fc5980a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:50.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc596ffd700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fc5980a22a0 tx=0x7fc580013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc580007860 con 0x7fc578003750 2022-07-20T02:25:50.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.239+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59814cb50 con 0x7fc578003750 2022-07-20T02:25:50.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc580007440 con 0x7fc578003750 2022-07-20T02:25:50.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc580004920 con 0x7fc578003750 2022-07-20T02:25:50.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59809ec20 con 0x7fc578003750 2022-07-20T02:25:50.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc58001d070 con 0x7fc578003750 2022-07-20T02:25:50.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5980ab5c0 con 0x7fc578003750 2022-07-20T02:25:50.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc58ffff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 0x7fc57c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc58006be00 con 0x7fc578003750 2022-07-20T02:25:50.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc5967fc700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 0x7fc57c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.243+0000 7fc5967fc700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 0x7fc57c055f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc58800b000 tx=0x7fc588006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.255+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc580042b20 con 0x7fc578003750 2022-07-20T02:25:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5980574e0 con 0x7fc578003750 2022-07-20T02:25:50.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc58ffff700 1 -- 172.21.15.26:0/500982785 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc580046140 con 0x7fc578003750 2022-07-20T02:25:50.389 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:50.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 msgr2=0x7fc57c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 0x7fc57c055f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc58800b000 tx=0x7fc588006df0).stop 2022-07-20T02:25:50.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 msgr2=0x7fc5980a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fc5980a22a0 tx=0x7fc580013000).stop 2022-07-20T02:25:50.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 shutdown_connections 2022-07-20T02:25:50.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc57c053aa0 0x7fc57c055f60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.383+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc59809ff60 0x7fc5980a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.387+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598156fd0 0x7fc598157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.387+0000 7fc597fff700 1 --2- 172.21.15.26:0/500982785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc578003750 0x7fc5980a2c40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.387+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 >> 172.21.15.26:0/500982785 conn(0x7fc5980b7090 msgr2=0x7fc59814be80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:50.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.387+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 shutdown_connections 2022-07-20T02:25:50.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.387+0000 7fc597fff700 1 -- 172.21.15.26:0/500982785 wait complete. 2022-07-20T02:25:50.402 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:25:50.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:25:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 Processor -- start 2022-07-20T02:25:50.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 -- start start 2022-07-20T02:25:50.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdb3c0614b0 0x7fdb3c061940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0ae370 con 0x7fdb3c0b14b0 2022-07-20T02:25:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b28b0 con 0x7fdb3c0adf90 2022-07-20T02:25:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb40b5f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b0610 con 0x7fdb3c0614b0 2022-07-20T02:25:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35540/0 (socket says 172.21.15.26:35540) 2022-07-20T02:25:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 -- 172.21.15.26:0/1197177386 learned_addr learned my addr 172.21.15.26:0/1197177386 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3ad9d700 1 --2- 172.21.15.26:0/1197177386 >> v2:172.21.15.120:3300/0 conn(0x7fdb3c0614b0 0x7fdb3c061940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 -- 172.21.15.26:0/1197177386 >> v2:172.21.15.120:3300/0 conn(0x7fdb3c0614b0 msgr2=0x7fdb3c061940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 --2- 172.21.15.26:0/1197177386 >> v2:172.21.15.120:3300/0 conn(0x7fdb3c0614b0 0x7fdb3c061940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 -- 172.21.15.26:0/1197177386 >> v1:172.21.15.120:6789/0 conn(0x7fdb3c0b14b0 legacy=0x7fdb3c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 -- 172.21.15.26:0/1197177386 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0b08b0 con 0x7fdb3c0adf90 2022-07-20T02:25:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3a59c700 1 --2- 172.21.15.26:0/1197177386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdb2c002f70 tx=0x7fdb2c006eb0).ready entity=mon.0 client_cookie=bca027ef7abde867 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3959a700 1 -- 172.21.15.26:0/1197177386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2c00f9c0 con 0x7fdb3c0adf90 2022-07-20T02:25:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3959a700 1 -- 172.21.15.26:0/1197177386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c000e90 con 0x7fdb3c0adf90 2022-07-20T02:25:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.507+0000 7fdb3959a700 1 -- 172.21.15.26:0/1197177386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2c00fd50 con 0x7fdb3c0adf90 2022-07-20T02:25:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- 172.21.15.26:0/1197177386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0adf90 msgr2=0x7fdb3c0b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/1197177386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdb2c002f70 tx=0x7fdb2c006eb0).stop 2022-07-20T02:25:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- 172.21.15.26:0/1197177386 shutdown_connections 2022-07-20T02:25:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/1197177386 >> v2:172.21.15.120:3300/0 conn(0x7fdb3c0614b0 0x7fdb3c061940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/1197177386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0adf90 0x7fdb3c0b2c60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- 172.21.15.26:0/1197177386 >> 172.21.15.26:0/1197177386 conn(0x7fdb3c0b66e0 msgr2=0x7fdb3c0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- 172.21.15.26:0/1197177386 shutdown_connections 2022-07-20T02:25:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- 172.21.15.26:0/1197177386 wait complete. 2022-07-20T02:25:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 Processor -- start 2022-07-20T02:25:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- start start 2022-07-20T02:25:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb3c0adf90 0x7fdb3c064aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 0x7fdb3c09ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0a0870 con 0x7fdb3c0614b0 2022-07-20T02:25:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c09f760 con 0x7fdb3c0b14b0 2022-07-20T02:25:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb40b5f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c09ec80 con 0x7fdb3c0adf90 2022-07-20T02:25:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb3a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb3ad9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 0x7fdb3c09ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb3a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35546/0 (socket says 172.21.15.26:35546) 2022-07-20T02:25:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb3a59c700 1 -- 172.21.15.26:0/4165056838 learned_addr learned my addr 172.21.15.26:0/4165056838 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.511+0000 7fdb39d9b700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb3c0adf90 0x7fdb3c064aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 -- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb3c0adf90 msgr2=0x7fdb3c064aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb3c0adf90 0x7fdb3c064aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 -- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 msgr2=0x7fdb3c09ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 0x7fdb3c09ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 -- 172.21.15.26:0/4165056838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c00b1e0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3ad9d700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 0x7fdb3c09ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb3a59c700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdb2c002f70 tx=0x7fdb2c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2c020070 con 0x7fdb3c0614b0 2022-07-20T02:25:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c010da0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c14ef30 con 0x7fdb3c0614b0 2022-07-20T02:25:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2c01a670 con 0x7fdb3c0614b0 2022-07-20T02:25:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c0a0aa0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c0574e0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.515+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb2c01a7d0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.519+0000 7fdb337fe700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 0x7fdb14055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.519+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdb2c06f030 con 0x7fdb3c0614b0 2022-07-20T02:25:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.519+0000 7fdb39d9b700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 0x7fdb14055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:50.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.519+0000 7fdb39d9b700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 0x7fdb14055f60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fdb2400d000 tx=0x7fdb24007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:50.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.523+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb2c045e80 con 0x7fdb3c0614b0 2022-07-20T02:25:50.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.679+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3c0683d0 con 0x7fdb3c0614b0 2022-07-20T02:25:50.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.679+0000 7fdb337fe700 1 -- 172.21.15.26:0/4165056838 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2489 (secure 0 0 0) 0x7fdb2c049460 con 0x7fdb3c0614b0 2022-07-20T02:25:50.684 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:50.685 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:25:37.669039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/4183261198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4183261198},{"type":"v1","addr":"172.21.15.26:6837","nonce":4183261198}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:25:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 msgr2=0x7fdb14055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 0x7fdb14055f60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fdb2400d000 tx=0x7fdb24007b90).stop 2022-07-20T02:25:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 msgr2=0x7fdb3c14ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdb2c002f70 tx=0x7fdb2c017000).stop 2022-07-20T02:25:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 shutdown_connections 2022-07-20T02:25:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb14053aa0 0x7fdb14055f60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb3c0b14b0 0x7fdb3c09ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb3c0adf90 0x7fdb3c064aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 --2- 172.21.15.26:0/4165056838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb3c0614b0 0x7fdb3c14ea40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 >> 172.21.15.26:0/4165056838 conn(0x7fdb3c0b66e0 msgr2=0x7fdb3c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 shutdown_connections 2022-07-20T02:25:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:50.683+0000 7fdb40b5f700 1 -- 172.21.15.26:0/4165056838 wait complete. 2022-07-20T02:25:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2022-07-20T02:25:51.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:25:35.946073+0000', 'modified': '2022-07-20T02:25:37.669039+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/4183261198', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 4183261198}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 4183261198}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:25:51.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:25:51.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:25:51.132 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":4266,"want_state":"up:active","state":"up:active","rank_uptime":15.174622036000001,"mdsmap_epoch":7,"osdmap_epoch":24,"osdmap_epoch_barrier":24,"uptime":21.541492243} 2022-07-20T02:25:51.133 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 4266, "mdsmap_epoch": 7, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "rank_uptime": 15.174622036, "state": "up:active", "uptime": 21.541492243, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:25:51.134 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:25:51.135 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:25:51.136 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:25:51.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:51.137 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:25:51.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:25:51.209 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:25:51.210 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:25:51.223 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:25:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:25:51.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:51.225 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:25:51.284 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:25:51.285 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:25:51.338 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:25:51.364 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:25:51.364 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:25:51.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:25:51.427+0000 7f89c3567080 -1 init, newargv = 0x55d08c8158a0 newargc=9ceph-fuse[ 2022-07-20T02:25:51.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:22222]: starting ceph client 2022-07-20T02:25:51.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22222]: starting fuse 2022-07-20T02:25:51.439 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:25:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:25:51.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:51.446 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:25:51.471 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:25:51.472 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:25:51.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:51.474 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:25:51.474 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:25:51.475 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:25:51.476 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:25:51.476 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:25:51.477 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:25:51.477 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:25:51.478 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:25:51.478 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:25:51.479 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:25:51.480 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:25:51.480 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:25:51.481 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:25:51.481 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:25:51.482 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:25:51.483 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:25:51.483 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:25:51.484 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:25:51.484 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:25:51.484 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:25:51.485 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:25:51.486 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:25:51.487 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:25:51.562 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22222.asok 2022-07-20T02:25:51.567 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22222.asok 2022-07-20T02:25:51.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:51.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22222.asok status 2022-07-20T02:25:51.706 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:25:51.707 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:25:51.708 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:25:51.708 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:25:51.708 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:25:51.709 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:25:51.709 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:25:51.709 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22222", 2022-07-20T02:25:51.710 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:25:51.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:25:51.711 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:25:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:25:51.712 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 4621, 2022-07-20T02:25:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:25:51.713 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:25:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:25:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 4621 2022-07-20T02:25:51.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:25:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:25:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:25:51.715 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:25:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3664053108 2022-07-20T02:25:51.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:25:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:25:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:25:51.717 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:25:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:25:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3664053108 2022-07-20T02:25:51.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:25:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.4621 172.21.15.159:0/3664053108", 2022-07-20T02:25:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/3664053108", 2022-07-20T02:25:51.719 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:25:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 7, 2022-07-20T02:25:51.720 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 24, 2022-07-20T02:25:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:25:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:25:51.721 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:25:51.722 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:25:51.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:51.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.835+0000 7f97f1043700 1 Processor -- start 2022-07-20T02:25:51.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 -- start start 2022-07-20T02:25:51.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 0x7f97ec0a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f97ec098320 0x7f97ec060020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec098800 con 0x7f97ec060510 2022-07-20T02:25:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec098aa0 con 0x7f97ec098320 2022-07-20T02:25:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97f1043700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec05f8e0 con 0x7f97ec0b1c60 2022-07-20T02:25:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f97ec098320 0x7f97ec060020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97eb59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f97ec060510 0x7f97ec143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34838/0 (socket says 172.21.15.26:34838) 2022-07-20T02:25:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97eb59e700 1 -- 172.21.15.26:0/4284487341 learned_addr learned my addr 172.21.15.26:0/4284487341 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ead9d700 1 --2- 172.21.15.26:0/4284487341 >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 0x7f97ec0a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ea59c700 1 -- 172.21.15.26:0/4284487341 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208719666 0 0) 0x7f97ec098800 con 0x7f97ec060510 2022-07-20T02:25:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ea59c700 1 -- 172.21.15.26:0/4284487341 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97cc003260 con 0x7f97ec060510 2022-07-20T02:25:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 -- 172.21.15.26:0/4284487341 >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 msgr2=0x7f97ec0a9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:51.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 --2- 172.21.15.26:0/4284487341 >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 0x7f97ec0a9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 -- 172.21.15.26:0/4284487341 >> v1:172.21.15.26:6789/0 conn(0x7f97ec060510 legacy=0x7f97ec143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 -- 172.21.15.26:0/4284487341 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec05fb80 con 0x7f97ec098320 2022-07-20T02:25:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ead9d700 1 --2- 172.21.15.26:0/4284487341 >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 0x7f97ec0a9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97e3fff700 1 --2- 172.21.15.26:0/4284487341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec060020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f97d4002f70 tx=0x7f97d4006df0).ready entity=mon.0 client_cookie=95923bed10f1fe79 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:51.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ea59c700 1 -- 172.21.15.26:0/4284487341 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97d400d4f0 con 0x7f97ec098320 2022-07-20T02:25:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ea59c700 1 -- 172.21.15.26:0/4284487341 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4000e90 con 0x7f97ec098320 2022-07-20T02:25:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.839+0000 7f97ea59c700 1 -- 172.21.15.26:0/4284487341 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97d400d840 con 0x7f97ec098320 2022-07-20T02:25:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- 172.21.15.26:0/4284487341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 msgr2=0x7f97ec060020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 --2- 172.21.15.26:0/4284487341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec060020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f97d4002f70 tx=0x7f97d4006df0).stop 2022-07-20T02:25:51.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- 172.21.15.26:0/4284487341 shutdown_connections 2022-07-20T02:25:51.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 --2- 172.21.15.26:0/4284487341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec060020 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 --2- 172.21.15.26:0/4284487341 >> v2:172.21.15.120:3300/0 conn(0x7f97ec0b1c60 0x7f97ec0a9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- 172.21.15.26:0/4284487341 >> 172.21.15.26:0/4284487341 conn(0x7f97ec0b5e70 msgr2=0x7f97ec0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- 172.21.15.26:0/4284487341 shutdown_connections 2022-07-20T02:25:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- 172.21.15.26:0/4284487341 wait complete. 2022-07-20T02:25:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 Processor -- start 2022-07-20T02:25:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.843+0000 7f97f1043700 1 -- start start 2022-07-20T02:25:51.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec060510 0x7f97ec0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97ec0b1c60 0x7f97ec098ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec09a0c0 con 0x7f97ec098320 2022-07-20T02:25:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec152ea0 con 0x7f97ec060510 2022-07-20T02:25:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec153140 con 0x7f97ec0b1c60 2022-07-20T02:25:51.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35562/0 (socket says 172.21.15.26:35562) 2022-07-20T02:25:51.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 -- 172.21.15.26:0/3585400764 learned_addr learned my addr 172.21.15.26:0/3585400764 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:51.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97ead9d700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec060510 0x7f97ec0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97eb59e700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97ec0b1c60 0x7f97ec098ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 -- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97ec0b1c60 msgr2=0x7f97ec098ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97ec0b1c60 0x7f97ec098ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 -- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec060510 msgr2=0x7f97ec0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec060510 0x7f97ec0ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 -- 172.21.15.26:0/3585400764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc00b180 con 0x7f97ec098320 2022-07-20T02:25:51.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e3fff700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f97d4002f70 tx=0x7f97d4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:51.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97d4016070 con 0x7f97ec098320 2022-07-20T02:25:51.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4013a00 con 0x7f97ec098320 2022-07-20T02:25:51.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d400b180 con 0x7f97ec098320 2022-07-20T02:25:51.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97d4013d60 con 0x7f97ec098320 2022-07-20T02:25:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec0adc50 con 0x7f97ec098320 2022-07-20T02:25:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec0ade10 con 0x7f97ec098320 2022-07-20T02:25:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f97d400f030 con 0x7f97ec098320 2022-07-20T02:25:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 0x7f97d0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.847+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f97d406cd20 con 0x7f97ec098320 2022-07-20T02:25:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.851+0000 7f97ead9d700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 0x7f97d0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.851+0000 7f97ead9d700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 0x7f97d0055e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f97dc00b000 tx=0x7f97dc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.855+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97d4043a40 con 0x7f97ec098320 2022-07-20T02:25:51.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.983+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97ec0574e0 con 0x7f97ec098320 2022-07-20T02:25:51.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.983+0000 7f97e8d99700 1 -- 172.21.15.26:0/3585400764 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f97d4023020 con 0x7f97ec098320 2022-07-20T02:25:51.988 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:51.988 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:51.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 msgr2=0x7f97d0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 0x7f97d0055e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f97dc00b000 tx=0x7f97dc008040).stop 2022-07-20T02:25:51.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 msgr2=0x7f97ec0abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:51.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f97d4002f70 tx=0x7f97d4014000).stop 2022-07-20T02:25:51.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 shutdown_connections 2022-07-20T02:25:51.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97ec098320 0x7f97ec0abfd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f97d0053950 0x7f97d0055e10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97ec0b1c60 0x7f97ec098ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 --2- 172.21.15.26:0/3585400764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec060510 0x7f97ec0ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:51.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 >> 172.21.15.26:0/3585400764 conn(0x7f97ec0b5e70 msgr2=0x7f97ec0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:51.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 shutdown_connections 2022-07-20T02:25:51.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:51.987+0000 7f97f1043700 1 -- 172.21.15.26:0/3585400764 wait complete. 2022-07-20T02:25:52.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:52.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:52.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 Processor -- start 2022-07-20T02:25:52.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 -- start start 2022-07-20T02:25:52.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8ca805cfd0 0x7f8ca8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8ca8060f00 0x7f8ca813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca81474e0 con 0x7f8ca8147000 2022-07-20T02:25:52.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca80b4e40 con 0x7f8ca805cfd0 2022-07-20T02:25:52.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cae1e1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca805d3b0 con 0x7f8ca8060f00 2022-07-20T02:25:52.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8ca805cfd0 0x7f8ca8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8ca805cfd0 0x7f8ca8143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35570/0 (socket says 172.21.15.26:35570) 2022-07-20T02:25:52.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 -- 172.21.15.26:0/3088134186 learned_addr learned my addr 172.21.15.26:0/3088134186 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad9e0700 1 --2- 172.21.15.26:0/3088134186 >> v2:172.21.15.120:3300/0 conn(0x7f8ca8060f00 0x7f8ca813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 -- 172.21.15.26:0/3088134186 >> v2:172.21.15.120:3300/0 conn(0x7f8ca8060f00 msgr2=0x7f8ca813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:52.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 --2- 172.21.15.26:0/3088134186 >> v2:172.21.15.120:3300/0 conn(0x7f8ca8060f00 0x7f8ca813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.115+0000 7f8cad1df700 1 -- 172.21.15.26:0/3088134186 >> v1:172.21.15.120:6789/0 conn(0x7f8ca8147000 legacy=0x7f8ca80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cad1df700 1 -- 172.21.15.26:0/3088134186 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca81431e0 con 0x7f8ca805cfd0 2022-07-20T02:25:52.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cad1df700 1 --2- 172.21.15.26:0/3088134186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca805cfd0 0x7f8ca8143bc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8ca00032f0 tx=0x7f8ca0009290).ready entity=mon.0 client_cookie=3194042992eb26a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8ca7fff700 1 -- 172.21.15.26:0/3088134186 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca000b870 con 0x7f8ca805cfd0 2022-07-20T02:25:52.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8ca7fff700 1 -- 172.21.15.26:0/3088134186 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca000b9d0 con 0x7f8ca805cfd0 2022-07-20T02:25:52.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8ca7fff700 1 -- 172.21.15.26:0/3088134186 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca000bb40 con 0x7f8ca805cfd0 2022-07-20T02:25:52.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- 172.21.15.26:0/3088134186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca805cfd0 msgr2=0x7f8ca8143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/3088134186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca805cfd0 0x7f8ca8143bc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8ca00032f0 tx=0x7f8ca0009290).stop 2022-07-20T02:25:52.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- 172.21.15.26:0/3088134186 shutdown_connections 2022-07-20T02:25:52.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/3088134186 >> v2:172.21.15.120:3300/0 conn(0x7f8ca8060f00 0x7f8ca813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/3088134186 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca805cfd0 0x7f8ca8143bc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- 172.21.15.26:0/3088134186 >> 172.21.15.26:0/3088134186 conn(0x7f8ca80620c0 msgr2=0x7f8ca805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- 172.21.15.26:0/3088134186 shutdown_connections 2022-07-20T02:25:52.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- 172.21.15.26:0/3088134186 wait complete. 2022-07-20T02:25:52.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 Processor -- start 2022-07-20T02:25:52.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.119+0000 7f8cae1e1700 1 -- start start 2022-07-20T02:25:52.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca8147000 0x7f8ca80ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ca80b00e0 0x7f8ca81421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b91e0 con 0x7f8ca80adfc0 2022-07-20T02:25:52.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b93e0 con 0x7f8ca8147000 2022-07-20T02:25:52.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca8138380 con 0x7f8ca80b00e0 2022-07-20T02:25:52.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35578/0 (socket says 172.21.15.26:35578) 2022-07-20T02:25:52.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 -- 172.21.15.26:0/2229031016 learned_addr learned my addr 172.21.15.26:0/2229031016 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad1df700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca8147000 0x7f8ca80ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cac9de700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ca80b00e0 0x7f8ca81421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 -- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ca80b00e0 msgr2=0x7f8ca81421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ca80b00e0 0x7f8ca81421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 -- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca8147000 msgr2=0x7f8ca80ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca8147000 0x7f8ca80ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 -- 172.21.15.26:0/2229031016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0009530 con 0x7f8ca80adfc0 2022-07-20T02:25:52.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cad9e0700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8c9800c000 tx=0x7f8c98006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c98011750 con 0x7f8ca80adfc0 2022-07-20T02:25:52.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c980118b0 con 0x7f8ca80adfc0 2022-07-20T02:25:52.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca8138620 con 0x7f8ca80adfc0 2022-07-20T02:25:52.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c98011c10 con 0x7f8ca80adfc0 2022-07-20T02:25:52.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca80b8d30 con 0x7f8ca80adfc0 2022-07-20T02:25:52.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.123+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca8143480 con 0x7f8ca80adfc0 2022-07-20T02:25:52.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.127+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8c98011d70 con 0x7f8ca80adfc0 2022-07-20T02:25:52.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.127+0000 7f8ca67fc700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 0x7f8c8c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.127+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8c9806c330 con 0x7f8ca80adfc0 2022-07-20T02:25:52.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.127+0000 7f8cad1df700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 0x7f8c8c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.127+0000 7f8cad1df700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 0x7f8c8c055f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8ca001b000 tx=0x7f8ca0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.135+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c980430f0 con 0x7f8ca80adfc0 2022-07-20T02:25:52.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8ca80b9550 con 0x7f8ca80adfc0 2022-07-20T02:25:52.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8ca67fc700 1 -- 172.21.15.26:0/2229031016 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8c98046710 con 0x7f8ca80adfc0 2022-07-20T02:25:52.263 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:52.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 msgr2=0x7f8c8c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 0x7f8c8c055f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8ca001b000 tx=0x7f8ca0016000).stop 2022-07-20T02:25:52.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 msgr2=0x7f8ca8135660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.259+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8c9800c000 tx=0x7f8c98006df0).stop 2022-07-20T02:25:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 shutdown_connections 2022-07-20T02:25:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8ca80adfc0 0x7f8ca8135660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c8c053a70 0x7f8c8c055f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ca80b00e0 0x7f8ca81421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 --2- 172.21.15.26:0/2229031016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca8147000 0x7f8ca80ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 >> 172.21.15.26:0/2229031016 conn(0x7f8ca8060f00 msgr2=0x7f8ca80b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 shutdown_connections 2022-07-20T02:25:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.263+0000 7f8cae1e1700 1 -- 172.21.15.26:0/2229031016 wait complete. 2022-07-20T02:25:52.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 Processor -- start 2022-07-20T02:25:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 -- start start 2022-07-20T02:25:52.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4fc0b9c80 0x7fd4fc1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd4fc062710 0x7fd4fc0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0ba060 con 0x7fd4fc0b6fe0 2022-07-20T02:25:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc12bee0 con 0x7fd4fc062710 2022-07-20T02:25:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd501425700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc061f80 con 0x7fd4fc0b9c80 2022-07-20T02:25:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd4fa7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd4fc062710 0x7fd4fc0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd4fa7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd4fc062710 0x7fd4fc0b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35582/0 (socket says 172.21.15.26:35582) 2022-07-20T02:25:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.387+0000 7fd4fa7fc700 1 -- 172.21.15.26:0/49357938 learned_addr learned my addr 172.21.15.26:0/49357938 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4faffd700 1 --2- 172.21.15.26:0/49357938 >> v2:172.21.15.120:3300/0 conn(0x7fd4fc0b9c80 0x7fd4fc1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 -- 172.21.15.26:0/49357938 >> v2:172.21.15.120:3300/0 conn(0x7fd4fc0b9c80 msgr2=0x7fd4fc1431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 --2- 172.21.15.26:0/49357938 >> v2:172.21.15.120:3300/0 conn(0x7fd4fc0b9c80 0x7fd4fc1431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 -- 172.21.15.26:0/49357938 >> v1:172.21.15.120:6789/0 conn(0x7fd4fc0b6fe0 legacy=0x7fd4fc1403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 -- 172.21.15.26:0/49357938 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc062220 con 0x7fd4fc062710 2022-07-20T02:25:52.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 --2- 172.21.15.26:0/49357938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc0b6af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd4ec002f70 tx=0x7fd4ec006eb0).ready entity=mon.0 client_cookie=69334cb8a9184afc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4f9ffb700 1 -- 172.21.15.26:0/49357938 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ec00d610 con 0x7fd4fc062710 2022-07-20T02:25:52.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4f9ffb700 1 -- 172.21.15.26:0/49357938 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec000e90 con 0x7fd4fc062710 2022-07-20T02:25:52.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4f9ffb700 1 -- 172.21.15.26:0/49357938 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ec00d9a0 con 0x7fd4fc062710 2022-07-20T02:25:52.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- 172.21.15.26:0/49357938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 msgr2=0x7fd4fc0b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- 172.21.15.26:0/49357938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc0b6af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd4ec002f70 tx=0x7fd4ec006eb0).stop 2022-07-20T02:25:52.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- 172.21.15.26:0/49357938 shutdown_connections 2022-07-20T02:25:52.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- 172.21.15.26:0/49357938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc0b6af0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- 172.21.15.26:0/49357938 >> v2:172.21.15.120:3300/0 conn(0x7fd4fc0b9c80 0x7fd4fc1431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- 172.21.15.26:0/49357938 >> 172.21.15.26:0/49357938 conn(0x7fd4fc066f70 msgr2=0x7fd4fc05faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- 172.21.15.26:0/49357938 shutdown_connections 2022-07-20T02:25:52.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- 172.21.15.26:0/49357938 wait complete. 2022-07-20T02:25:52.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 Processor -- start 2022-07-20T02:25:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- start start 2022-07-20T02:25:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4fc0b6fe0 0x7fd4fc1523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4fc0b9c80 0x7fd4fc1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc12d620 con 0x7fd4fc062710 2022-07-20T02:25:52.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc12c020 con 0x7fd4fc0b9c80 2022-07-20T02:25:52.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd501425700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0b35b0 con 0x7fd4fc0b6fe0 2022-07-20T02:25:52.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35588/0 (socket says 172.21.15.26:35588) 2022-07-20T02:25:52.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4faffd700 1 -- 172.21.15.26:0/163246783 learned_addr learned my addr 172.21.15.26:0/163246783 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fa7fc700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4fc0b6fe0 0x7fd4fc1523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4fb7fe700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4fc0b9c80 0x7fd4fc1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.391+0000 7fd4faffd700 1 -- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4fc0b6fe0 msgr2=0x7fd4fc1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4faffd700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4fc0b6fe0 0x7fd4fc1523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4faffd700 1 -- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4fc0b9c80 msgr2=0x7fd4fc1528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4faffd700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4fc0b9c80 0x7fd4fc1528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4faffd700 1 -- 172.21.15.26:0/163246783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec00b1e0 con 0x7fd4fc062710 2022-07-20T02:25:52.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4faffd700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd4e400a000 tx=0x7fd4e4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4e4011780 con 0x7fd4fc062710 2022-07-20T02:25:52.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e40118e0 con 0x7fd4fc062710 2022-07-20T02:25:52.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc12c250 con 0x7fd4fc062710 2022-07-20T02:25:52.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4e4011c40 con 0x7fd4fc062710 2022-07-20T02:25:52.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc05cdb0 con 0x7fd4fc062710 2022-07-20T02:25:52.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc0b6650 con 0x7fd4fc062710 2022-07-20T02:25:52.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd4e4012420 con 0x7fd4fc062710 2022-07-20T02:25:52.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 0x7fd4e0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4fa7fc700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 0x7fd4e0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.395+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd4e406bf10 con 0x7fd4fc062710 2022-07-20T02:25:52.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.399+0000 7fd4fa7fc700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 0x7fd4e0055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd4ec01c000 tx=0x7fd4ec015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.403+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4e4042b60 con 0x7fd4fc062710 2022-07-20T02:25:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd4fc0574e0 con 0x7fd4fc062710 2022-07-20T02:25:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd4f3fff700 1 -- 172.21.15.26:0/163246783 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd4e4046180 con 0x7fd4fc062710 2022-07-20T02:25:52.539 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 msgr2=0x7fd4e0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 0x7fd4e0055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd4ec01c000 tx=0x7fd4ec015000).stop 2022-07-20T02:25:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 msgr2=0x7fd4fc142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd4e400a000 tx=0x7fd4e4008040).stop 2022-07-20T02:25:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 shutdown_connections 2022-07-20T02:25:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4e0053a70 0x7fd4e0055f30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4fc0b9c80 0x7fd4fc1528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4fc0b6fe0 0x7fd4fc1523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 --2- 172.21.15.26:0/163246783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4fc062710 0x7fd4fc142cd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 >> 172.21.15.26:0/163246783 conn(0x7fd4fc066f70 msgr2=0x7fd4fc12b610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 shutdown_connections 2022-07-20T02:25:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.534+0000 7fd501425700 1 -- 172.21.15.26:0/163246783 wait complete. 2022-07-20T02:25:52.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.662+0000 7fd36a896700 1 Processor -- start 2022-07-20T02:25:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 -- start start 2022-07-20T02:25:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3640b0050 0x7fd3640b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd36405d050 0x7fd364143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3640a9e10 con 0x7fd364143e00 2022-07-20T02:25:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3640617a0 con 0x7fd36405d050 2022-07-20T02:25:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd36a896700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640aa040 con 0x7fd3640b0050 2022-07-20T02:25:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd36405d050 0x7fd364143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd36405d050 0x7fd364143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35600/0 (socket says 172.21.15.26:35600) 2022-07-20T02:25:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1120116381 learned_addr learned my addr 172.21.15.26:0/1120116381 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd363fff700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3640b0050 0x7fd3640b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd3637fe700 1 -- 172.21.15.26:0/1120116381 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148192225 0 0) 0x7fd3640a9e10 con 0x7fd364143e00 2022-07-20T02:25:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1120116381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3640b0050 msgr2=0x7fd3640b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3640b0050 0x7fd3640b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1120116381 >> v1:172.21.15.120:6789/0 conn(0x7fd364143e00 legacy=0x7fd3640b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1120116381 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640aa2e0 con 0x7fd36405d050 2022-07-20T02:25:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd35b7fe700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd36405d050 0x7fd364143910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd34c002f70 tx=0x7fd34c006e50).ready entity=mon.0 client_cookie=2a10c81e3f41e11e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd3637fe700 1 -- 172.21.15.26:0/1120116381 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34c00d5c0 con 0x7fd36405d050 2022-07-20T02:25:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd3637fe700 1 -- 172.21.15.26:0/1120116381 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c000e90 con 0x7fd36405d050 2022-07-20T02:25:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.666+0000 7fd3637fe700 1 -- 172.21.15.26:0/1120116381 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34c00d910 con 0x7fd36405d050 2022-07-20T02:25:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- 172.21.15.26:0/1120116381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd36405d050 msgr2=0x7fd364143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd36405d050 0x7fd364143910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd34c002f70 tx=0x7fd34c006e50).stop 2022-07-20T02:25:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- 172.21.15.26:0/1120116381 shutdown_connections 2022-07-20T02:25:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd36405d050 0x7fd364143910 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 --2- 172.21.15.26:0/1120116381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3640b0050 0x7fd3640b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- 172.21.15.26:0/1120116381 >> 172.21.15.26:0/1120116381 conn(0x7fd364060950 msgr2=0x7fd364060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- 172.21.15.26:0/1120116381 shutdown_connections 2022-07-20T02:25:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- 172.21.15.26:0/1120116381 wait complete. 2022-07-20T02:25:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 Processor -- start 2022-07-20T02:25:52.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.670+0000 7fd36a896700 1 -- start start 2022-07-20T02:25:52.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd36405d050 0x7fd364065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd364143e00 0x7fd36413f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640b6ac0 con 0x7fd3640b0050 2022-07-20T02:25:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36413cb20 con 0x7fd364143e00 2022-07-20T02:25:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36413f560 con 0x7fd36405d050 2022-07-20T02:25:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35604/0 (socket says 172.21.15.26:35604) 2022-07-20T02:25:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1290808382 learned_addr learned my addr 172.21.15.26:0/1290808382 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd368e33700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd364143e00 0x7fd36413f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd363fff700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd36405d050 0x7fd364065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd36405d050 msgr2=0x7fd364065bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd36405d050 0x7fd364065bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd364143e00 msgr2=0x7fd36413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd364143e00 0x7fd36413f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 -- 172.21.15.26:0/1290808382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c00b1e0 con 0x7fd3640b0050 2022-07-20T02:25:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd35b7fe700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd3640b5d30 tx=0x7fd34c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34c013e50 con 0x7fd3640b0050 2022-07-20T02:25:52.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c006850 con 0x7fd3640b0050 2022-07-20T02:25:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3640b6820 con 0x7fd3640b0050 2022-07-20T02:25:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34c01e5f0 con 0x7fd3640b0050 2022-07-20T02:25:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3640b5e10 con 0x7fd3640b0050 2022-07-20T02:25:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.674+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3640ae520 con 0x7fd3640b0050 2022-07-20T02:25:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.678+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd34c025030 con 0x7fd3640b0050 2022-07-20T02:25:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.678+0000 7fd361ffb700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 0x7fd33c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.678+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd34c06cf40 con 0x7fd3640b0050 2022-07-20T02:25:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.678+0000 7fd363fff700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 0x7fd33c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.678+0000 7fd363fff700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 0x7fd33c055f60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fd35400b000 tx=0x7fd354006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.686+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd34c043c30 con 0x7fd3640b0050 2022-07-20T02:25:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.814+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3640574e0 con 0x7fd3640b0050 2022-07-20T02:25:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.814+0000 7fd361ffb700 1 -- 172.21.15.26:0/1290808382 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd34c002f50 con 0x7fd3640b0050 2022-07-20T02:25:52.821 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:52.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 msgr2=0x7fd33c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 0x7fd33c055f60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fd35400b000 tx=0x7fd354006df0).stop 2022-07-20T02:25:52.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 msgr2=0x7fd36413eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd3640b5d30 tx=0x7fd34c014000).stop 2022-07-20T02:25:52.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 shutdown_connections 2022-07-20T02:25:52.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3640b0050 0x7fd36413eb80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd33c053aa0 0x7fd33c055f60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd364143e00 0x7fd36413f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 --2- 172.21.15.26:0/1290808382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd36405d050 0x7fd364065bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 >> 172.21.15.26:0/1290808382 conn(0x7fd364060950 msgr2=0x7fd3640ace00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 shutdown_connections 2022-07-20T02:25:52.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.818+0000 7fd36a896700 1 -- 172.21.15.26:0/1290808382 wait complete. 2022-07-20T02:25:52.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:52.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 Processor -- start 2022-07-20T02:25:52.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 -- start start 2022-07-20T02:25:52.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34f0142fa0 0x7f34f00aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34f00b1cb0 0x7f34f0143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:52.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f005d3b0 con 0x7f34f005cfd0 2022-07-20T02:25:52.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f00b1a60 con 0x7f34f0142fa0 2022-07-20T02:25:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5d73700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f00b1520 con 0x7f34f00b1cb0 2022-07-20T02:25:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34f0142fa0 0x7f34f00aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34f0142fa0 0x7f34f00aaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35612/0 (socket says 172.21.15.26:35612) 2022-07-20T02:25:52.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 -- 172.21.15.26:0/989632539 learned_addr learned my addr 172.21.15.26:0/989632539 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:52.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34f5572700 1 --2- 172.21.15.26:0/989632539 >> v2:172.21.15.120:3300/0 conn(0x7f34f00b1cb0 0x7f34f0143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:52.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34ef7fe700 1 -- 172.21.15.26:0/989632539 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073135178 0 0) 0x7f34f005d3b0 con 0x7f34f005cfd0 2022-07-20T02:25:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34ef7fe700 1 -- 172.21.15.26:0/989632539 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d4003260 con 0x7f34f005cfd0 2022-07-20T02:25:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 -- 172.21.15.26:0/989632539 >> v2:172.21.15.120:3300/0 conn(0x7f34f00b1cb0 msgr2=0x7f34f0143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 --2- 172.21.15.26:0/989632539 >> v2:172.21.15.120:3300/0 conn(0x7f34f00b1cb0 0x7f34f0143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 -- 172.21.15.26:0/989632539 >> v1:172.21.15.120:6789/0 conn(0x7f34f005cfd0 legacy=0x7f34f0142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 -- 172.21.15.26:0/989632539 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f00b17c0 con 0x7f34f0142fa0 2022-07-20T02:25:52.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34effff700 1 --2- 172.21.15.26:0/989632539 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f0142fa0 0x7f34f00aaf70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f34e0002f70 tx=0x7f34e0006eb0).ready entity=mon.0 client_cookie=fdfc36a6fae317ef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:52.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34ef7fe700 1 -- 172.21.15.26:0/989632539 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e000d610 con 0x7f34f0142fa0 2022-07-20T02:25:52.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34ef7fe700 1 -- 172.21.15.26:0/989632539 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0000e90 con 0x7f34f0142fa0 2022-07-20T02:25:52.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.938+0000 7f34ef7fe700 1 -- 172.21.15.26:0/989632539 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e000d9a0 con 0x7f34f0142fa0 2022-07-20T02:25:52.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- 172.21.15.26:0/989632539 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f0142fa0 msgr2=0x7f34f00aaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:52.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- 172.21.15.26:0/989632539 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f0142fa0 0x7f34f00aaf70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f34e0002f70 tx=0x7f34e0006eb0).stop 2022-07-20T02:25:52.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- 172.21.15.26:0/989632539 shutdown_connections 2022-07-20T02:25:52.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- 172.21.15.26:0/989632539 >> v2:172.21.15.120:3300/0 conn(0x7f34f00b1cb0 0x7f34f0143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- 172.21.15.26:0/989632539 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f0142fa0 0x7f34f00aaf70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:52.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- 172.21.15.26:0/989632539 >> 172.21.15.26:0/989632539 conn(0x7f34f00b7090 msgr2=0x7f34f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:52.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- 172.21.15.26:0/989632539 shutdown_connections 2022-07-20T02:25:52.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- 172.21.15.26:0/989632539 wait complete. 2022-07-20T02:25:52.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 Processor -- start 2022-07-20T02:25:52.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- start start 2022-07-20T02:25:52.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f005cfd0 0x7f34f01483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f0142fa0 0x7f34f0152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0065ea0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0156730 con 0x7f34f0142fa0 2022-07-20T02:25:53.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5d73700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f01569d0 con 0x7f34f005cfd0 2022-07-20T02:25:53.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35616/0 (socket says 172.21.15.26:35616) 2022-07-20T02:25:53.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 -- 172.21.15.26:0/4095915489 learned_addr learned my addr 172.21.15.26:0/4095915489 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:53.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f5572700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f0142fa0 0x7f34f0152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34f4d71700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f005cfd0 0x7f34f01483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 -- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f005cfd0 msgr2=0x7f34f01483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f005cfd0 0x7f34f01483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 -- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f0142fa0 msgr2=0x7f34f0152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f0142fa0 0x7f34f0152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 -- 172.21.15.26:0/4095915489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e000b1e0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.942+0000 7f34effff700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f34e0002f70 tx=0x7f34e0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:53.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e00117e0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0011940 con 0x7f34f00b1cb0 2022-07-20T02:25:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f00aee70 con 0x7f34f00b1cb0 2022-07-20T02:25:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e0011ca0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f0063800 con 0x7f34f00b1cb0 2022-07-20T02:25:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f00b2100 con 0x7f34f00b1cb0 2022-07-20T02:25:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f34e0025030 con 0x7f34f00b1cb0 2022-07-20T02:25:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 0x7f34c8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f34e006d150 con 0x7f34f00b1cb0 2022-07-20T02:25:53.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34f4d71700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 0x7f34c8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.946+0000 7f34f4d71700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 0x7f34c8055fc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f34d800b000 tx=0x7f34d8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:53.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:52.950+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34e0043f10 con 0x7f34f00b1cb0 2022-07-20T02:25:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34f00574e0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34edffb700 1 -- 172.21.15.26:0/4095915489 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f34e00230a0 con 0x7f34f00b1cb0 2022-07-20T02:25:53.083 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:53.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 msgr2=0x7f34c8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 0x7f34c8055fc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f34d800b000 tx=0x7f34d8009040).stop 2022-07-20T02:25:53.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 msgr2=0x7f34f00a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f34e0002f70 tx=0x7f34e0015000).stop 2022-07-20T02:25:53.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 shutdown_connections 2022-07-20T02:25:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34f00b1cb0 0x7f34f00a2f10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f34c8053b00 0x7f34c8055fc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f0142fa0 0x7f34f0152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 --2- 172.21.15.26:0/4095915489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34f005cfd0 0x7f34f01483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 >> 172.21.15.26:0/4095915489 conn(0x7f34f00b7090 msgr2=0x7f34f00afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 shutdown_connections 2022-07-20T02:25:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.078+0000 7f34f5d73700 1 -- 172.21.15.26:0/4095915489 wait complete. 2022-07-20T02:25:53.107 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:25:53.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:25:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 Processor -- start 2022-07-20T02:25:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 -- start start 2022-07-20T02:25:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7ac0b5ef0 0x7fd7ac1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0b62d0 con 0x7fd7ac0a8c40 2022-07-20T02:25:53.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac14aac0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b3595700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ac14ad60 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b2d94700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd7ac0a8c40 0x7fd7ac151dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34908/0 (socket says 172.21.15.26:34908) 2022-07-20T02:25:53.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b2d94700 1 -- 172.21.15.26:0/2269794748 learned_addr learned my addr 172.21.15.26:0/2269794748 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:53.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b2593700 1 --2- 172.21.15.26:0/2269794748 >> v2:172.21.15.120:3300/0 conn(0x7fd7ac0b5ef0 0x7fd7ac1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.210+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368326908 0 0) 0x7fd7ac0b62d0 con 0x7fd7ac0a8c40 2022-07-20T02:25:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd794003260 con 0x7fd7ac0a8c40 2022-07-20T02:25:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269694449 0 0) 0x7fd7ac14aac0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac0b62d0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b2593700 1 -- 172.21.15.26:0/2269794748 >> v1:172.21.15.120:6789/0 conn(0x7fd7ac0a7fc0 legacy=0x7fd7ac0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b2593700 1 -- 172.21.15.26:0/2269794748 >> v1:172.21.15.26:6789/0 conn(0x7fd7ac0a8c40 legacy=0x7fd7ac151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b2593700 1 -- 172.21.15.26:0/2269794748 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac14b000 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b2593700 1 --2- 172.21.15.26:0/2269794748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac1474d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd79c002f70 tx=0x7fd79c006df0).ready entity=mon.1 client_cookie=390ccb9d89e531d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79c00d540 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c000e90 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b1591700 1 -- 172.21.15.26:0/2269794748 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79c00d890 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 -- 172.21.15.26:0/2269794748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 msgr2=0x7fd7ac1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 --2- 172.21.15.26:0/2269794748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac1474d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd79c002f70 tx=0x7fd79c006df0).stop 2022-07-20T02:25:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 -- 172.21.15.26:0/2269794748 shutdown_connections 2022-07-20T02:25:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 --2- 172.21.15.26:0/2269794748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac1474d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 -- 172.21.15.26:0/2269794748 >> 172.21.15.26:0/2269794748 conn(0x7fd7ac060310 msgr2=0x7fd7ac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 -- 172.21.15.26:0/2269794748 shutdown_connections 2022-07-20T02:25:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.214+0000 7fd7b3595700 1 -- 172.21.15.26:0/2269794748 wait complete. 2022-07-20T02:25:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 Processor -- start 2022-07-20T02:25:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 -- start start 2022-07-20T02:25:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 0x7fd7ac0674d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac0679c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac067eb0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac15a9e0 con 0x7fd7ac0b5ef0 2022-07-20T02:25:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b3595700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0627d0 con 0x7fd7ac0a8c40 2022-07-20T02:25:53.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35630/0 (socket says 172.21.15.26:35630) 2022-07-20T02:25:53.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 -- 172.21.15.26:0/213990324 learned_addr learned my addr 172.21.15.26:0/213990324 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:53.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2d94700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac0679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b1d92700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 0x7fd7ac0674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 -- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 msgr2=0x7fd7ac0674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 0x7fd7ac0674d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 -- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 msgr2=0x7fd7ac0679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac0679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 -- 172.21.15.26:0/213990324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c00b180 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b1d92700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 0x7fd7ac0674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:53.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2d94700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac0679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:53.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.218+0000 7fd7b2593700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd79c002f70 tx=0x7fd79c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:53.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79c00d1c0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c013d50 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac062a70 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79c01e5f0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac1453d0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac14b2a0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd79c025030 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 0x7fd78c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:53.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd79c06d250 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd7b1d92700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 0x7fd78c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:53.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.222+0000 7fd7b1d92700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 0x7fd78c055fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd7a800b000 tx=0x7fd7a8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.230+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd79c044010 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.390+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7ac0574e0 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.390+0000 7fd79b7fe700 1 -- 172.21.15.26:0/213990324 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2489 (secure 0 0 0) 0x7fd79c023070 con 0x7fd7ac0a7fc0 2022-07-20T02:25:53.397 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:53.398 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:25:37.669039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/4183261198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4183261198},{"type":"v1","addr":"172.21.15.26:6837","nonce":4183261198}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:25:53.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 msgr2=0x7fd78c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 0x7fd78c055fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd7a800b000 tx=0x7fd7a8009040).stop 2022-07-20T02:25:53.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 msgr2=0x7fd7ac144ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd79c002f70 tx=0x7fd79c014000).stop 2022-07-20T02:25:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 shutdown_connections 2022-07-20T02:25:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd78c053b00 0x7fd78c055fc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7ac0b5ef0 0x7fd7ac0679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7ac0a8c40 0x7fd7ac0674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 --2- 172.21.15.26:0/213990324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7ac0a7fc0 0x7fd7ac144ee0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 >> 172.21.15.26:0/213990324 conn(0x7fd7ac060310 msgr2=0x7fd7ac0b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:53.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 shutdown_connections 2022-07-20T02:25:53.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:53.394+0000 7fd7b3595700 1 -- 172.21.15.26:0/213990324 wait complete. 2022-07-20T02:25:53.409 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2022-07-20T02:25:53.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:25:35.946073+0000', 'modified': '2022-07-20T02:25:37.669039+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/4183261198', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 4183261198}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 4183261198}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:25:53.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:25:53.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:25:53.841 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":4266,"want_state":"up:active","state":"up:active","rank_uptime":17.883019135000001,"mdsmap_epoch":7,"osdmap_epoch":24,"osdmap_epoch_barrier":24,"uptime":24.249880513000001} 2022-07-20T02:25:53.841 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 4266, "mdsmap_epoch": 7, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "rank_uptime": 17.883019135, "state": "up:active", "uptime": 24.249880513, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:25:53.842 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:25:53.842 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:25:53.843 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:25:53.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:53.843 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:25:53.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:25:53.922 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:25:53.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:25:53.934 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:25:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:25:53.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:53.936 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:25:53.998 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:25:53.999 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:25:54.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:25:54.038 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:25:54.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:25:54.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:25:54.089+0000 7f52e385c080 -1 init, newargv = 0x558ca087da10 newargc=9 2022-07-20T02:25:54.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22327]: starting ceph client 2022-07-20T02:25:54.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22327]: starting fuse 2022-07-20T02:25:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:25:54.109 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:25:54.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.110 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:25:54.166 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:25:54.167 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:25:54.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:25:54.169 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:25:54.169 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:25:54.170 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:25:54.170 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:25:54.171 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:25:54.172 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:25:54.172 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:25:54.173 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:25:54.177 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:25:54.177 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:25:54.178 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:25:54.180 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:25:54.180 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:25:54.181 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:25:54.182 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:25:54.183 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:25:54.183 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:25:54.184 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:25:54.185 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:25:54.186 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:25:54.187 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:25:54.188 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:25:54.234 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22327.asok 2022-07-20T02:25:54.241 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22327.asok 2022-07-20T02:25:54.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22327.asok status 2022-07-20T02:25:54.377 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:25:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:25:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:25:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:25:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:25:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:25:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:25:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22327", 2022-07-20T02:25:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:25:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:25:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:25:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:25:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 4674, 2022-07-20T02:25:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:25:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:25:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:25:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 4674 2022-07-20T02:25:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:25:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:25:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:25:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:25:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3375777989 2022-07-20T02:25:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:25:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:25:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:25:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:25:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:25:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3375777989 2022-07-20T02:25:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:25:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.4674 172.21.15.176:0/3375777989", 2022-07-20T02:25:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/3375777989", 2022-07-20T02:25:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:25:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 7, 2022-07-20T02:25:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 24, 2022-07-20T02:25:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:25:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:25:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:25:54.400 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:25:54.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.403 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:25:54.403 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:25:54.413 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:25:54.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:25:54.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.415 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:25:54.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.436 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:25:54.436 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:25:54.447 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:25:54.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:25:54.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:25:54.448 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:25:54.519 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2022-07-20T02:25:54.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 Processor -- start 2022-07-20T02:25:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 -- start start 2022-07-20T02:25:54.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 0x7f11c0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11c0147000 0x7f11c00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c01474e0 con 0x7f11c0060f00 2022-07-20T02:25:54.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c005d3b0 con 0x7f11c0147000 2022-07-20T02:25:54.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c01431e0 con 0x7f11c005cfd0 2022-07-20T02:25:54.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11c0147000 0x7f11c00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11c0147000 0x7f11c00b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35638/0 (socket says 172.21.15.26:35638) 2022-07-20T02:25:54.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 -- 172.21.15.26:0/3745810468 learned_addr learned my addr 172.21.15.26:0/3745810468 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:54.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11bffff700 1 --1- 172.21.15.26:0/3745810468 >> v1:172.21.15.26:6789/0 conn(0x7f11c0060f00 0x7f11c013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34922/0 (socket says 172.21.15.26:34922) 2022-07-20T02:25:54.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11bf7fe700 1 --2- 172.21.15.26:0/3745810468 >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 0x7f11c0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 -- 172.21.15.26:0/3745810468 >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 msgr2=0x7f11c0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:54.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11be7fc700 1 -- 172.21.15.26:0/3745810468 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942711316 0 0) 0x7f11c01474e0 con 0x7f11c0060f00 2022-07-20T02:25:54.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 --2- 172.21.15.26:0/3745810468 >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 0x7f11c0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 -- 172.21.15.26:0/3745810468 >> v1:172.21.15.26:6789/0 conn(0x7f11c0060f00 legacy=0x7f11c013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 -- 172.21.15.26:0/3745810468 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0143480 con 0x7f11c0147000 2022-07-20T02:25:54.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11bf7fe700 1 --2- 172.21.15.26:0/3745810468 >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 0x7f11c0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:54.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11beffd700 1 --2- 172.21.15.26:0/3745810468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0147000 0x7f11c00b4730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f11b4002f70 tx=0x7f11b4006eb0).ready entity=mon.0 client_cookie=98196345f525ebcb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:54.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11be7fc700 1 -- 172.21.15.26:0/3745810468 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b400d610 con 0x7f11c0147000 2022-07-20T02:25:54.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11be7fc700 1 -- 172.21.15.26:0/3745810468 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b4000e90 con 0x7f11c0147000 2022-07-20T02:25:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11be7fc700 1 -- 172.21.15.26:0/3745810468 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b400d960 con 0x7f11c0147000 2022-07-20T02:25:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 -- 172.21.15.26:0/3745810468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0147000 msgr2=0x7f11c00b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.685+0000 7f11c4990700 1 --2- 172.21.15.26:0/3745810468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0147000 0x7f11c00b4730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f11b4002f70 tx=0x7f11b4006eb0).stop 2022-07-20T02:25:54.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- 172.21.15.26:0/3745810468 shutdown_connections 2022-07-20T02:25:54.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 --2- 172.21.15.26:0/3745810468 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0147000 0x7f11c00b4730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 --2- 172.21.15.26:0/3745810468 >> v2:172.21.15.120:3300/0 conn(0x7f11c005cfd0 0x7f11c0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- 172.21.15.26:0/3745810468 >> 172.21.15.26:0/3745810468 conn(0x7f11c00620c0 msgr2=0x7f11c005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:54.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- 172.21.15.26:0/3745810468 shutdown_connections 2022-07-20T02:25:54.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- 172.21.15.26:0/3745810468 wait complete. 2022-07-20T02:25:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 Processor -- start 2022-07-20T02:25:54.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- start start 2022-07-20T02:25:54.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 0x7f11c0156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 0x7f11c00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00b1c10 con 0x7f11c0060f00 2022-07-20T02:25:54.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0156520 con 0x7f11c00ba050 2022-07-20T02:25:54.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11c4990700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00b2db0 con 0x7f11c0147000 2022-07-20T02:25:54.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35644/0 (socket says 172.21.15.26:35644) 2022-07-20T02:25:54.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 -- 172.21.15.26:0/2765641622 learned_addr learned my addr 172.21.15.26:0/2765641622 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:54.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11beffd700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 0x7f11c0156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bffff700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 0x7f11c00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 -- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 msgr2=0x7f11c0156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 0x7f11c0156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 -- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 msgr2=0x7f11c00b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 0x7f11c00b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 -- 172.21.15.26:0/2765641622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b400b1e0 con 0x7f11c0060f00 2022-07-20T02:25:54.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11beffd700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 0x7f11c0156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:54.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bffff700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 0x7f11c00b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:54.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.689+0000 7f11bf7fe700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f11c01566b0 tx=0x7f11b0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:54.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b0011780 con 0x7f11c0060f00 2022-07-20T02:25:54.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b00118e0 con 0x7f11c0060f00 2022-07-20T02:25:54.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c00b3050 con 0x7f11c0060f00 2022-07-20T02:25:54.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b0011c40 con 0x7f11c0060f00 2022-07-20T02:25:54.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c00b24f0 con 0x7f11c0060f00 2022-07-20T02:25:54.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c0143720 con 0x7f11c0060f00 2022-07-20T02:25:54.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11b0012420 con 0x7f11c0060f00 2022-07-20T02:25:54.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 0x7f11a0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11b006bfe0 con 0x7f11c0060f00 2022-07-20T02:25:54.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.693+0000 7f11beffd700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 0x7f11a0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.697+0000 7f11beffd700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 0x7f11a0055ea0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f11b401a000 tx=0x7f11b4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:54.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.697+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11b0042d40 con 0x7f11c0060f00 2022-07-20T02:25:54.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.825+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c00574e0 con 0x7f11c0060f00 2022-07-20T02:25:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11bcff9700 1 -- 172.21.15.26:0/2765641622 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f11b0046360 con 0x7f11c0060f00 2022-07-20T02:25:54.836 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:54.836 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 msgr2=0x7f11a0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 0x7f11a0055ea0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f11b401a000 tx=0x7f11b4015000).stop 2022-07-20T02:25:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 msgr2=0x7f11c01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f11c01566b0 tx=0x7f11b0008040).stop 2022-07-20T02:25:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 shutdown_connections 2022-07-20T02:25:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11a00539e0 0x7f11a0055ea0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c00ba050 0x7f11c00b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11c0147000 0x7f11c0156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 --2- 172.21.15.26:0/2765641622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11c0060f00 0x7f11c01421d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 >> 172.21.15.26:0/2765641622 conn(0x7f11c005cfd0 msgr2=0x7f11c013a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 shutdown_connections 2022-07-20T02:25:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.829+0000 7f11c4990700 1 -- 172.21.15.26:0/2765641622 wait complete. 2022-07-20T02:25:54.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:54.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:54.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 Processor -- start 2022-07-20T02:25:54.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 -- start start 2022-07-20T02:25:54.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35d80b06d0 0x7f35d813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35d80ac8e0 0x7f35d813fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:54.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d80b1090 con 0x7f35d80a9520 2022-07-20T02:25:54.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d80b0bd0 con 0x7f35d80b06d0 2022-07-20T02:25:54.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35dc84c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d80b12c0 con 0x7f35d80ac8e0 2022-07-20T02:25:54.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35d80b06d0 0x7f35d813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35d80b06d0 0x7f35d813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35654/0 (socket says 172.21.15.26:35654) 2022-07-20T02:25:54.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 -- 172.21.15.26:0/2295159883 learned_addr learned my addr 172.21.15.26:0/2295159883 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:54.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d759e700 1 --2- 172.21.15.26:0/2295159883 >> v2:172.21.15.120:3300/0 conn(0x7f35d80ac8e0 0x7f35d813fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:54.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 -- 172.21.15.26:0/2295159883 >> v2:172.21.15.120:3300/0 conn(0x7f35d80ac8e0 msgr2=0x7f35d813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:54.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 --2- 172.21.15.26:0/2295159883 >> v2:172.21.15.120:3300/0 conn(0x7f35d80ac8e0 0x7f35d813fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 -- 172.21.15.26:0/2295159883 >> v1:172.21.15.120:6789/0 conn(0x7f35d80a9520 legacy=0x7f35d805c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.961+0000 7f35d659c700 1 -- 172.21.15.26:0/2295159883 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d80b1560 con 0x7f35d80b06d0 2022-07-20T02:25:54.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35d659c700 1 --2- 172.21.15.26:0/2295159883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d813f6e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f35cc002f70 tx=0x7f35cc006eb0).ready entity=mon.0 client_cookie=39f0582f3c9a8af6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:54.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35d5d9b700 1 -- 172.21.15.26:0/2295159883 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35cc00d610 con 0x7f35d80b06d0 2022-07-20T02:25:54.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35d5d9b700 1 -- 172.21.15.26:0/2295159883 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc000e90 con 0x7f35d80b06d0 2022-07-20T02:25:54.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35d5d9b700 1 -- 172.21.15.26:0/2295159883 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35cc00d9a0 con 0x7f35d80b06d0 2022-07-20T02:25:54.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 -- 172.21.15.26:0/2295159883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 msgr2=0x7f35d813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:54.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 --2- 172.21.15.26:0/2295159883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d813f6e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f35cc002f70 tx=0x7f35cc006eb0).stop 2022-07-20T02:25:54.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 -- 172.21.15.26:0/2295159883 shutdown_connections 2022-07-20T02:25:54.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 --2- 172.21.15.26:0/2295159883 >> v2:172.21.15.120:3300/0 conn(0x7f35d80ac8e0 0x7f35d813fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 --2- 172.21.15.26:0/2295159883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d813f6e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:54.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 -- 172.21.15.26:0/2295159883 >> 172.21.15.26:0/2295159883 conn(0x7f35d8060310 msgr2=0x7f35d80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:54.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 -- 172.21.15.26:0/2295159883 shutdown_connections 2022-07-20T02:25:54.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.965+0000 7f35dc84c700 1 -- 172.21.15.26:0/2295159883 wait complete. 2022-07-20T02:25:55.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 Processor -- start 2022-07-20T02:25:55.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- start start 2022-07-20T02:25:55.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 0x7f35d80b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d80ac8e0 0x7f35d8139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8064470 con 0x7f35d80b06d0 2022-07-20T02:25:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8063f30 con 0x7f35d80a9520 2022-07-20T02:25:55.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d80641d0 con 0x7f35d80ac8e0 2022-07-20T02:25:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35662/0 (socket says 172.21.15.26:35662) 2022-07-20T02:25:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 -- 172.21.15.26:0/4203645900 learned_addr learned my addr 172.21.15.26:0/4203645900 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:55.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d6d9d700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 0x7f35d80b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d659c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d80ac8e0 0x7f35d8139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 -- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d80ac8e0 msgr2=0x7f35d8139680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d80ac8e0 0x7f35d8139680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 -- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 msgr2=0x7f35d80b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 0x7f35d80b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 -- 172.21.15.26:0/4203645900 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc00b1e0 con 0x7f35d80b06d0 2022-07-20T02:25:55.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d6d9d700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 0x7f35d80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:55.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35d759e700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f35d000a000 tx=0x7f35d0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d0009b60 con 0x7f35d80b06d0 2022-07-20T02:25:55.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d0009cc0 con 0x7f35d80b06d0 2022-07-20T02:25:55.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d813a060 con 0x7f35d80b06d0 2022-07-20T02:25:55.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d001a660 con 0x7f35d80b06d0 2022-07-20T02:25:55.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.969+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d8067a20 con 0x7f35d80b06d0 2022-07-20T02:25:55.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d80a8a60 con 0x7f35d80b06d0 2022-07-20T02:25:55.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35d0021060 con 0x7f35d80b06d0 2022-07-20T02:25:55.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35c7fff700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 0x7f35b4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f35d006eb70 con 0x7f35d80b06d0 2022-07-20T02:25:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35d6d9d700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 0x7f35b4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.973+0000 7f35d6d9d700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 0x7f35b4055e40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f35c800b000 tx=0x7f35c8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:54.981+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35d80a8a60 con 0x7f35d80b06d0 2022-07-20T02:25:55.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.105+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f35d80a8a60 con 0x7f35d80b06d0 2022-07-20T02:25:55.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.105+0000 7f35c7fff700 1 -- 172.21.15.26:0/4203645900 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f35d80a8a60 con 0x7f35d80b06d0 2022-07-20T02:25:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:55.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 msgr2=0x7f35b4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 0x7f35b4055e40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f35c800b000 tx=0x7f35c8009040).stop 2022-07-20T02:25:55.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 msgr2=0x7f35d8139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f35d000a000 tx=0x7f35d0008040).stop 2022-07-20T02:25:55.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 shutdown_connections 2022-07-20T02:25:55.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35d80b06d0 0x7f35d8139b70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35b4053980 0x7f35b4055e40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d80ac8e0 0x7f35d8139680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 --2- 172.21.15.26:0/4203645900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d80a9520 0x7f35d80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 >> 172.21.15.26:0/4203645900 conn(0x7f35d8060310 msgr2=0x7f35d813ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 shutdown_connections 2022-07-20T02:25:55.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.109+0000 7f35dc84c700 1 -- 172.21.15.26:0/4203645900 wait complete. 2022-07-20T02:25:55.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:55.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 Processor -- start 2022-07-20T02:25:55.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 -- start start 2022-07-20T02:25:55.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13c0064990 0x7f13c0064d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f13c01432e0 0x7f13c0140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c00b2cd0 con 0x7f13c00b27a0 2022-07-20T02:25:55.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c00608d0 con 0x7f13c01432e0 2022-07-20T02:25:55.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c7571700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c00b3860 con 0x7f13c0064990 2022-07-20T02:25:55.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c4b0c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f13c00b27a0 0x7f13c0060d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34948/0 (socket says 172.21.15.26:34948) 2022-07-20T02:25:55.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c4b0c700 1 -- 172.21.15.26:0/3319867823 learned_addr learned my addr 172.21.15.26:0/3319867823 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:55.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/3319867823 >> v2:172.21.15.26:3300/0 conn(0x7f13c01432e0 0x7f13c0140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13c530d700 1 --2- 172.21.15.26:0/3319867823 >> v2:172.21.15.120:3300/0 conn(0x7f13c0064990 0x7f13c0064d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.229+0000 7f13affff700 1 -- 172.21.15.26:0/3319867823 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547874738 0 0) 0x7f13c00b2cd0 con 0x7f13c00b27a0 2022-07-20T02:25:55.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13affff700 1 -- 172.21.15.26:0/3319867823 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8003260 con 0x7f13c00b27a0 2022-07-20T02:25:55.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c5b0e700 1 -- 172.21.15.26:0/3319867823 >> v2:172.21.15.120:3300/0 conn(0x7f13c0064990 msgr2=0x7f13c0064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:55.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/3319867823 >> v2:172.21.15.120:3300/0 conn(0x7f13c0064990 0x7f13c0064d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c5b0e700 1 -- 172.21.15.26:0/3319867823 >> v1:172.21.15.26:6789/0 conn(0x7f13c00b27a0 legacy=0x7f13c0060d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c5b0e700 1 -- 172.21.15.26:0/3319867823 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00b2f00 con 0x7f13c01432e0 2022-07-20T02:25:55.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/3319867823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c0140680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f13c00b8920 tx=0x7f13b80091d0).ready entity=mon.0 client_cookie=dfeb9078c5092427 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13affff700 1 -- 172.21.15.26:0/3319867823 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b800b5a0 con 0x7f13c01432e0 2022-07-20T02:25:55.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13affff700 1 -- 172.21.15.26:0/3319867823 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b800b700 con 0x7f13c01432e0 2022-07-20T02:25:55.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13affff700 1 -- 172.21.15.26:0/3319867823 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b800b870 con 0x7f13c01432e0 2022-07-20T02:25:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- 172.21.15.26:0/3319867823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 msgr2=0x7f13c0140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 --2- 172.21.15.26:0/3319867823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c0140680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f13c00b8920 tx=0x7f13b80091d0).stop 2022-07-20T02:25:55.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- 172.21.15.26:0/3319867823 shutdown_connections 2022-07-20T02:25:55.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 --2- 172.21.15.26:0/3319867823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c0140680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 --2- 172.21.15.26:0/3319867823 >> v2:172.21.15.120:3300/0 conn(0x7f13c0064990 0x7f13c0064d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- 172.21.15.26:0/3319867823 >> 172.21.15.26:0/3319867823 conn(0x7f13c005c580 msgr2=0x7f13c005c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- 172.21.15.26:0/3319867823 shutdown_connections 2022-07-20T02:25:55.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- 172.21.15.26:0/3319867823 wait complete. 2022-07-20T02:25:55.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 Processor -- start 2022-07-20T02:25:55.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.233+0000 7f13c7571700 1 -- start start 2022-07-20T02:25:55.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13c0064990 0x7f13c00aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13c00b27a0 0x7f13c0140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c0155ba0 con 0x7f13c01432e0 2022-07-20T02:25:55.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c0155e40 con 0x7f13c00b27a0 2022-07-20T02:25:55.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c01560e0 con 0x7f13c0064990 2022-07-20T02:25:55.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35676/0 (socket says 172.21.15.26:35676) 2022-07-20T02:25:55.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 -- 172.21.15.26:0/4088576347 learned_addr learned my addr 172.21.15.26:0/4088576347 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:55.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c530d700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13c0064990 0x7f13c00aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c4b0c700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13c00b27a0 0x7f13c0140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 -- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13c0064990 msgr2=0x7f13c00aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13c0064990 0x7f13c00aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 -- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13c00b27a0 msgr2=0x7f13c0140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13c00b27a0 0x7f13c0140680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 -- 172.21.15.26:0/4088576347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b8009470 con 0x7f13c01432e0 2022-07-20T02:25:55.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c5b0e700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f13c00b8920 tx=0x7f13b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b8011c10 con 0x7f13c01432e0 2022-07-20T02:25:55.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b8011d70 con 0x7f13c01432e0 2022-07-20T02:25:55.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c0156380 con 0x7f13c01432e0 2022-07-20T02:25:55.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b801f5e0 con 0x7f13c01432e0 2022-07-20T02:25:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.237+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c0156820 con 0x7f13c01432e0 2022-07-20T02:25:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c0156510 con 0x7f13c01432e0 2022-07-20T02:25:55.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f13b8026030 con 0x7f13c01432e0 2022-07-20T02:25:55.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13ae7fc700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 0x7f13a0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f13b806ca50 con 0x7f13c01432e0 2022-07-20T02:25:55.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13c530d700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 0x7f13a0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.241+0000 7f13c530d700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 0x7f13a0055f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f13bc00b000 tx=0x7f13bc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.249+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13b8043790 con 0x7f13c01432e0 2022-07-20T02:25:55.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.373+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f13c00574e0 con 0x7f13c01432e0 2022-07-20T02:25:55.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.373+0000 7f13ae7fc700 1 -- 172.21.15.26:0/4088576347 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f13b80150a0 con 0x7f13c01432e0 2022-07-20T02:25:55.381 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:55.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 msgr2=0x7f13a0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 0x7f13a0055f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f13bc00b000 tx=0x7f13bc006df0).stop 2022-07-20T02:25:55.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 msgr2=0x7f13c01524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f13c00b8920 tx=0x7f13b8016000).stop 2022-07-20T02:25:55.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 shutdown_connections 2022-07-20T02:25:55.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13c01432e0 0x7f13c01524d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f13a0053a70 0x7f13a0055f30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13c00b27a0 0x7f13c0140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 --2- 172.21.15.26:0/4088576347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13c0064990 0x7f13c00aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 >> 172.21.15.26:0/4088576347 conn(0x7f13c005c580 msgr2=0x7f13c00b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 shutdown_connections 2022-07-20T02:25:55.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.377+0000 7f13c7571700 1 -- 172.21.15.26:0/4088576347 wait complete. 2022-07-20T02:25:55.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 Processor -- start 2022-07-20T02:25:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 -- start start 2022-07-20T02:25:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f548c05cfd0 0x7f548c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 0x7f548c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c1474e0 con 0x7f548c147000 2022-07-20T02:25:55.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548c0b4e40 con 0x7f548c05cfd0 2022-07-20T02:25:55.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491839700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c05d3b0 con 0x7f548c060f00 2022-07-20T02:25:55.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491038700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 0x7f548c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f548bfff700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f548c147000 0x7f548c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49174/0 (socket says 172.21.15.26:49174) 2022-07-20T02:25:55.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f548bfff700 1 -- 172.21.15.26:0/4129868066 learned_addr learned my addr 172.21.15.26:0/4129868066 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:55.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5490837700 1 --2- 172.21.15.26:0/4129868066 >> v2:172.21.15.120:3300/0 conn(0x7f548c05cfd0 0x7f548c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f548b7fe700 1 -- 172.21.15.26:0/4129868066 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385317777 0 0) 0x7f548c1474e0 con 0x7f548c147000 2022-07-20T02:25:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f548b7fe700 1 -- 172.21.15.26:0/4129868066 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478003260 con 0x7f548c147000 2022-07-20T02:25:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491038700 1 -- 172.21.15.26:0/4129868066 >> v2:172.21.15.120:3300/0 conn(0x7f548c05cfd0 msgr2=0x7f548c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491038700 1 --2- 172.21.15.26:0/4129868066 >> v2:172.21.15.120:3300/0 conn(0x7f548c05cfd0 0x7f548c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.517+0000 7f5491038700 1 -- 172.21.15.26:0/4129868066 >> v1:172.21.15.120:6789/0 conn(0x7f548c147000 legacy=0x7f548c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491038700 1 -- 172.21.15.26:0/4129868066 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c1431e0 con 0x7f548c060f00 2022-07-20T02:25:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491038700 1 --2- 172.21.15.26:0/4129868066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 0x7f548c13c9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f54840032f0 tx=0x7f54840091d0).ready entity=mon.2 client_cookie=b2d5c4201529d5e1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f548b7fe700 1 -- 172.21.15.26:0/4129868066 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f548400b600 con 0x7f548c060f00 2022-07-20T02:25:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f548b7fe700 1 -- 172.21.15.26:0/4129868066 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548400b760 con 0x7f548c060f00 2022-07-20T02:25:55.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f548b7fe700 1 -- 172.21.15.26:0/4129868066 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f548400b8d0 con 0x7f548c060f00 2022-07-20T02:25:55.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- 172.21.15.26:0/4129868066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 msgr2=0x7f548c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- 172.21.15.26:0/4129868066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 0x7f548c13c9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f54840032f0 tx=0x7f54840091d0).stop 2022-07-20T02:25:55.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- 172.21.15.26:0/4129868066 shutdown_connections 2022-07-20T02:25:55.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- 172.21.15.26:0/4129868066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c060f00 0x7f548c13c9b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- 172.21.15.26:0/4129868066 >> v2:172.21.15.120:3300/0 conn(0x7f548c05cfd0 0x7f548c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- 172.21.15.26:0/4129868066 >> 172.21.15.26:0/4129868066 conn(0x7f548c0620c0 msgr2=0x7f548c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- 172.21.15.26:0/4129868066 shutdown_connections 2022-07-20T02:25:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- 172.21.15.26:0/4129868066 wait complete. 2022-07-20T02:25:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 Processor -- start 2022-07-20T02:25:55.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- start start 2022-07-20T02:25:55.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c147000 0x7f548c0af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c0af8e0 0x7f548c155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.521+0000 7f5491839700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c0ae200 con 0x7f548c060f00 2022-07-20T02:25:55.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491839700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c0afd70 con 0x7f548c147000 2022-07-20T02:25:55.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491839700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c144a30 con 0x7f548c0af8e0 2022-07-20T02:25:55.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491038700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c0af8e0 0x7f548c155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f548bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c147000 0x7f548c0af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35686/0 (socket says 172.21.15.26:35686) 2022-07-20T02:25:55.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 -- 172.21.15.26:0/2933661920 learned_addr learned my addr 172.21.15.26:0/2933661920 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:55.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 -- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c0af8e0 msgr2=0x7f548c155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c0af8e0 0x7f548c155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 -- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c147000 msgr2=0x7f548c0af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c147000 0x7f548c0af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 -- 172.21.15.26:0/2933661920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484009470 con 0x7f548c060f00 2022-07-20T02:25:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5490837700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f548000c000 tx=0x7f5480006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5480011750 con 0x7f548c060f00 2022-07-20T02:25:55.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54800118b0 con 0x7f548c060f00 2022-07-20T02:25:55.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c0b00e0 con 0x7f548c060f00 2022-07-20T02:25:55.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5480011c10 con 0x7f548c060f00 2022-07-20T02:25:55.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c0b0620 con 0x7f548c060f00 2022-07-20T02:25:55.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c143480 con 0x7f548c060f00 2022-07-20T02:25:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.525+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5480011d70 con 0x7f548c060f00 2022-07-20T02:25:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.529+0000 7f5489ffb700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 0x7f546c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.529+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f548006c450 con 0x7f548c060f00 2022-07-20T02:25:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.529+0000 7f548bfff700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 0x7f546c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.529+0000 7f548bfff700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 0x7f546c055fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f547c00d000 tx=0x7f547c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.537+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54800431b0 con 0x7f548c060f00 2022-07-20T02:25:55.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.657+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f548c0574e0 con 0x7f548c060f00 2022-07-20T02:25:55.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.657+0000 7f5489ffb700 1 -- 172.21.15.26:0/2933661920 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f54800467d0 con 0x7f548c060f00 2022-07-20T02:25:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 msgr2=0x7f546c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 0x7f546c055fc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f547c00d000 tx=0x7f547c009040).stop 2022-07-20T02:25:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 msgr2=0x7f548c1463d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f548000c000 tx=0x7f5480006df0).stop 2022-07-20T02:25:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 shutdown_connections 2022-07-20T02:25:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f546c053b00 0x7f546c055fc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c0af8e0 0x7f548c155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c147000 0x7f548c0af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 --2- 172.21.15.26:0/2933661920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c060f00 0x7f548c1463d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 >> 172.21.15.26:0/2933661920 conn(0x7f548c05cfd0 msgr2=0x7f548c0b97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 shutdown_connections 2022-07-20T02:25:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.661+0000 7f5491839700 1 -- 172.21.15.26:0/2933661920 wait complete. 2022-07-20T02:25:55.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:55.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 Processor -- start 2022-07-20T02:25:55.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 -- start start 2022-07-20T02:25:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9c405cfd0 0x7fa9c40aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c405d3b0 con 0x7fa9c4142fa0 2022-07-20T02:25:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c40b1a60 con 0x7fa9c40b1cb0 2022-07-20T02:25:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9cb2ac700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40b2140 con 0x7fa9c405cfd0 2022-07-20T02:25:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9ca2aa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9c405cfd0 0x7fa9c40aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9caaab700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9c4142fa0 0x7fa9c4143810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34976/0 (socket says 172.21.15.26:34976) 2022-07-20T02:25:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9ca2aa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9c405cfd0 0x7fa9c40aaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35694/0 (socket says 172.21.15.26:35694) 2022-07-20T02:25:55.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9caaab700 1 -- 172.21.15.26:0/159971653 learned_addr learned my addr 172.21.15.26:0/159971653 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:55.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/159971653 >> v1:172.21.15.120:6789/0 conn(0x7fa9c40b1cb0 legacy=0x7fa9c4142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/159971653 >> v1:172.21.15.26:6789/0 conn(0x7fa9c4142fa0 legacy=0x7fa9c4143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.789+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/159971653 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40b1520 con 0x7fa9c405cfd0 2022-07-20T02:25:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9ca2aa700 1 --2- 172.21.15.26:0/159971653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c40aaf70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa9c00032f0 tx=0x7fa9c00091d0).ready entity=mon.0 client_cookie=fa928c0a6c09adb2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9c92a8700 1 -- 172.21.15.26:0/159971653 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c0009ea0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9c92a8700 1 -- 172.21.15.26:0/159971653 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c000b8a0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9c92a8700 1 -- 172.21.15.26:0/159971653 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c000ba10 con 0x7fa9c405cfd0 2022-07-20T02:25:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/159971653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 msgr2=0x7fa9c40aaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/159971653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c40aaf70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa9c00032f0 tx=0x7fa9c00091d0).stop 2022-07-20T02:25:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/159971653 shutdown_connections 2022-07-20T02:25:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/159971653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c40aaf70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/159971653 >> 172.21.15.26:0/159971653 conn(0x7fa9c40b7090 msgr2=0x7fa9c405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/159971653 shutdown_connections 2022-07-20T02:25:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.793+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/159971653 wait complete. 2022-07-20T02:25:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 Processor -- start 2022-07-20T02:25:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 -- start start 2022-07-20T02:25:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9c40b1cb0 0x7fa9c413d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9c4142fa0 0x7fa9c413d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4060120 con 0x7fa9c405cfd0 2022-07-20T02:25:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c405f9c0 con 0x7fa9c40b1cb0 2022-07-20T02:25:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9cb2ac700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4066b30 con 0x7fa9c4142fa0 2022-07-20T02:25:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35700/0 (socket says 172.21.15.26:35700) 2022-07-20T02:25:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/1351004399 learned_addr learned my addr 172.21.15.26:0/1351004399 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9c9aa9700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9c40b1cb0 0x7fa9c413d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9caaab700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9c4142fa0 0x7fa9c413d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9c4142fa0 msgr2=0x7fa9c413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9c4142fa0 0x7fa9c413d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9c40b1cb0 msgr2=0x7fa9c413d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9c40b1cb0 0x7fa9c413d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 -- 172.21.15.26:0/1351004399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0009470 con 0x7fa9c405cfd0 2022-07-20T02:25:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.797+0000 7fa9ca2aa700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa9c00032f0 tx=0x7fa9c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c0009ea0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c001f3f0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c413db90 con 0x7fa9c405cfd0 2022-07-20T02:25:55.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c001f750 con 0x7fa9c405cfd0 2022-07-20T02:25:55.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c4060590 con 0x7fa9c405cfd0 2022-07-20T02:25:55.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40b17c0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9c000f070 con 0x7fa9c405cfd0 2022-07-20T02:25:55.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.801+0000 7fa9b37fe700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 0x7fa9a4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:55.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.805+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa9c006caa0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.805+0000 7fa9c9aa9700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 0x7fa9a4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:55.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.805+0000 7fa9c9aa9700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 0x7fa9a4055fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa9b400d000 tx=0x7fa9b4007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:55.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.809+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9c00436d0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9c40574e0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9b37fe700 1 -- 172.21.15.26:0/1351004399 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa9c00150a0 con 0x7fa9c405cfd0 2022-07-20T02:25:55.943 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:55.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 msgr2=0x7fa9a4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 0x7fa9a4055fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa9b400d000 tx=0x7fa9b4007b90).stop 2022-07-20T02:25:55.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 msgr2=0x7fa9c41407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:55.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.937+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa9c00032f0 tx=0x7fa9c0016000).stop 2022-07-20T02:25:55.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 shutdown_connections 2022-07-20T02:25:55.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9a4053b00 0x7fa9a4055fc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9c4142fa0 0x7fa9c413d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9c40b1cb0 0x7fa9c413d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 --2- 172.21.15.26:0/1351004399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9c405cfd0 0x7fa9c41407b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:55.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 >> 172.21.15.26:0/1351004399 conn(0x7fa9c40b7090 msgr2=0x7fa9c40b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:55.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 shutdown_connections 2022-07-20T02:25:55.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:55.941+0000 7fa9cb2ac700 1 -- 172.21.15.26:0/1351004399 wait complete. 2022-07-20T02:25:55.962 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:56.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 Processor -- start 2022-07-20T02:25:56.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 -- start start 2022-07-20T02:25:56.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 0x7f0854064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854064880 con 0x7f08540670c0 2022-07-20T02:25:56.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08540b85f0 con 0x7f08541444a0 2022-07-20T02:25:56.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f085956c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08540b35a0 con 0x7f0854064450 2022-07-20T02:25:56.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f0858d6b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 0x7f0854064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f0858d6b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35712/0 (socket says 172.21.15.26:35712) 2022-07-20T02:25:56.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f0853fff700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f08540670c0 0x7f08540b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49200/0 (socket says 172.21.15.26:49200) 2022-07-20T02:25:56.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f0858d6b700 1 -- 172.21.15.26:0/1013155003 learned_addr learned my addr 172.21.15.26:0/1013155003 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f08537fe700 1 -- 172.21.15.26:0/1013155003 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270084257 0 0) 0x7f0854064880 con 0x7f08540670c0 2022-07-20T02:25:56.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f08537fe700 1 -- 172.21.15.26:0/1013155003 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0834003260 con 0x7f08540670c0 2022-07-20T02:25:56.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 -- 172.21.15.26:0/1013155003 >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 msgr2=0x7f0854065440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:56.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 --2- 172.21.15.26:0/1013155003 >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 -- 172.21.15.26:0/1013155003 >> v1:172.21.15.120:6789/0 conn(0x7f08540670c0 legacy=0x7f08540b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 -- 172.21.15.26:0/1013155003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540b80b0 con 0x7f0854064450 2022-07-20T02:25:56.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f0858d6b700 1 --2- 172.21.15.26:0/1013155003 >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:56.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f084b7fe700 1 --2- 172.21.15.26:0/1013155003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 0x7f0854064f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f083c002f70 tx=0x7f083c006df0).ready entity=mon.2 client_cookie=67f4332302c363fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f08537fe700 1 -- 172.21.15.26:0/1013155003 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f083c00d4a0 con 0x7f0854064450 2022-07-20T02:25:56.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f08537fe700 1 -- 172.21.15.26:0/1013155003 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c000e90 con 0x7f0854064450 2022-07-20T02:25:56.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.065+0000 7f08537fe700 1 -- 172.21.15.26:0/1013155003 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f083c00d7f0 con 0x7f0854064450 2022-07-20T02:25:56.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- 172.21.15.26:0/1013155003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 msgr2=0x7f0854064f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 --2- 172.21.15.26:0/1013155003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 0x7f0854064f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f083c002f70 tx=0x7f083c006df0).stop 2022-07-20T02:25:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- 172.21.15.26:0/1013155003 shutdown_connections 2022-07-20T02:25:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 --2- 172.21.15.26:0/1013155003 >> v2:172.21.15.26:3300/0 conn(0x7f08541444a0 0x7f0854065440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 --2- 172.21.15.26:0/1013155003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0854064450 0x7f0854064f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- 172.21.15.26:0/1013155003 >> 172.21.15.26:0/1013155003 conn(0x7f085405f840 msgr2=0x7f085405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- 172.21.15.26:0/1013155003 shutdown_connections 2022-07-20T02:25:56.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- 172.21.15.26:0/1013155003 wait complete. 2022-07-20T02:25:56.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 Processor -- start 2022-07-20T02:25:56.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.069+0000 7f085956c700 1 -- start start 2022-07-20T02:25:56.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08541444a0 0x7f08540b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08540b12b0 0x7f085413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08540ae3b0 con 0x7f08540670c0 2022-07-20T02:25:56.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0854141180 con 0x7f08541444a0 2022-07-20T02:25:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08541419f0 con 0x7f08540b12b0 2022-07-20T02:25:56.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f084b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08541444a0 0x7f08540b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35714/0 (socket says 172.21.15.26:35714) 2022-07-20T02:25:56.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 -- 172.21.15.26:0/4121604460 learned_addr learned my addr 172.21.15.26:0/4121604460 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0858d6b700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08540b12b0 0x7f085413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 -- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08540b12b0 msgr2=0x7f085413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08540b12b0 0x7f085413ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 -- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08541444a0 msgr2=0x7f08540b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08541444a0 0x7f08540b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 -- 172.21.15.26:0/4121604460 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c009530 con 0x7f08540670c0 2022-07-20T02:25:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0853fff700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f08540b0a00 tx=0x7f0844007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08440070d0 con 0x7f08540670c0 2022-07-20T02:25:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844007230 con 0x7f08540670c0 2022-07-20T02:25:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c00b180 con 0x7f08540670c0 2022-07-20T02:25:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0844005740 con 0x7f08540670c0 2022-07-20T02:25:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08540b1c20 con 0x7f08540670c0 2022-07-20T02:25:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.073+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08540b0ab0 con 0x7f08540670c0 2022-07-20T02:25:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.077+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0844010070 con 0x7f08540670c0 2022-07-20T02:25:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.077+0000 7f0851ffb700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 0x7f0838055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.077+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f084406b780 con 0x7f08540670c0 2022-07-20T02:25:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.077+0000 7f084b7fe700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 0x7f0838055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.077+0000 7f084b7fe700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 0x7f0838055e10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f083c01a000 tx=0x7f083c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.081+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0844042540 con 0x7f08540670c0 2022-07-20T02:25:56.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.209+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08540b8350 con 0x7f08540670c0 2022-07-20T02:25:56.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f0851ffb700 1 -- 172.21.15.26:0/4121604460 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f0844045b60 con 0x7f08540670c0 2022-07-20T02:25:56.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:56.220 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:56.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 msgr2=0x7f0838055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 0x7f0838055e10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f083c01a000 tx=0x7f083c014000).stop 2022-07-20T02:25:56.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 msgr2=0x7f08540ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f08540b0a00 tx=0x7f0844007b00).stop 2022-07-20T02:25:56.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 shutdown_connections 2022-07-20T02:25:56.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0838053950 0x7f0838055e10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08540b12b0 0x7f085413ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08541444a0 0x7f08540b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 --2- 172.21.15.26:0/4121604460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08540670c0 0x7f08540ad8c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 >> 172.21.15.26:0/4121604460 conn(0x7f085405f840 msgr2=0x7f0854143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 shutdown_connections 2022-07-20T02:25:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.213+0000 7f085956c700 1 -- 172.21.15.26:0/4121604460 wait complete. 2022-07-20T02:25:56.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:56.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.341+0000 7f0d35e93700 1 Processor -- start 2022-07-20T02:25:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.341+0000 7f0d35e93700 1 -- start start 2022-07-20T02:25:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d300b5ef0 0x7f0d301474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0d300a8c40 0x7f0d300af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d300b62d0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3014aac0 con 0x7f0d300a8c40 2022-07-20T02:25:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3014ad60 con 0x7f0d300b5ef0 2022-07-20T02:25:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0d300a8c40 0x7f0d300af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0d300a8c40 0x7f0d300af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35724/0 (socket says 172.21.15.26:35724) 2022-07-20T02:25:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 -- 172.21.15.26:0/3851785287 learned_addr learned my addr 172.21.15.26:0/3851785287 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d34e91700 1 --2- 172.21.15.26:0/3851785287 >> v2:172.21.15.120:3300/0 conn(0x7f0d300b5ef0 0x7f0d301474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 -- 172.21.15.26:0/3851785287 >> v2:172.21.15.120:3300/0 conn(0x7f0d300b5ef0 msgr2=0x7f0d301474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 --2- 172.21.15.26:0/3851785287 >> v2:172.21.15.120:3300/0 conn(0x7f0d300b5ef0 0x7f0d301474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 -- 172.21.15.26:0/3851785287 >> v1:172.21.15.120:6789/0 conn(0x7f0d300a7fc0 legacy=0x7f0d300aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 -- 172.21.15.26:0/3851785287 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3014b000 con 0x7f0d300a8c40 2022-07-20T02:25:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35692700 1 --2- 172.21.15.26:0/3851785287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a8c40 0x7f0d300af550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0d28002f70 tx=0x7f0d28006eb0).ready entity=mon.0 client_cookie=3e9418cedf81dc53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d2f7fe700 1 -- 172.21.15.26:0/3851785287 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2800d5c0 con 0x7f0d300a8c40 2022-07-20T02:25:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d2f7fe700 1 -- 172.21.15.26:0/3851785287 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d28000e90 con 0x7f0d300a8c40 2022-07-20T02:25:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d2f7fe700 1 -- 172.21.15.26:0/3851785287 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2800d950 con 0x7f0d300a8c40 2022-07-20T02:25:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- 172.21.15.26:0/3851785287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a8c40 msgr2=0x7f0d300af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 --2- 172.21.15.26:0/3851785287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a8c40 0x7f0d300af550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0d28002f70 tx=0x7f0d28006eb0).stop 2022-07-20T02:25:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- 172.21.15.26:0/3851785287 shutdown_connections 2022-07-20T02:25:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 --2- 172.21.15.26:0/3851785287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a8c40 0x7f0d300af550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 --2- 172.21.15.26:0/3851785287 >> v2:172.21.15.120:3300/0 conn(0x7f0d300b5ef0 0x7f0d301474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- 172.21.15.26:0/3851785287 >> 172.21.15.26:0/3851785287 conn(0x7f0d30060310 msgr2=0x7f0d300b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- 172.21.15.26:0/3851785287 shutdown_connections 2022-07-20T02:25:56.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.345+0000 7f0d35e93700 1 -- 172.21.15.26:0/3851785287 wait complete. 2022-07-20T02:25:56.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 Processor -- start 2022-07-20T02:25:56.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- start start 2022-07-20T02:25:56.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d300a8c40 0x7f0d3015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d300b5ef0 0x7f0d30144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d300ab3d0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d301461d0 con 0x7f0d300b5ef0 2022-07-20T02:25:56.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3015a770 con 0x7f0d300a8c40 2022-07-20T02:25:56.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35728/0 (socket says 172.21.15.26:35728) 2022-07-20T02:25:56.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 -- 172.21.15.26:0/1642569349 learned_addr learned my addr 172.21.15.26:0/1642569349 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35692700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d300b5ef0 0x7f0d30144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d2ffff700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d300a8c40 0x7f0d3015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 -- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d300a8c40 msgr2=0x7f0d3015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d300a8c40 0x7f0d3015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 -- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d300b5ef0 msgr2=0x7f0d30144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d300b5ef0 0x7f0d30144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 -- 172.21.15.26:0/1642569349 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2800b1e0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d34e91700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d20006e00 tx=0x7f0d200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2000c610 con 0x7f0d300a7fc0 2022-07-20T02:25:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2000c770 con 0x7f0d300a7fc0 2022-07-20T02:25:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d300ab600 con 0x7f0d300a7fc0 2022-07-20T02:25:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2000cad0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.349+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3015aa10 con 0x7f0d300a7fc0 2022-07-20T02:25:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d300ab790 con 0x7f0d300a7fc0 2022-07-20T02:25:56.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0d20022070 con 0x7f0d300a7fc0 2022-07-20T02:25:56.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d2dffb700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 0x7f0d14055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0d2006c6e0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d2ffff700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 0x7f0d14055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.353+0000 7f0d2ffff700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 0x7f0d14055ed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0d1800d000 tx=0x7f0d18007b50).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.357+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d20043390 con 0x7f0d300a7fc0 2022-07-20T02:25:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.480+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0d3014b2a0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.480+0000 7f0d2dffb700 1 -- 172.21.15.26:0/1642569349 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0d200160b0 con 0x7f0d300a7fc0 2022-07-20T02:25:56.488 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 msgr2=0x7f0d14055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 0x7f0d14055ed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0d1800d000 tx=0x7f0d18007b50).stop 2022-07-20T02:25:56.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 msgr2=0x7f0d30159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d20006e00 tx=0x7f0d200091d0).stop 2022-07-20T02:25:56.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 shutdown_connections 2022-07-20T02:25:56.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0d14053a10 0x7f0d14055ed0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d300b5ef0 0x7f0d30144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d300a8c40 0x7f0d3015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 --2- 172.21.15.26:0/1642569349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d300a7fc0 0x7f0d30159be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 >> 172.21.15.26:0/1642569349 conn(0x7f0d30060310 msgr2=0x7f0d3014a270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 shutdown_connections 2022-07-20T02:25:56.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.484+0000 7f0d35e93700 1 -- 172.21.15.26:0/1642569349 wait complete. 2022-07-20T02:25:56.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:56.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.600+0000 7f8a862be700 1 Processor -- start 2022-07-20T02:25:56.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.600+0000 7f8a862be700 1 -- start start 2022-07-20T02:25:56.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a8005cfd0 0x7f8a800b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a8005f840 0x7f8a80144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800b27a0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8005d3b0 con 0x7f8a8005f840 2022-07-20T02:25:56.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800b29d0 con 0x7f8a8005cfd0 2022-07-20T02:25:56.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a8005f840 0x7f8a80144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a8005f840 0x7f8a80144bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35736/0 (socket says 172.21.15.26:35736) 2022-07-20T02:25:56.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 -- 172.21.15.26:0/3037750232 learned_addr learned my addr 172.21.15.26:0/3037750232 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a852bc700 1 --2- 172.21.15.26:0/3037750232 >> v2:172.21.15.120:3300/0 conn(0x7f8a8005cfd0 0x7f8a800b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 -- 172.21.15.26:0/3037750232 >> v2:172.21.15.120:3300/0 conn(0x7f8a8005cfd0 msgr2=0x7f8a800b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:56.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 --2- 172.21.15.26:0/3037750232 >> v2:172.21.15.120:3300/0 conn(0x7f8a8005cfd0 0x7f8a800b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 -- 172.21.15.26:0/3037750232 >> v1:172.21.15.120:6789/0 conn(0x7f8a800a9ed0 legacy=0x7f8a800ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 -- 172.21.15.26:0/3037750232 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800b2c70 con 0x7f8a8005f840 2022-07-20T02:25:56.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a84abb700 1 --2- 172.21.15.26:0/3037750232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a8005f840 0x7f8a80144bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8a7c002f70 tx=0x7f8a7c006eb0).ready entity=mon.0 client_cookie=f620ea8e7d8e874b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a73fff700 1 -- 172.21.15.26:0/3037750232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7c00d610 con 0x7f8a8005f840 2022-07-20T02:25:56.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a73fff700 1 -- 172.21.15.26:0/3037750232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c000e90 con 0x7f8a8005f840 2022-07-20T02:25:56.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a73fff700 1 -- 172.21.15.26:0/3037750232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7c00d9a0 con 0x7f8a8005f840 2022-07-20T02:25:56.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- 172.21.15.26:0/3037750232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a8005f840 msgr2=0x7f8a80144bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 --2- 172.21.15.26:0/3037750232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a8005f840 0x7f8a80144bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8a7c002f70 tx=0x7f8a7c006eb0).stop 2022-07-20T02:25:56.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- 172.21.15.26:0/3037750232 shutdown_connections 2022-07-20T02:25:56.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 --2- 172.21.15.26:0/3037750232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a8005f840 0x7f8a80144bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 --2- 172.21.15.26:0/3037750232 >> v2:172.21.15.120:3300/0 conn(0x7f8a8005cfd0 0x7f8a800b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- 172.21.15.26:0/3037750232 >> 172.21.15.26:0/3037750232 conn(0x7f8a800b7090 msgr2=0x7f8a8005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- 172.21.15.26:0/3037750232 shutdown_connections 2022-07-20T02:25:56.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.604+0000 7f8a862be700 1 -- 172.21.15.26:0/3037750232 wait complete. 2022-07-20T02:25:56.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 Processor -- start 2022-07-20T02:25:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 -- start start 2022-07-20T02:25:56.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 0x7f8a800acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8005f840 0x7f8a801535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80066370 con 0x7f8a800a9ed0 2022-07-20T02:25:56.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800657b0 con 0x7f8a8005cfd0 2022-07-20T02:25:56.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80065a50 con 0x7f8a8005f840 2022-07-20T02:25:56.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35746/0 (socket says 172.21.15.26:35746) 2022-07-20T02:25:56.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 -- 172.21.15.26:0/3717579966 learned_addr learned my addr 172.21.15.26:0/3717579966 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a84abb700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8005f840 0x7f8a801535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a852bc700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 0x7f8a800acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 -- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8005f840 msgr2=0x7f8a801535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8005f840 0x7f8a801535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 -- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 msgr2=0x7f8a800acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 0x7f8a800acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 -- 172.21.15.26:0/3717579966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c00b1e0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a852bc700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 0x7f8a800acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:56.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a85abd700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8a800b7640 tx=0x7f8a78007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7800e4a0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7800e600 con 0x7f8a800a9ed0 2022-07-20T02:25:56.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a80064a70 con 0x7f8a800a9ed0 2022-07-20T02:25:56.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.608+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a7800e960 con 0x7f8a800a9ed0 2022-07-20T02:25:56.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a80153fd0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80065c10 con 0x7f8a800a9ed0 2022-07-20T02:25:56.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a78019030 con 0x7f8a800a9ed0 2022-07-20T02:25:56.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a727fc700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 0x7f8a68055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8a7806bd20 con 0x7f8a800a9ed0 2022-07-20T02:25:56.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.612+0000 7f8a852bc700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 0x7f8a68055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.616+0000 7f8a852bc700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 0x7f8a68055ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8a7400c000 tx=0x7f8a74008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.620+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a780429d0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.740+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8a80144730 con 0x7f8a800a9ed0 2022-07-20T02:25:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.740+0000 7f8a727fc700 1 -- 172.21.15.26:0/3717579966 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8a78045ff0 con 0x7f8a800a9ed0 2022-07-20T02:25:56.751 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 msgr2=0x7f8a68055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 0x7f8a68055ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8a7400c000 tx=0x7f8a74008040).stop 2022-07-20T02:25:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 msgr2=0x7f8a80153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8a800b7640 tx=0x7f8a78007f40).stop 2022-07-20T02:25:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 shutdown_connections 2022-07-20T02:25:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a800a9ed0 0x7f8a80153ae0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a68053a10 0x7f8a68055ed0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a8005f840 0x7f8a801535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 --2- 172.21.15.26:0/3717579966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a8005cfd0 0x7f8a800acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 >> 172.21.15.26:0/3717579966 conn(0x7f8a800b7090 msgr2=0x7f8a800b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 shutdown_connections 2022-07-20T02:25:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.744+0000 7f8a862be700 1 -- 172.21.15.26:0/3717579966 wait complete. 2022-07-20T02:25:56.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:56.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 Processor -- start 2022-07-20T02:25:56.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- start start 2022-07-20T02:25:56.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 0x7f2b400a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b40060f10 0x7f2b400b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b400b2f10 con 0x7f2b40147660 2022-07-20T02:25:56.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b400a8e30 con 0x7f2b40060f10 2022-07-20T02:25:56.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b400a7310 con 0x7f2b400b2b30 2022-07-20T02:25:56.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b40060f10 0x7f2b400b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b40060f10 0x7f2b400b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35750/0 (socket says 172.21.15.26:35750) 2022-07-20T02:25:56.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 -- 172.21.15.26:0/1340120624 learned_addr learned my addr 172.21.15.26:0/1340120624 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3f7fe700 1 --2- 172.21.15.26:0/1340120624 >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 0x7f2b400a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 -- 172.21.15.26:0/1340120624 >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 msgr2=0x7f2b400a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:56.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 --2- 172.21.15.26:0/1340120624 >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 0x7f2b400a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 -- 172.21.15.26:0/1340120624 >> v1:172.21.15.120:6789/0 conn(0x7f2b40147660 legacy=0x7f2b40066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 -- 172.21.15.26:0/1340120624 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400a7500 con 0x7f2b40060f10 2022-07-20T02:25:56.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3e7fc700 1 -- 172.21.15.26:0/1340120624 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774199467 0 0) 0x7f2b400b2f10 con 0x7f2b40147660 2022-07-20T02:25:56.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3f7fe700 1 --2- 172.21.15.26:0/1340120624 >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 0x7f2b400a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:56.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3effd700 1 --2- 172.21.15.26:0/1340120624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b40060f10 0x7f2b400b1990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2b30002f70 tx=0x7f2b30006eb0).ready entity=mon.0 client_cookie=1a3ccc5b32264184 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3e7fc700 1 -- 172.21.15.26:0/1340120624 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3000d5c0 con 0x7f2b40060f10 2022-07-20T02:25:56.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3e7fc700 1 -- 172.21.15.26:0/1340120624 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b30000e90 con 0x7f2b40060f10 2022-07-20T02:25:56.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b3e7fc700 1 -- 172.21.15.26:0/1340120624 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3000d910 con 0x7f2b40060f10 2022-07-20T02:25:56.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- 172.21.15.26:0/1340120624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b40060f10 msgr2=0x7f2b400b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1340120624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b40060f10 0x7f2b400b1990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2b30002f70 tx=0x7f2b30006eb0).stop 2022-07-20T02:25:56.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- 172.21.15.26:0/1340120624 shutdown_connections 2022-07-20T02:25:56.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1340120624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b40060f10 0x7f2b400b1990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1340120624 >> v2:172.21.15.120:3300/0 conn(0x7f2b400b2b30 0x7f2b400a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- 172.21.15.26:0/1340120624 >> 172.21.15.26:0/1340120624 conn(0x7f2b400b6110 msgr2=0x7f2b400b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:56.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- 172.21.15.26:0/1340120624 shutdown_connections 2022-07-20T02:25:56.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.872+0000 7f2b4616e700 1 -- 172.21.15.26:0/1340120624 wait complete. 2022-07-20T02:25:56.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 Processor -- start 2022-07-20T02:25:56.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 -- start start 2022-07-20T02:25:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b40060f10 0x7f2b40149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b40147660 0x7f2b40156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b40148ba0 con 0x7f2b400b2b30 2022-07-20T02:25:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4015a6b0 con 0x7f2b40147660 2022-07-20T02:25:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b4616e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4015a8a0 con 0x7f2b40060f10 2022-07-20T02:25:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35756/0 (socket says 172.21.15.26:35756) 2022-07-20T02:25:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 -- 172.21.15.26:0/1831045982 learned_addr learned my addr 172.21.15.26:0/1831045982 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3f7fe700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b40060f10 0x7f2b40149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3ffff700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b40147660 0x7f2b40156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 -- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b40060f10 msgr2=0x7f2b40149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b40060f10 0x7f2b40149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 -- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b40147660 msgr2=0x7f2b40156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b40147660 0x7f2b40156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 -- 172.21.15.26:0/1831045982 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3400b180 con 0x7f2b400b2b30 2022-07-20T02:25:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.876+0000 7f2b3effd700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f2b30002f70 tx=0x7f2b30015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b30011760 con 0x7f2b400b2b30 2022-07-20T02:25:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b300118c0 con 0x7f2b400b2b30 2022-07-20T02:25:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b30011c20 con 0x7f2b400b2b30 2022-07-20T02:25:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3000b1e0 con 0x7f2b400b2b30 2022-07-20T02:25:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b40148d90 con 0x7f2b400b2b30 2022-07-20T02:25:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b40148f50 con 0x7f2b400b2b30 2022-07-20T02:25:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b3001a040 con 0x7f2b400b2b30 2022-07-20T02:25:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 0x7f2b20055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2b3006cd90 con 0x7f2b400b2b30 2022-07-20T02:25:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.880+0000 7f2b3f7fe700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 0x7f2b20055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.884+0000 7f2b3f7fe700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 0x7f2b20055e10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2b3400b000 tx=0x7f2b34006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:56.888+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b30043b50 con 0x7f2b400b2b30 2022-07-20T02:25:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.016+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b400a77a0 con 0x7f2b400b2b30 2022-07-20T02:25:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.016+0000 7f2b3cff9700 1 -- 172.21.15.26:0/1831045982 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2b30047170 con 0x7f2b400b2b30 2022-07-20T02:25:57.026 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 msgr2=0x7f2b20055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 0x7f2b20055e10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2b3400b000 tx=0x7f2b34006df0).stop 2022-07-20T02:25:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 msgr2=0x7f2b4015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f2b30002f70 tx=0x7f2b30015000).stop 2022-07-20T02:25:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 shutdown_connections 2022-07-20T02:25:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b400b2b30 0x7f2b4015a1c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b20053950 0x7f2b20055e10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b40147660 0x7f2b40156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 --2- 172.21.15.26:0/1831045982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b40060f10 0x7f2b40149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 >> 172.21.15.26:0/1831045982 conn(0x7f2b400b6110 msgr2=0x7f2b400b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 shutdown_connections 2022-07-20T02:25:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.020+0000 7f2b4616e700 1 -- 172.21.15.26:0/1831045982 wait complete. 2022-07-20T02:25:57.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:57.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.164+0000 7f4a8a45b700 1 Processor -- start 2022-07-20T02:25:57.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.164+0000 7f4a8a45b700 1 -- start start 2022-07-20T02:25:57.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4a840ace00 0x7f4a84061390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a84146bb0 con 0x7f4a8405ce30 2022-07-20T02:25:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a84146e50 con 0x7f4a84066c00 2022-07-20T02:25:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a841470f0 con 0x7f4a840ace00 2022-07-20T02:25:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89c5a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4a840ace00 0x7f4a84061390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89459700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4a8405ce30 0x7f4a840ac6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35042/0 (socket says 172.21.15.26:35042) 2022-07-20T02:25:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89459700 1 -- 172.21.15.26:0/1615557255 learned_addr learned my addr 172.21.15.26:0/1615557255 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595587413 0 0) 0x7f4a84146bb0 con 0x7f4a8405ce30 2022-07-20T02:25:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a6c003260 con 0x7f4a8405ce30 2022-07-20T02:25:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89c5a700 1 -- 172.21.15.26:0/1615557255 >> v1:172.21.15.120:6789/0 conn(0x7f4a84066c00 legacy=0x7f4a84060c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447723765 0 0) 0x7f4a84146e50 con 0x7f4a84066c00 2022-07-20T02:25:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89c5a700 1 -- 172.21.15.26:0/1615557255 >> v1:172.21.15.26:6789/0 conn(0x7f4a8405ce30 legacy=0x7f4a840ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89c5a700 1 -- 172.21.15.26:0/1615557255 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84147390 con 0x7f4a840ace00 2022-07-20T02:25:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a89c5a700 1 --2- 172.21.15.26:0/1615557255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a840ace00 0x7f4a84061390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4a7c0032f0 tx=0x7f4a7c0091d0).ready entity=mon.0 client_cookie=540987ffab5a6b27 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7c00b6c0 con 0x7f4a840ace00 2022-07-20T02:25:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c00b820 con 0x7f4a840ace00 2022-07-20T02:25:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a73fff700 1 -- 172.21.15.26:0/1615557255 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7c00b990 con 0x7f4a840ace00 2022-07-20T02:25:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1615557255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a840ace00 msgr2=0x7f4a84061390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1615557255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a840ace00 0x7f4a84061390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f4a7c0032f0 tx=0x7f4a7c0091d0).stop 2022-07-20T02:25:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1615557255 shutdown_connections 2022-07-20T02:25:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1615557255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a840ace00 0x7f4a84061390 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1615557255 >> 172.21.15.26:0/1615557255 conn(0x7f4a840b7090 msgr2=0x7f4a8405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1615557255 shutdown_connections 2022-07-20T02:25:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.168+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1615557255 wait complete. 2022-07-20T02:25:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 Processor -- start 2022-07-20T02:25:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- start start 2022-07-20T02:25:57.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a84066c00 0x7f4a84063b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a840ace00 0x7f4a84064070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8413fd90 con 0x7f4a8405ce30 2022-07-20T02:25:57.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8413f280 con 0x7f4a840ace00 2022-07-20T02:25:57.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8413f520 con 0x7f4a84066c00 2022-07-20T02:25:57.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35768/0 (socket says 172.21.15.26:35768) 2022-07-20T02:25:57.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 -- 172.21.15.26:0/1310660697 learned_addr learned my addr 172.21.15.26:0/1310660697 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:57.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89c5a700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a840ace00 0x7f4a84064070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a88c58700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a84066c00 0x7f4a84063b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 -- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a84066c00 msgr2=0x7f4a84063b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a84066c00 0x7f4a84063b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 -- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a840ace00 msgr2=0x7f4a84064070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a840ace00 0x7f4a84064070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 -- 172.21.15.26:0/1310660697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c009470 con 0x7f4a8405ce30 2022-07-20T02:25:57.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a89459700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4a80001110 tx=0x7f4a80007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a8000ce80 con 0x7f4a8405ce30 2022-07-20T02:25:57.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a800070d0 con 0x7f4a8405ce30 2022-07-20T02:25:57.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84145080 con 0x7f4a8405ce30 2022-07-20T02:25:57.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a80007430 con 0x7f4a8405ce30 2022-07-20T02:25:57.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.172+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a84064560 con 0x7f4a8405ce30 2022-07-20T02:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84140870 con 0x7f4a8405ce30 2022-07-20T02:25:57.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4a80010070 con 0x7f4a8405ce30 2022-07-20T02:25:57.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a727fc700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 0x7f4a64055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4a8006b720 con 0x7f4a8405ce30 2022-07-20T02:25:57.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a88c58700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 0x7f4a64055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.176+0000 7f4a88c58700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 0x7f4a64055e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4a7400d000 tx=0x7f4a74007a40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.180+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a84140870 con 0x7f4a8405ce30 2022-07-20T02:25:57.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.308+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a84140870 con 0x7f4a8405ce30 2022-07-20T02:25:57.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.308+0000 7f4a727fc700 1 -- 172.21.15.26:0/1310660697 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4a84140870 con 0x7f4a8405ce30 2022-07-20T02:25:57.327 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:57.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 msgr2=0x7f4a64055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 0x7f4a64055e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4a7400d000 tx=0x7f4a74007a40).stop 2022-07-20T02:25:57.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 msgr2=0x7f4a84144890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4a80001110 tx=0x7f4a80007aa0).stop 2022-07-20T02:25:57.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 shutdown_connections 2022-07-20T02:25:57.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a64053950 0x7f4a64055e10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a840ace00 0x7f4a84064070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a84066c00 0x7f4a84063b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 --2- 172.21.15.26:0/1310660697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a8405ce30 0x7f4a84144890 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 >> 172.21.15.26:0/1310660697 conn(0x7f4a840b7090 msgr2=0x7f4a840ac620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 shutdown_connections 2022-07-20T02:25:57.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.312+0000 7f4a8a45b700 1 -- 172.21.15.26:0/1310660697 wait complete. 2022-07-20T02:25:57.333 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:57.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.436+0000 7f5dcffff700 1 Processor -- start 2022-07-20T02:25:57.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.436+0000 7f5dcffff700 1 -- start start 2022-07-20T02:25:57.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd00b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5dd00957f0 0x7f5dd01433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd0147080 con 0x7f5dd0067440 2022-07-20T02:25:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd0143000 con 0x7f5dd00957f0 2022-07-20T02:25:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd01472b0 con 0x7f5dd00b2f20 2022-07-20T02:25:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5dd00957f0 0x7f5dd01433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dce7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5dd0067440 0x7f5dd00950e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35058/0 (socket says 172.21.15.26:35058) 2022-07-20T02:25:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dce7fc700 1 -- 172.21.15.26:0/47122279 learned_addr learned my addr 172.21.15.26:0/47122279 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dceffd700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd00b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 -- 172.21.15.26:0/47122279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 msgr2=0x7f5dd00b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd00b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 -- 172.21.15.26:0/47122279 >> v1:172.21.15.26:6789/0 conn(0x7f5dd0067440 legacy=0x7f5dd00950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 -- 172.21.15.26:0/47122279 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00a9920 con 0x7f5dd00957f0 2022-07-20T02:25:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcf7fe700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd00957f0 0x7f5dd01433f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5dc80032f0 tx=0x7f5dc80091d0).ready entity=mon.0 client_cookie=7b3963c2fca56602 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcdffb700 1 -- 172.21.15.26:0/47122279 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc8009ea0 con 0x7f5dd00957f0 2022-07-20T02:25:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcdffb700 1 -- 172.21.15.26:0/47122279 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc800b8a0 con 0x7f5dd00957f0 2022-07-20T02:25:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcdffb700 1 -- 172.21.15.26:0/47122279 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc800ba10 con 0x7f5dd00957f0 2022-07-20T02:25:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- 172.21.15.26:0/47122279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd00957f0 msgr2=0x7f5dd01433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd00957f0 0x7f5dd01433f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5dc80032f0 tx=0x7f5dc80091d0).stop 2022-07-20T02:25:57.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- 172.21.15.26:0/47122279 shutdown_connections 2022-07-20T02:25:57.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd00957f0 0x7f5dd01433f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 --2- 172.21.15.26:0/47122279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd00b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- 172.21.15.26:0/47122279 >> 172.21.15.26:0/47122279 conn(0x7f5dd00b67f0 msgr2=0x7f5dd00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- 172.21.15.26:0/47122279 shutdown_connections 2022-07-20T02:25:57.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.440+0000 7f5dcffff700 1 -- 172.21.15.26:0/47122279 wait complete. 2022-07-20T02:25:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 Processor -- start 2022-07-20T02:25:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- start start 2022-07-20T02:25:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd00957f0 0x7f5dd0091160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd0092ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd0090d00 con 0x7f5dd0067440 2022-07-20T02:25:57.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00992d0 con 0x7f5dd00957f0 2022-07-20T02:25:57.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd009ad00 con 0x7f5dd00b2f20 2022-07-20T02:25:57.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35784/0 (socket says 172.21.15.26:35784) 2022-07-20T02:25:57.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 -- 172.21.15.26:0/2781310457 learned_addr learned my addr 172.21.15.26:0/2781310457 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:57.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dce7fc700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd00957f0 0x7f5dd0091160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcf7fe700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd0092ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 -- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 msgr2=0x7f5dd0092ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd0092ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 -- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd00957f0 msgr2=0x7f5dd0091160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd00957f0 0x7f5dd0091160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 -- 172.21.15.26:0/2781310457 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009470 con 0x7f5dd0067440 2022-07-20T02:25:57.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dceffd700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f5dc00030b0 tx=0x7f5dc0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc0011780 con 0x7f5dd0067440 2022-07-20T02:25:57.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc00118e0 con 0x7f5dd0067440 2022-07-20T02:25:57.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc0011c40 con 0x7f5dd0067440 2022-07-20T02:25:57.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd009afa0 con 0x7f5dd0067440 2022-07-20T02:25:57.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.444+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd009b240 con 0x7f5dd0067440 2022-07-20T02:25:57.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5dc001c460 con 0x7f5dd0067440 2022-07-20T02:25:57.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd00a95a0 con 0x7f5dd0067440 2022-07-20T02:25:57.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dbffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 0x7f5dac055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5dc006c2e0 con 0x7f5dd0067440 2022-07-20T02:25:57.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dce7fc700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 0x7f5dac055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.448+0000 7f5dce7fc700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 0x7f5dac055e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5dc400d000 tx=0x7f5dc4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.452+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5dc0043040 con 0x7f5dd0067440 2022-07-20T02:25:57.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dd00574e0 con 0x7f5dd0067440 2022-07-20T02:25:57.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dbffff700 1 -- 172.21.15.26:0/2781310457 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f5dc0046660 con 0x7f5dd0067440 2022-07-20T02:25:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:57.586 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:57.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 msgr2=0x7f5dac055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 0x7f5dac055e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5dc400d000 tx=0x7f5dc4009040).stop 2022-07-20T02:25:57.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 msgr2=0x7f5dd0098de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.580+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f5dc00030b0 tx=0x7f5dc0008040).stop 2022-07-20T02:25:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 shutdown_connections 2022-07-20T02:25:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5dac053980 0x7f5dac055e40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2f20 0x7f5dd0092ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd00957f0 0x7f5dd0091160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 --2- 172.21.15.26:0/2781310457 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dd0067440 0x7f5dd0098de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 >> 172.21.15.26:0/2781310457 conn(0x7f5dd00b67f0 msgr2=0x7f5dd005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 shutdown_connections 2022-07-20T02:25:57.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.584+0000 7f5dcffff700 1 -- 172.21.15.26:0/2781310457 wait complete. 2022-07-20T02:25:57.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:57.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 Processor -- start 2022-07-20T02:25:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 -- start start 2022-07-20T02:25:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 0x7f3430061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34300af6d0 0x7f34300afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300aa1e0 con 0x7f34300add30 2022-07-20T02:25:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34300ae110 con 0x7f34300af6d0 2022-07-20T02:25:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3435dc0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300a9f40 con 0x7f34300b2ab0 2022-07-20T02:25:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3434dbe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f34300add30 0x7f3430060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35070/0 (socket says 172.21.15.26:35070) 2022-07-20T02:25:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f3434dbe700 1 -- 172.21.15.26:0/65075053 learned_addr learned my addr 172.21.15.26:0/65075053 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34300af6d0 0x7f34300afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f342ffff700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 0x7f3430061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f342f7fe700 1 -- 172.21.15.26:0/65075053 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462635039 0 0) 0x7f34300aa1e0 con 0x7f34300add30 2022-07-20T02:25:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f342f7fe700 1 -- 172.21.15.26:0/65075053 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c003260 con 0x7f34300add30 2022-07-20T02:25:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 -- 172.21.15.26:0/65075053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 msgr2=0x7f3430061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 0x7f3430061060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 -- 172.21.15.26:0/65075053 >> v1:172.21.15.26:6789/0 conn(0x7f34300add30 legacy=0x7f3430060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 -- 172.21.15.26:0/65075053 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300b1110 con 0x7f34300af6d0 2022-07-20T02:25:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f342ffff700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 0x7f3430061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.704+0000 7f34355bf700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300afb60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f3428002f70 tx=0x7f3428006df0).ready entity=mon.0 client_cookie=325a0f4a23dc0056 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f342f7fe700 1 -- 172.21.15.26:0/65075053 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3428014070 con 0x7f34300af6d0 2022-07-20T02:25:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f342f7fe700 1 -- 172.21.15.26:0/65075053 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428000e90 con 0x7f34300af6d0 2022-07-20T02:25:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f342f7fe700 1 -- 172.21.15.26:0/65075053 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f342800fb60 con 0x7f34300af6d0 2022-07-20T02:25:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- 172.21.15.26:0/65075053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 msgr2=0x7f34300afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300afb60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f3428002f70 tx=0x7f3428006df0).stop 2022-07-20T02:25:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- 172.21.15.26:0/65075053 shutdown_connections 2022-07-20T02:25:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300afb60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 --2- 172.21.15.26:0/65075053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300b2ab0 0x7f3430061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- 172.21.15.26:0/65075053 >> 172.21.15.26:0/65075053 conn(0x7f34300b3050 msgr2=0x7f34300b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- 172.21.15.26:0/65075053 shutdown_connections 2022-07-20T02:25:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- 172.21.15.26:0/65075053 wait complete. 2022-07-20T02:25:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 Processor -- start 2022-07-20T02:25:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.708+0000 7f3435dc0700 1 -- start start 2022-07-20T02:25:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 0x7f34300b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34300b2ab0 0x7f34300abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300ac360 con 0x7f34300af6d0 2022-07-20T02:25:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3430063fc0 con 0x7f34300b2ab0 2022-07-20T02:25:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3430064260 con 0x7f34300add30 2022-07-20T02:25:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35796/0 (socket says 172.21.15.26:35796) 2022-07-20T02:25:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 -- 172.21.15.26:0/1175584752 learned_addr learned my addr 172.21.15.26:0/1175584752 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3434dbe700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 0x7f34300b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f34355bf700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34300b2ab0 0x7f34300abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 -- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 msgr2=0x7f34300b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 0x7f34300b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 -- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34300b2ab0 msgr2=0x7f34300abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34300b2ab0 0x7f34300abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 -- 172.21.15.26:0/1175584752 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342800b180 con 0x7f34300af6d0 2022-07-20T02:25:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3434dbe700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 0x7f34300b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342ffff700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3424006940 tx=0x7f3424008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3424011780 con 0x7f34300af6d0 2022-07-20T02:25:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34240118e0 con 0x7f34300af6d0 2022-07-20T02:25:57.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3430064500 con 0x7f34300af6d0 2022-07-20T02:25:57.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3424011c40 con 0x7f34300af6d0 2022-07-20T02:25:57.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34300acf80 con 0x7f34300af6d0 2022-07-20T02:25:57.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.712+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34300ad140 con 0x7f34300af6d0 2022-07-20T02:25:57.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.716+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3424012420 con 0x7f34300af6d0 2022-07-20T02:25:57.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.716+0000 7f342dffb700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 0x7f3410055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.716+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f342406c060 con 0x7f34300af6d0 2022-07-20T02:25:57.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.716+0000 7f3434dbe700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 0x7f3410055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.716+0000 7f3434dbe700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 0x7f3410055ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f342000b000 tx=0x7f3420009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:57.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.724+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3424042cb0 con 0x7f34300af6d0 2022-07-20T02:25:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.848+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f34300b13b0 con 0x7f34300af6d0 2022-07-20T02:25:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.848+0000 7f342dffb700 1 -- 172.21.15.26:0/1175584752 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f34240462d0 con 0x7f34300af6d0 2022-07-20T02:25:57.854 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 msgr2=0x7f3410055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 0x7f3410055ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f342000b000 tx=0x7f3420009040).stop 2022-07-20T02:25:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 msgr2=0x7f34300ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3424006940 tx=0x7f3424008040).stop 2022-07-20T02:25:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 shutdown_connections 2022-07-20T02:25:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34300af6d0 0x7f34300ab980 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3410053a10 0x7f3410055ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34300b2ab0 0x7f34300abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 --2- 172.21.15.26:0/1175584752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34300add30 0x7f34300b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 >> 172.21.15.26:0/1175584752 conn(0x7f34300b3050 msgr2=0x7f34301433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 shutdown_connections 2022-07-20T02:25:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.852+0000 7f3435dc0700 1 -- 172.21.15.26:0/1175584752 wait complete. 2022-07-20T02:25:57.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.984+0000 7f271eb63700 1 Processor -- start 2022-07-20T02:25:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.984+0000 7f271eb63700 1 -- start start 2022-07-20T02:25:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f2718144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f271814ac80 0x7f2718144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:57.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2718061c80 con 0x7f27180b4240 2022-07-20T02:25:57.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27180b1ab0 con 0x7f271814ac80 2022-07-20T02:25:57.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271814b190 con 0x7f27180618a0 2022-07-20T02:25:57.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f271814ac80 0x7f2718144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271db61700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f2718144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:57.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f271814ac80 0x7f2718144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:55496/0 (socket says 172.21.15.26:55496) 2022-07-20T02:25:57.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 -- 172.21.15.26:0/1482259371 learned_addr learned my addr 172.21.15.26:0/1482259371 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271e362700 1 --1- 172.21.15.26:0/1482259371 >> v1:172.21.15.120:6789/0 conn(0x7f27180b4240 0x7f2718147800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49300/0 (socket says 172.21.15.26:49300) 2022-07-20T02:25:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 -- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 msgr2=0x7f2718144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 --2- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f2718144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 -- 172.21.15.26:0/1482259371 >> v1:172.21.15.120:6789/0 conn(0x7f27180b4240 legacy=0x7f2718147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 -- 172.21.15.26:0/1482259371 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271814b430 con 0x7f271814ac80 2022-07-20T02:25:57.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271d360700 1 --2- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271814ac80 0x7f2718144d40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f2714002f70 tx=0x7f2714006eb0).ready entity=mon.1 client_cookie=95616ac387cd4bc8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271cb5f700 1 -- 172.21.15.26:0/1482259371 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f271400d570 con 0x7f271814ac80 2022-07-20T02:25:58.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271cb5f700 1 -- 172.21.15.26:0/1482259371 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714000e90 con 0x7f271814ac80 2022-07-20T02:25:58.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271cb5f700 1 -- 172.21.15.26:0/1482259371 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f271400d900 con 0x7f271814ac80 2022-07-20T02:25:58.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271814ac80 msgr2=0x7f2718144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 --2- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271814ac80 0x7f2718144d40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f2714002f70 tx=0x7f2714006eb0).stop 2022-07-20T02:25:58.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- 172.21.15.26:0/1482259371 shutdown_connections 2022-07-20T02:25:58.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 --2- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271814ac80 0x7f2718144d40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 --2- 172.21.15.26:0/1482259371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f2718144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- 172.21.15.26:0/1482259371 >> 172.21.15.26:0/1482259371 conn(0x7f2718066f70 msgr2=0x7f2718067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- 172.21.15.26:0/1482259371 shutdown_connections 2022-07-20T02:25:58.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.988+0000 7f271eb63700 1 -- 172.21.15.26:0/1482259371 wait complete. 2022-07-20T02:25:58.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 Processor -- start 2022-07-20T02:25:58.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 -- start start 2022-07-20T02:25:58.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f271814a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180b4240 0x7f271814c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718141f50 con 0x7f271814ac80 2022-07-20T02:25:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718149fd0 con 0x7f27180b4240 2022-07-20T02:25:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271eb63700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271814a270 con 0x7f27180618a0 2022-07-20T02:25:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35814/0 (socket says 172.21.15.26:35814) 2022-07-20T02:25:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 -- 172.21.15.26:0/2195990550 learned_addr learned my addr 172.21.15.26:0/2195990550 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271d360700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180b4240 0x7f271814c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271db61700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f271814a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 -- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 msgr2=0x7f271814a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f271814a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 -- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180b4240 msgr2=0x7f271814c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180b4240 0x7f271814c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 -- 172.21.15.26:0/2195990550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271400b1e0 con 0x7f271814ac80 2022-07-20T02:25:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f271e362700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f27180b7420 tx=0x7f2710007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.992+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2710005ae0 con 0x7f271814ac80 2022-07-20T02:25:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710005c40 con 0x7f271814ac80 2022-07-20T02:25:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718149be0 con 0x7f271814ac80 2022-07-20T02:25:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27100070d0 con 0x7f271814ac80 2022-07-20T02:25:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2718141b20 con 0x7f271814ac80 2022-07-20T02:25:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2718149e30 con 0x7f271814ac80 2022-07-20T02:25:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f271001a060 con 0x7f271814ac80 2022-07-20T02:25:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f2706ffd700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 0x7f26f8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f271006bbc0 con 0x7f271814ac80 2022-07-20T02:25:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f271db61700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 0x7f26f8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:57.996+0000 7f271db61700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 0x7f26f8055e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f270800b000 tx=0x7f2708006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.000+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2718149e30 con 0x7f271814ac80 2022-07-20T02:25:58.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.128+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2718149e30 con 0x7f271814ac80 2022-07-20T02:25:58.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.128+0000 7f2706ffd700 1 -- 172.21.15.26:0/2195990550 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2718149e30 con 0x7f271814ac80 2022-07-20T02:25:58.135 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:58.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 msgr2=0x7f26f8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 0x7f26f8055e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f270800b000 tx=0x7f2708006df0).stop 2022-07-20T02:25:58.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 msgr2=0x7f2718159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f27180b7420 tx=0x7f2710007f10).stop 2022-07-20T02:25:58.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 shutdown_connections 2022-07-20T02:25:58.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271814ac80 0x7f2718159ea0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26f8053950 0x7f26f8055e10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180b4240 0x7f271814c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 --2- 172.21.15.26:0/2195990550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27180618a0 0x7f271814a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 >> 172.21.15.26:0/2195990550 conn(0x7f2718066f70 msgr2=0x7f2718144c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 shutdown_connections 2022-07-20T02:25:58.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.132+0000 7f271eb63700 1 -- 172.21.15.26:0/2195990550 wait complete. 2022-07-20T02:25:58.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:58.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.268+0000 7fc716f78700 1 Processor -- start 2022-07-20T02:25:58.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.268+0000 7fc716f78700 1 -- start start 2022-07-20T02:25:58.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc71005cfd0 0x7fc7101407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc7100b1160 0x7fc710140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71005d3b0 con 0x7fc710146e80 2022-07-20T02:25:58.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7101435b0 con 0x7fc7100b1160 2022-07-20T02:25:58.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100b4bc0 con 0x7fc71005cfd0 2022-07-20T02:25:58.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc7100b1160 0x7fc710140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc715515700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc710146e80 0x7fc710147290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35100/0 (socket says 172.21.15.26:35100) 2022-07-20T02:25:58.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc715515700 1 -- 172.21.15.26:0/2367369251 learned_addr learned my addr 172.21.15.26:0/2367369251 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:58.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc714d14700 1 --2- 172.21.15.26:0/2367369251 >> v2:172.21.15.120:3300/0 conn(0x7fc71005cfd0 0x7fc7101407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 -- 172.21.15.26:0/2367369251 >> v2:172.21.15.120:3300/0 conn(0x7fc71005cfd0 msgr2=0x7fc7101407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:58.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 --2- 172.21.15.26:0/2367369251 >> v2:172.21.15.120:3300/0 conn(0x7fc71005cfd0 0x7fc7101407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 -- 172.21.15.26:0/2367369251 >> v1:172.21.15.26:6789/0 conn(0x7fc710146e80 legacy=0x7fc710147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 -- 172.21.15.26:0/2367369251 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100b4e60 con 0x7fc7100b1160 2022-07-20T02:25:58.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70ffff700 1 --2- 172.21.15.26:0/2367369251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc710140cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc700002f70 tx=0x7fc700006df0).ready entity=mon.0 client_cookie=f1226daf52184cb6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70f7fe700 1 -- 172.21.15.26:0/2367369251 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc70000d5c0 con 0x7fc7100b1160 2022-07-20T02:25:58.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70f7fe700 1 -- 172.21.15.26:0/2367369251 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc700000e90 con 0x7fc7100b1160 2022-07-20T02:25:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc70f7fe700 1 -- 172.21.15.26:0/2367369251 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc70000d910 con 0x7fc7100b1160 2022-07-20T02:25:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- 172.21.15.26:0/2367369251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 msgr2=0x7fc710140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 --2- 172.21.15.26:0/2367369251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc710140cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc700002f70 tx=0x7fc700006df0).stop 2022-07-20T02:25:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- 172.21.15.26:0/2367369251 shutdown_connections 2022-07-20T02:25:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 --2- 172.21.15.26:0/2367369251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc710140cb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 --2- 172.21.15.26:0/2367369251 >> v2:172.21.15.120:3300/0 conn(0x7fc71005cfd0 0x7fc7101407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- 172.21.15.26:0/2367369251 >> 172.21.15.26:0/2367369251 conn(0x7fc710066f70 msgr2=0x7fc71005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- 172.21.15.26:0/2367369251 shutdown_connections 2022-07-20T02:25:58.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.272+0000 7fc716f78700 1 -- 172.21.15.26:0/2367369251 wait complete. 2022-07-20T02:25:58.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 Processor -- start 2022-07-20T02:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 -- start start 2022-07-20T02:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc710146e80 0x7fc71013a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc710140130 con 0x7fc7100b1160 2022-07-20T02:25:58.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100b7a80 con 0x7fc710146e80 2022-07-20T02:25:58.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc716f78700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100b9070 con 0x7fc71005cfd0 2022-07-20T02:25:58.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc714d14700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc715515700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc710146e80 0x7fc71013a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc714d14700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57340/0 (socket says 172.21.15.26:57340) 2022-07-20T02:25:58.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc714d14700 1 -- 172.21.15.26:0/851647733 learned_addr learned my addr 172.21.15.26:0/851647733 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35826/0 (socket says 172.21.15.26:35826) 2022-07-20T02:25:58.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 -- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 msgr2=0x7fc71013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 -- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc710146e80 msgr2=0x7fc71013a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc710146e80 0x7fc71013a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 -- 172.21.15.26:0/851647733 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70000b180 con 0x7fc7100b1160 2022-07-20T02:25:58.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc714d14700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:58.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70ffff700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc700002f70 tx=0x7fc700014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.276+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc700013de0 con 0x7fc7100b1160 2022-07-20T02:25:58.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70001e400 con 0x7fc7100b1160 2022-07-20T02:25:58.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7100b9310 con 0x7fc7100b1160 2022-07-20T02:25:58.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc70001e760 con 0x7fc7100b1160 2022-07-20T02:25:58.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7100b7670 con 0x7fc7100b1160 2022-07-20T02:25:58.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7100b8ed0 con 0x7fc7100b1160 2022-07-20T02:25:58.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc700025030 con 0x7fc7100b1160 2022-07-20T02:25:58.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc70dffb700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 0x7fc6f0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc70006cce0 con 0x7fc7100b1160 2022-07-20T02:25:58.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc714d14700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 0x7fc6f0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.280+0000 7fc714d14700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 0x7fc6f0055e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc704008000 tx=0x7fc704008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.288+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc7100b8ed0 con 0x7fc7100b1160 2022-07-20T02:25:58.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.408+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7100b8ed0 con 0x7fc7100b1160 2022-07-20T02:25:58.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc70dffb700 1 -- 172.21.15.26:0/851647733 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc7100b8ed0 con 0x7fc7100b1160 2022-07-20T02:25:58.415 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:58.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 msgr2=0x7fc6f0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 0x7fc6f0055e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc704008000 tx=0x7fc704008040).stop 2022-07-20T02:25:58.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 msgr2=0x7fc7100afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc700002f70 tx=0x7fc700014000).stop 2022-07-20T02:25:58.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 shutdown_connections 2022-07-20T02:25:58.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7100b1160 0x7fc7100afd80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc6f0053950 0x7fc6f0055e10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc710146e80 0x7fc71013a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 --2- 172.21.15.26:0/851647733 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc71005cfd0 0x7fc71013f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 >> 172.21.15.26:0/851647733 conn(0x7fc710066f70 msgr2=0x7fc7100b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 shutdown_connections 2022-07-20T02:25:58.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.412+0000 7fc716f78700 1 -- 172.21.15.26:0/851647733 wait complete. 2022-07-20T02:25:58.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:58.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.536+0000 7f1dc9edf700 1 Processor -- start 2022-07-20T02:25:58.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.536+0000 7f1dc9edf700 1 -- start start 2022-07-20T02:25:58.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1dc40b9c80 0x7f1dc41431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40ba060 con 0x7f1dc4062710 2022-07-20T02:25:58.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc412bee0 con 0x7f1dc40b9c80 2022-07-20T02:25:58.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4061f80 con 0x7f1dc40b5b70 2022-07-20T02:25:58.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57350/0 (socket says 172.21.15.26:57350) 2022-07-20T02:25:58.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 -- 172.21.15.26:0/2203853000 learned_addr learned my addr 172.21.15.26:0/2203853000 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc37fe700 1 --2- 172.21.15.26:0/2203853000 >> v2:172.21.15.120:3300/0 conn(0x7f1dc40b9c80 0x7f1dc41431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc27fc700 1 -- 172.21.15.26:0/2203853000 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516540592 0 0) 0x7f1dc40ba060 con 0x7f1dc4062710 2022-07-20T02:25:58.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc27fc700 1 -- 172.21.15.26:0/2203853000 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0003260 con 0x7f1dc4062710 2022-07-20T02:25:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 -- 172.21.15.26:0/2203853000 >> v2:172.21.15.120:3300/0 conn(0x7f1dc40b9c80 msgr2=0x7f1dc41431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 --2- 172.21.15.26:0/2203853000 >> v2:172.21.15.120:3300/0 conn(0x7f1dc40b9c80 0x7f1dc41431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 -- 172.21.15.26:0/2203853000 >> v1:172.21.15.120:6789/0 conn(0x7f1dc4062710 legacy=0x7f1dc40b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 -- 172.21.15.26:0/2203853000 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4062220 con 0x7f1dc40b5b70 2022-07-20T02:25:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc3fff700 1 --2- 172.21.15.26:0/2203853000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1dbc0032f0 tx=0x7f1dbc009290).ready entity=mon.2 client_cookie=a309f815bad76b24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc27fc700 1 -- 172.21.15.26:0/2203853000 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dbc00b6c0 con 0x7f1dc40b5b70 2022-07-20T02:25:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc27fc700 1 -- 172.21.15.26:0/2203853000 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dbc00b820 con 0x7f1dc40b5b70 2022-07-20T02:25:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc27fc700 1 -- 172.21.15.26:0/2203853000 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dbc00b990 con 0x7f1dc40b5b70 2022-07-20T02:25:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- 172.21.15.26:0/2203853000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 msgr2=0x7f1dc41403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/2203853000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1dbc0032f0 tx=0x7f1dbc009290).stop 2022-07-20T02:25:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- 172.21.15.26:0/2203853000 shutdown_connections 2022-07-20T02:25:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/2203853000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b5b70 0x7f1dc41403e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/2203853000 >> v2:172.21.15.120:3300/0 conn(0x7f1dc40b9c80 0x7f1dc41431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- 172.21.15.26:0/2203853000 >> 172.21.15.26:0/2203853000 conn(0x7f1dc4066f70 msgr2=0x7f1dc405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- 172.21.15.26:0/2203853000 shutdown_connections 2022-07-20T02:25:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.540+0000 7f1dc9edf700 1 -- 172.21.15.26:0/2203853000 wait complete. 2022-07-20T02:25:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 Processor -- start 2022-07-20T02:25:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 -- start start 2022-07-20T02:25:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc40b5b70 0x7f1dc40b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b9c80 0x7f1dc4155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40b0c10 con 0x7f1dc4062710 2022-07-20T02:25:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40b17d0 con 0x7f1dc40b5b70 2022-07-20T02:25:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc9edf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4145ec0 con 0x7f1dc40b9c80 2022-07-20T02:25:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35838/0 (socket says 172.21.15.26:35838) 2022-07-20T02:25:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 -- 172.21.15.26:0/1244332224 learned_addr learned my addr 172.21.15.26:0/1244332224 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc3fff700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b9c80 0x7f1dc4155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc2ffd700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc40b5b70 0x7f1dc40b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 -- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b9c80 msgr2=0x7f1dc4155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b9c80 0x7f1dc4155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 -- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc40b5b70 msgr2=0x7f1dc40b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc40b5b70 0x7f1dc40b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 -- 172.21.15.26:0/1244332224 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc009530 con 0x7f1dc4062710 2022-07-20T02:25:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.544+0000 7f1dc37fe700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1dc40b1960 tx=0x7f1db8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db8010780 con 0x7f1dc4062710 2022-07-20T02:25:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80108e0 con 0x7f1dc4062710 2022-07-20T02:25:58.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc40b0e40 con 0x7f1dc4062710 2022-07-20T02:25:58.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db8010c40 con 0x7f1dc4062710 2022-07-20T02:25:58.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4146160 con 0x7f1dc4062710 2022-07-20T02:25:58.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1db801c460 con 0x7f1dc4062710 2022-07-20T02:25:58.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 0x7f1da4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc40b6650 con 0x7f1dc4062710 2022-07-20T02:25:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1db806c400 con 0x7f1dc4062710 2022-07-20T02:25:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc2ffd700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 0x7f1da4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.548+0000 7f1dc2ffd700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 0x7f1da4055ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1db400d000 tx=0x7f1db4007ab0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.552+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1db80430c0 con 0x7f1dc4062710 2022-07-20T02:25:58.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1dc40574e0 con 0x7f1dc4062710 2022-07-20T02:25:58.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc0ff9700 1 -- 172.21.15.26:0/1244332224 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1db80466e0 con 0x7f1dc4062710 2022-07-20T02:25:58.684 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:58.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 msgr2=0x7f1da4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 0x7f1da4055ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1db400d000 tx=0x7f1db4007ab0).stop 2022-07-20T02:25:58.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 msgr2=0x7f1dc4142740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1dc40b1960 tx=0x7f1db8006df0).stop 2022-07-20T02:25:58.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 shutdown_connections 2022-07-20T02:25:58.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1da4053b30 0x7f1da4055ff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc40b9c80 0x7f1dc4155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc40b5b70 0x7f1dc40b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 --2- 172.21.15.26:0/1244332224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dc4062710 0x7f1dc4142740 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 >> 172.21.15.26:0/1244332224 conn(0x7f1dc4066f70 msgr2=0x7f1dc40b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 shutdown_connections 2022-07-20T02:25:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.680+0000 7f1dc9edf700 1 -- 172.21.15.26:0/1244332224 wait complete. 2022-07-20T02:25:58.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:25:58.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 Processor -- start 2022-07-20T02:25:58.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 -- start start 2022-07-20T02:25:58.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0b62d0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c14aac0 con 0x7f4b7c147be0 2022-07-20T02:25:58.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8116d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7c14ad60 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8096c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8096c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35850/0 (socket says 172.21.15.26:35850) 2022-07-20T02:25:58.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b8096c700 1 -- 172.21.15.26:0/2285095515 learned_addr learned my addr 172.21.15.26:0/2285095515 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.812+0000 7f4b7bfff700 1 --1- 172.21.15.26:0/2285095515 >> v1:172.21.15.26:6789/0 conn(0x7f4b7c0b5ef0 0x7f4b7c1474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35128/0 (socket says 172.21.15.26:35128) 2022-07-20T02:25:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8096c700 1 -- 172.21.15.26:0/2285095515 >> v1:172.21.15.120:6789/0 conn(0x7f4b7c147be0 legacy=0x7f4b7c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8096c700 1 -- 172.21.15.26:0/2285095515 >> v1:172.21.15.26:6789/0 conn(0x7f4b7c0b5ef0 legacy=0x7f4b7c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8096c700 1 -- 172.21.15.26:0/2285095515 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c14b000 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8096c700 1 --2- 172.21.15.26:0/2285095515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4b7c06b790 tx=0x7f4b74009290).ready entity=mon.0 client_cookie=c225f0c70930c629 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b7b7fe700 1 -- 172.21.15.26:0/2285095515 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b7400b860 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b7b7fe700 1 -- 172.21.15.26:0/2285095515 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7400b9c0 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b7b7fe700 1 -- 172.21.15.26:0/2285095515 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b7400bb30 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 -- 172.21.15.26:0/2285095515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0a7fc0 msgr2=0x7f4b7c0af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 --2- 172.21.15.26:0/2285095515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4b7c06b790 tx=0x7f4b74009290).stop 2022-07-20T02:25:58.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 -- 172.21.15.26:0/2285095515 shutdown_connections 2022-07-20T02:25:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 --2- 172.21.15.26:0/2285095515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c0af550 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 -- 172.21.15.26:0/2285095515 >> 172.21.15.26:0/2285095515 conn(0x7f4b7c060310 msgr2=0x7f4b7c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 -- 172.21.15.26:0/2285095515 shutdown_connections 2022-07-20T02:25:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.816+0000 7f4b8116d700 1 -- 172.21.15.26:0/2285095515 wait complete. 2022-07-20T02:25:58.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 Processor -- start 2022-07-20T02:25:58.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- start start 2022-07-20T02:25:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b7c147be0 0x7f4b7c063840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c15ab60 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c147a10 con 0x7f4b7c0a7fc0 2022-07-20T02:25:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0645e0 con 0x7f4b7c147be0 2022-07-20T02:25:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35854/0 (socket says 172.21.15.26:35854) 2022-07-20T02:25:58.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 -- 172.21.15.26:0/1221409526 learned_addr learned my addr 172.21.15.26:0/1221409526 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8096c700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b7c147be0 0x7f4b7c063840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b7bfff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 -- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b7c147be0 msgr2=0x7f4b7c063840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b7c147be0 0x7f4b7c063840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 -- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 msgr2=0x7f4b7c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c15a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 -- 172.21.15.26:0/1221409526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b74009530 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b7bfff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:25:58.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b73fff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4b7c15aca0 tx=0x7f4b64006d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b64012410 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64012570 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c064880 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b640128d0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.820+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c065060 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b6400f030 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b79ffb700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 0x7f4b60055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:58.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4b6406bd00 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b7bfff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 0x7f4b60055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c14b2a0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.824+0000 7f4b7bfff700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 0x7f4b60055ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4b6c008000 tx=0x7f4b6c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.832+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b64042ac0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.956+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b7c0574e0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b79ffb700 1 -- 172.21.15.26:0/1221409526 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16347 (secure 0 0 0) 0x7f4b640460e0 con 0x7f4b7c0b5ef0 2022-07-20T02:25:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:25:58.964 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:25:35.945703+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:25:31.063095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T02:25:33.651615+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:25:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 msgr2=0x7f4b60055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 0x7f4b60055ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4b6c008000 tx=0x7f4b6c009040).stop 2022-07-20T02:25:58.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 msgr2=0x7f4b7c11d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:58.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4b7c15aca0 tx=0x7f4b64006d40).stop 2022-07-20T02:25:58.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 shutdown_connections 2022-07-20T02:25:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b7c0b5ef0 0x7f4b7c11d350 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b60053a10 0x7f4b60055ed0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b7c147be0 0x7f4b7c063840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 --2- 172.21.15.26:0/1221409526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b7c0a7fc0 0x7f4b7c15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:58.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 >> 172.21.15.26:0/1221409526 conn(0x7f4b7c060310 msgr2=0x7f4b7c05ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:58.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 shutdown_connections 2022-07-20T02:25:58.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:58.960+0000 7f4b8116d700 1 -- 172.21.15.26:0/1221409526 wait complete. 2022-07-20T02:25:58.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:25:31.063095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:25:33.651615+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:25:58.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:25:59.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.084+0000 7f9ca6aef700 1 Processor -- start 2022-07-20T02:25:59.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- start start 2022-07-20T02:25:59.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca0065980 0x7f9ca0065d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ca013dd40 0x7f9ca013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca005d220 con 0x7f9ca01449d0 2022-07-20T02:25:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00b35d0 con 0x7f9ca013dd40 2022-07-20T02:25:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b3870 con 0x7f9ca0065980 2022-07-20T02:25:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ca013dd40 0x7f9ca013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ca013dd40 0x7f9ca013e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35864/0 (socket says 172.21.15.26:35864) 2022-07-20T02:25:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 -- 172.21.15.26:0/61060486 learned_addr learned my addr 172.21.15.26:0/61060486 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca488b700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca0065980 0x7f9ca0065d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 -- 172.21.15.26:0/61060486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca0065980 msgr2=0x7f9ca0065d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca0065980 0x7f9ca0065d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 -- 172.21.15.26:0/61060486 >> v1:172.21.15.120:6789/0 conn(0x7f9ca01449d0 legacy=0x7f9ca0144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 -- 172.21.15.26:0/61060486 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca0144530 con 0x7f9ca013dd40 2022-07-20T02:25:59.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca508c700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca013e1b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9c980032f0 tx=0x7f9c98009290).ready entity=mon.0 client_cookie=aef72c050efc091b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9c9f7fe700 1 -- 172.21.15.26:0/61060486 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c9800b8a0 con 0x7f9ca013dd40 2022-07-20T02:25:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9c9f7fe700 1 -- 172.21.15.26:0/61060486 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9800ba00 con 0x7f9ca013dd40 2022-07-20T02:25:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9c9f7fe700 1 -- 172.21.15.26:0/61060486 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c9800bb70 con 0x7f9ca013dd40 2022-07-20T02:25:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- 172.21.15.26:0/61060486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 msgr2=0x7f9ca013e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca013e1b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9c980032f0 tx=0x7f9c98009290).stop 2022-07-20T02:25:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- 172.21.15.26:0/61060486 shutdown_connections 2022-07-20T02:25:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca013e1b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/61060486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca0065980 0x7f9ca0065d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- 172.21.15.26:0/61060486 >> 172.21.15.26:0/61060486 conn(0x7f9ca005c580 msgr2=0x7f9ca005c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- 172.21.15.26:0/61060486 shutdown_connections 2022-07-20T02:25:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.088+0000 7f9ca6aef700 1 -- 172.21.15.26:0/61060486 wait complete. 2022-07-20T02:25:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 Processor -- start 2022-07-20T02:25:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 -- start start 2022-07-20T02:25:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ca0065980 0x7f9ca0147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca01449d0 0x7f9ca00b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca0142130 con 0x7f9ca013dd40 2022-07-20T02:25:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca0151120 con 0x7f9ca0065980 2022-07-20T02:25:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca0142360 con 0x7f9ca01449d0 2022-07-20T02:25:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35868/0 (socket says 172.21.15.26:35868) 2022-07-20T02:25:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 -- 172.21.15.26:0/2999964688 learned_addr learned my addr 172.21.15.26:0/2999964688 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca508c700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca01449d0 0x7f9ca00b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca488b700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ca0065980 0x7f9ca0147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 -- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca01449d0 msgr2=0x7f9ca00b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca01449d0 0x7f9ca00b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 -- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ca0065980 msgr2=0x7f9ca0147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ca0065980 0x7f9ca0147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 -- 172.21.15.26:0/2999964688 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c98009530 con 0x7f9ca013dd40 2022-07-20T02:25:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9ffff700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f9ca0145560 tx=0x7f9c88007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c8800ce30 con 0x7f9ca013dd40 2022-07-20T02:25:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8800f470 con 0x7f9ca013dd40 2022-07-20T02:25:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca01504f0 con 0x7f9ca013dd40 2022-07-20T02:25:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.092+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c8800f7d0 con 0x7f9ca013dd40 2022-07-20T02:25:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca0150220 con 0x7f9ca013dd40 2022-07-20T02:25:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca00b5180 con 0x7f9ca013dd40 2022-07-20T02:25:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9c88019030 con 0x7f9ca013dd40 2022-07-20T02:25:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9c9dffb700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 0x7f9c84055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9ca488b700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 0x7f9c84055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9c8806b400 con 0x7f9ca013dd40 2022-07-20T02:25:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.096+0000 7f9ca488b700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 0x7f9c84055f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9c9000c000 tx=0x7f9c90008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.104+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c880421c0 con 0x7f9ca013dd40 2022-07-20T02:25:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.224+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9ca00574e0 con 0x7f9ca013dd40 2022-07-20T02:25:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9c9dffb700 1 -- 172.21.15.26:0/2999964688 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v24) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9c880457e0 con 0x7f9ca013dd40 2022-07-20T02:25:59.235 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:25:59.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 msgr2=0x7f9c84055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 0x7f9c84055f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9c9000c000 tx=0x7f9c90008040).stop 2022-07-20T02:25:59.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 msgr2=0x7f9ca0150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f9ca0145560 tx=0x7f9c88007c50).stop 2022-07-20T02:25:59.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 shutdown_connections 2022-07-20T02:25:59.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ca013dd40 0x7f9ca0150c30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c84053a70 0x7f9c84055f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ca01449d0 0x7f9ca00b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 --2- 172.21.15.26:0/2999964688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ca0065980 0x7f9ca0147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 >> 172.21.15.26:0/2999964688 conn(0x7f9ca005c580 msgr2=0x7f9ca0065db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 shutdown_connections 2022-07-20T02:25:59.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.228+0000 7f9ca6aef700 1 -- 172.21.15.26:0/2999964688 wait complete. 2022-07-20T02:25:59.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:25:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 Processor -- start 2022-07-20T02:25:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- start start 2022-07-20T02:25:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244060a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc244061010 0x7fc2440aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 0x7fc2441447e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2440b1d00 con 0x7fc244061010 2022-07-20T02:25:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2440ac430 con 0x7fc2440b0460 2022-07-20T02:25:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440b2ea0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc244061010 0x7fc2440aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc244061010 0x7fc2440aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35876/0 (socket says 172.21.15.26:35876) 2022-07-20T02:25:59.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 -- 172.21.15.26:0/4174009363 learned_addr learned my addr 172.21.15.26:0/4174009363 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc243fff700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244060a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc248884700 1 --2- 172.21.15.26:0/4174009363 >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 0x7fc2441447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 -- 172.21.15.26:0/4174009363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 msgr2=0x7fc244060a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244060a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 -- 172.21.15.26:0/4174009363 >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 msgr2=0x7fc2441447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/4174009363 >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 0x7fc2441447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 -- 172.21.15.26:0/4174009363 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440ac190 con 0x7fc244061010 2022-07-20T02:25:59.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc248884700 1 --2- 172.21.15.26:0/4174009363 >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 0x7fc2441447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:59.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc244061010 0x7fc2440aff70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c006eb0).ready entity=mon.0 client_cookie=c9e5951ad7ad4a7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc2437fe700 1 -- 172.21.15.26:0/4174009363 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00d570 con 0x7fc244061010 2022-07-20T02:25:59.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc2437fe700 1 -- 172.21.15.26:0/4174009363 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c000e90 con 0x7fc244061010 2022-07-20T02:25:59.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc2437fe700 1 -- 172.21.15.26:0/4174009363 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00d900 con 0x7fc244061010 2022-07-20T02:25:59.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- 172.21.15.26:0/4174009363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc244061010 msgr2=0x7fc2440aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc244061010 0x7fc2440aff70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c006eb0).stop 2022-07-20T02:25:59.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- 172.21.15.26:0/4174009363 shutdown_connections 2022-07-20T02:25:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/4174009363 >> v2:172.21.15.120:3300/0 conn(0x7fc2440b0460 0x7fc2441447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc244061010 0x7fc2440aff70 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c006eb0).stop 2022-07-20T02:25:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/4174009363 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244060a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- 172.21.15.26:0/4174009363 >> 172.21.15.26:0/4174009363 conn(0x7fc2440b60a0 msgr2=0x7fc2440b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- 172.21.15.26:0/4174009363 shutdown_connections 2022-07-20T02:25:59.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.352+0000 7fc24a2e7700 1 -- 172.21.15.26:0/4174009363 wait complete. 2022-07-20T02:25:59.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 Processor -- start 2022-07-20T02:25:59.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 -- start start 2022-07-20T02:25:59.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 0x7fc244141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2440aa170 0x7fc244062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440aa6f0 con 0x7fc2440b0460 2022-07-20T02:25:59.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440ab0d0 con 0x7fc2440aa170 2022-07-20T02:25:59.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc24a2e7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440ab370 con 0x7fc2440b2ac0 2022-07-20T02:25:59.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc243fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 0x7fc244141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57398/0 (socket says 172.21.15.26:57398) 2022-07-20T02:25:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc243fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 0x7fc244141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35880/0 (socket says 172.21.15.26:35880) 2022-07-20T02:25:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 -- 172.21.15.26:0/3282788588 learned_addr learned my addr 172.21.15.26:0/3282788588 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc248884700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2440aa170 0x7fc244062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 -- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2440aa170 msgr2=0x7fc244062e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2440aa170 0x7fc244062e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 -- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 msgr2=0x7fc244141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 0x7fc244141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 -- 172.21.15.26:0/3282788588 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c00b1e0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc23b7fe700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.356+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c006be0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2440a9cd0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0068b0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00dea0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2441420d0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2440a9e60 con 0x7fc2440b2ac0 2022-07-20T02:25:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc22c01a040 con 0x7fc2440b2ac0 2022-07-20T02:25:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc241ffb700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 0x7fc228055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc22c06ce60 con 0x7fc2440b2ac0 2022-07-20T02:25:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc243fff700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 0x7fc228055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.360+0000 7fc243fff700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 0x7fc228055e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc23400b000 tx=0x7fc234006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.372+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc22c043bc0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.500+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc2440b13a0 con 0x7fc2440b2ac0 2022-07-20T02:25:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.500+0000 7fc241ffb700 1 -- 172.21.15.26:0/3282788588 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc22c023020 con 0x7fc2440b2ac0 2022-07-20T02:25:59.517 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:59.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 msgr2=0x7fc228055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 0x7fc228055e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc23400b000 tx=0x7fc234006df0).stop 2022-07-20T02:25:59.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 msgr2=0x7fc244062980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c015000).stop 2022-07-20T02:25:59.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 shutdown_connections 2022-07-20T02:25:59.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc228053950 0x7fc228055e10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2440aa170 0x7fc244062e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2440b2ac0 0x7fc244062980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 --2- 172.21.15.26:0/3282788588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2440b0460 0x7fc244141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 >> 172.21.15.26:0/3282788588 conn(0x7fc2440b60a0 msgr2=0x7fc2440b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 shutdown_connections 2022-07-20T02:25:59.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.504+0000 7fc24a2e7700 1 -- 172.21.15.26:0/3282788588 wait complete. 2022-07-20T02:25:59.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:25:59.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 Processor -- start 2022-07-20T02:25:59.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 -- start start 2022-07-20T02:25:59.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f936405cfd0 0x7f93641407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93640b1160 0x7f9364140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9364147a00 con 0x7f9364146e80 2022-07-20T02:25:59.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93641435b0 con 0x7f936405cfd0 2022-07-20T02:25:59.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9368a0f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936405d3b0 con 0x7f93640b1160 2022-07-20T02:25:59.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f935bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93640b1160 0x7f9364140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f936405cfd0 0x7f93641407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9362d9d700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9364146e80 0x7f93641472f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49384/0 (socket says 172.21.15.26:49384) 2022-07-20T02:25:59.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9362d9d700 1 -- 172.21.15.26:0/1234236692 learned_addr learned my addr 172.21.15.26:0/1234236692 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:25:59.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9361d9b700 1 -- 172.21.15.26:0/1234236692 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840280726 0 0) 0x7f9364147a00 con 0x7f9364146e80 2022-07-20T02:25:59.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9361d9b700 1 -- 172.21.15.26:0/1234236692 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9344003260 con 0x7f9364146e80 2022-07-20T02:25:59.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 -- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93640b1160 msgr2=0x7f9364140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 --2- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93640b1160 0x7f9364140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 -- 172.21.15.26:0/1234236692 >> v1:172.21.15.120:6789/0 conn(0x7f9364146e80 legacy=0x7f93641472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 -- 172.21.15.26:0/1234236692 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93640b4bc0 con 0x7f936405cfd0 2022-07-20T02:25:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f936259c700 1 --2- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f936405cfd0 0x7f93641407c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9354002f70 tx=0x7f9354006df0).ready entity=mon.1 client_cookie=8ad38b746185471f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9361d9b700 1 -- 172.21.15.26:0/1234236692 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400d4a0 con 0x7f936405cfd0 2022-07-20T02:25:59.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9361d9b700 1 -- 172.21.15.26:0/1234236692 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9354000e90 con 0x7f936405cfd0 2022-07-20T02:25:59.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.668+0000 7f9361d9b700 1 -- 172.21.15.26:0/1234236692 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400d7f0 con 0x7f936405cfd0 2022-07-20T02:25:59.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f936405cfd0 msgr2=0x7f93641407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f936405cfd0 0x7f93641407c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9354002f70 tx=0x7f9354006df0).stop 2022-07-20T02:25:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- 172.21.15.26:0/1234236692 shutdown_connections 2022-07-20T02:25:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93640b1160 0x7f9364140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1234236692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f936405cfd0 0x7f93641407c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- 172.21.15.26:0/1234236692 >> 172.21.15.26:0/1234236692 conn(0x7f9364066f70 msgr2=0x7f936405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- 172.21.15.26:0/1234236692 shutdown_connections 2022-07-20T02:25:59.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- 172.21.15.26:0/1234236692 wait complete. 2022-07-20T02:25:59.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 Processor -- start 2022-07-20T02:25:59.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- start start 2022-07-20T02:25:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f936405cfd0 0x7f93641421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 0x7f936413be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641524d0 con 0x7f9364146e80 2022-07-20T02:25:59.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9364141dd0 con 0x7f93640b1160 2022-07-20T02:25:59.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9368a0f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93641526c0 con 0x7f936405cfd0 2022-07-20T02:25:59.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35898/0 (socket says 172.21.15.26:35898) 2022-07-20T02:25:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 -- 172.21.15.26:0/1123443386 learned_addr learned my addr 172.21.15.26:0/1123443386 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f936259c700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f936405cfd0 0x7f93641421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f935bfff700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 0x7f936413be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 -- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f936405cfd0 msgr2=0x7f93641421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f936405cfd0 0x7f93641421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 -- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 msgr2=0x7f936413be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 0x7f936413be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f9362d9d700 1 -- 172.21.15.26:0/1123443386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935400b180 con 0x7f9364146e80 2022-07-20T02:25:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.672+0000 7f935bfff700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 0x7f936413be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:25:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f9362d9d700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f935c001490 tx=0x7f935c007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c005ba0 con 0x7f9364146e80 2022-07-20T02:25:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c005d00 con 0x7f9364146e80 2022-07-20T02:25:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9364152960 con 0x7f9364146e80 2022-07-20T02:25:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c0072c0 con 0x7f9364146e80 2022-07-20T02:25:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93641532f0 con 0x7f9364146e80 2022-07-20T02:25:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f935c010070 con 0x7f9364146e80 2022-07-20T02:25:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9364152af0 con 0x7f9364146e80 2022-07-20T02:25:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 0x7f9348055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f935c06bbd0 con 0x7f9364146e80 2022-07-20T02:25:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f936259c700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 0x7f9348055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.676+0000 7f936259c700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 0x7f9348055ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9354018000 tx=0x7f9354014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.684+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f935c042800 con 0x7f9364146e80 2022-07-20T02:25:59.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.808+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93640574e0 con 0x7f9364146e80 2022-07-20T02:25:59.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.808+0000 7f935b7fe700 1 -- 172.21.15.26:0/1123443386 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f935c045e20 con 0x7f9364146e80 2022-07-20T02:25:59.814 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:25:59.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 msgr2=0x7f9348055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 0x7f9348055ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9354018000 tx=0x7f9354014000).stop 2022-07-20T02:25:59.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 msgr2=0x7f9364152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f935c001490 tx=0x7f935c007e70).stop 2022-07-20T02:25:59.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 shutdown_connections 2022-07-20T02:25:59.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9364146e80 0x7f9364152e00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f93480539e0 0x7f9348055ea0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93640b1160 0x7f936413be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 --2- 172.21.15.26:0/1123443386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f936405cfd0 0x7f93641421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 >> 172.21.15.26:0/1123443386 conn(0x7f9364066f70 msgr2=0x7f93640b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 shutdown_connections 2022-07-20T02:25:59.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.812+0000 7f9368a0f700 1 -- 172.21.15.26:0/1123443386 wait complete. 2022-07-20T02:25:59.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:25:59.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 Processor -- start 2022-07-20T02:25:59.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 -- start start 2022-07-20T02:25:59.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 0x7f3cd8143350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3cd80952f0 0x7f3cd80b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd805f840 0x7f3cd80b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd80973c0 con 0x7f3cd80952f0 2022-07-20T02:25:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd8142c50 con 0x7f3cd8096fe0 2022-07-20T02:25:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.952+0000 7f3cdf0ea700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b2f30 con 0x7f3cd805f840 2022-07-20T02:25:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3cd80952f0 0x7f3cd80b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3cd80952f0 0x7f3cd80b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35906/0 (socket says 172.21.15.26:35906) 2022-07-20T02:25:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 -- 172.21.15.26:0/1677684924 learned_addr learned my addr 172.21.15.26:0/1677684924 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cde0e8700 1 --2- 172.21.15.26:0/1677684924 >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 0x7f3cd8143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cde8e9700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd805f840 0x7f3cd80b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 -- 172.21.15.26:0/1677684924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd805f840 msgr2=0x7f3cd80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd805f840 0x7f3cd80b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 -- 172.21.15.26:0/1677684924 >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 msgr2=0x7f3cd8143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:25:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/1677684924 >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 0x7f3cd8143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 -- 172.21.15.26:0/1677684924 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd80b2c90 con 0x7f3cd80952f0 2022-07-20T02:25:59.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cde0e8700 1 --2- 172.21.15.26:0/1677684924 >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 0x7f3cd8143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:25:59.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd80952f0 0x7f3cd80b06d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3cc8002f70 tx=0x7f3cc8006eb0).ready entity=mon.0 client_cookie=889a787a884fda93 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd0e6700 1 -- 172.21.15.26:0/1677684924 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc800d570 con 0x7f3cd80952f0 2022-07-20T02:25:59.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd0e6700 1 -- 172.21.15.26:0/1677684924 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc8000e90 con 0x7f3cd80952f0 2022-07-20T02:25:59.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdd0e6700 1 -- 172.21.15.26:0/1677684924 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc800d900 con 0x7f3cd80952f0 2022-07-20T02:25:59.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/1677684924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd80952f0 msgr2=0x7f3cd80b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd80952f0 0x7f3cd80b06d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3cc8002f70 tx=0x7f3cc8006eb0).stop 2022-07-20T02:25:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/1677684924 shutdown_connections 2022-07-20T02:25:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd805f840 0x7f3cd80b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/1677684924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd80952f0 0x7f3cd80b06d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/1677684924 >> v2:172.21.15.120:3300/0 conn(0x7f3cd8096fe0 0x7f3cd8143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/1677684924 >> 172.21.15.26:0/1677684924 conn(0x7f3cd8063030 msgr2=0x7f3cd8063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:25:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/1677684924 shutdown_connections 2022-07-20T02:25:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.956+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/1677684924 wait complete. 2022-07-20T02:25:59.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 Processor -- start 2022-07-20T02:25:59.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 -- start start 2022-07-20T02:25:59.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 0x7f3cd809ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd8096fe0 0x7f3cd809b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:25:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80994c0 con 0x7f3cd805f840 2022-07-20T02:25:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd809a4c0 con 0x7f3cd80952f0 2022-07-20T02:25:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdf0ea700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd8099020 con 0x7f3cd8096fe0 2022-07-20T02:25:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35908/0 (socket says 172.21.15.26:35908) 2022-07-20T02:25:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 -- 172.21.15.26:0/3174312435 learned_addr learned my addr 172.21.15.26:0/3174312435 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:25:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde8e9700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd8096fe0 0x7f3cd809b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 0x7f3cd809ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:25:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 -- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd8096fe0 msgr2=0x7f3cd809b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd8096fe0 0x7f3cd809b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 -- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 msgr2=0x7f3cd809ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:25:59.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 0x7f3cd809ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:25:59.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 -- 172.21.15.26:0/3174312435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd400c310 con 0x7f3cd805f840 2022-07-20T02:25:59.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 0x7f3cd809ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:25:59.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.960+0000 7f3cde0e8700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3cd400d000 tx=0x7f3cd4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:25:59.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd400aa20 con 0x7f3cd805f840 2022-07-20T02:25:59.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd400ab80 con 0x7f3cd805f840 2022-07-20T02:25:59.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc800b1e0 con 0x7f3cd805f840 2022-07-20T02:26:00.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd40113f0 con 0x7f3cd805f840 2022-07-20T02:26:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd8140640 con 0x7f3cd805f840 2022-07-20T02:26:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd8065500 con 0x7f3cd805f840 2022-07-20T02:26:00.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cd4021070 con 0x7f3cd805f840 2022-07-20T02:26:00.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 0x7f3cb8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 0x7f3cb8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.964+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3cd406c430 con 0x7f3cd805f840 2022-07-20T02:26:00.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.968+0000 7f3cdd8e7700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 0x7f3cb8055fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3cc801a000 tx=0x7f3cc8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:00.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:25:59.972+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cd40430e0 con 0x7f3cd805f840 2022-07-20T02:26:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.096+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3cd80996c0 con 0x7f3cd805f840 2022-07-20T02:26:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.096+0000 7f3cc77fe700 1 -- 172.21.15.26:0/3174312435 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3cd4046700 con 0x7f3cd805f840 2022-07-20T02:26:00.102 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:26:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 msgr2=0x7f3cb8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 0x7f3cb8055fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3cc801a000 tx=0x7f3cc8015000).stop 2022-07-20T02:26:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 msgr2=0x7f3cd8064cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3cd400d000 tx=0x7f3cd4008040).stop 2022-07-20T02:26:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 shutdown_connections 2022-07-20T02:26:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3cb8053b00 0x7f3cb8055fc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd8096fe0 0x7f3cd809b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd80952f0 0x7f3cd809ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 --2- 172.21.15.26:0/3174312435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cd805f840 0x7f3cd8064cd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 >> 172.21.15.26:0/3174312435 conn(0x7f3cd8063030 msgr2=0x7f3cd80944a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:00.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 shutdown_connections 2022-07-20T02:26:00.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.100+0000 7f3cdf0ea700 1 -- 172.21.15.26:0/3174312435 wait complete. 2022-07-20T02:26:00.174 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T02:26:00.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-20T02:26:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 Processor -- start 2022-07-20T02:26:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 -- start start 2022-07-20T02:26:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fceb40a8c40 0x7fceb40af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb40b62d0 con 0x7fceb40a7fc0 2022-07-20T02:26:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb414aac0 con 0x7fceb40a8c40 2022-07-20T02:26:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb414ad60 con 0x7fceb40b5ef0 2022-07-20T02:26:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb2ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fceb40a7fc0 0x7fceb40aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35198/0 (socket says 172.21.15.26:35198) 2022-07-20T02:26:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb2ffd700 1 -- 172.21.15.26:0/2515983113 learned_addr learned my addr 172.21.15.26:0/2515983113 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 --2- 172.21.15.26:0/2515983113 >> v2:172.21.15.120:3300/0 conn(0x7fceb40a8c40 0x7fceb40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb37fe700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb27fc700 1 -- 172.21.15.26:0/2515983113 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430136709 0 0) 0x7fceb40b62d0 con 0x7fceb40a7fc0 2022-07-20T02:26:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb27fc700 1 -- 172.21.15.26:0/2515983113 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce94003260 con 0x7fceb40a7fc0 2022-07-20T02:26:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 -- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 msgr2=0x7fceb41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 -- 172.21.15.26:0/2515983113 >> v1:172.21.15.26:6789/0 conn(0x7fceb40a7fc0 legacy=0x7fceb40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 -- 172.21.15.26:0/2515983113 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb414b000 con 0x7fceb40a8c40 2022-07-20T02:26:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb3fff700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb40af550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fceac002f70 tx=0x7fceac006df0).ready entity=mon.1 client_cookie=83388f945fd48eec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb27fc700 1 -- 172.21.15.26:0/2515983113 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac00d4a0 con 0x7fceb40a8c40 2022-07-20T02:26:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb27fc700 1 -- 172.21.15.26:0/2515983113 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac000e90 con 0x7fceb40a8c40 2022-07-20T02:26:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb27fc700 1 -- 172.21.15.26:0/2515983113 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac00d7f0 con 0x7fceb40a8c40 2022-07-20T02:26:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 -- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 msgr2=0x7fceb40af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.296+0000 7fceb893c700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb40af550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fceac002f70 tx=0x7fceac006df0).stop 2022-07-20T02:26:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 -- 172.21.15.26:0/2515983113 shutdown_connections 2022-07-20T02:26:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb40af550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 --2- 172.21.15.26:0/2515983113 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 -- 172.21.15.26:0/2515983113 >> 172.21.15.26:0/2515983113 conn(0x7fceb4060310 msgr2=0x7fceb40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 -- 172.21.15.26:0/2515983113 shutdown_connections 2022-07-20T02:26:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 -- 172.21.15.26:0/2515983113 wait complete. 2022-07-20T02:26:00.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 Processor -- start 2022-07-20T02:26:00.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.300+0000 7fceb893c700 1 -- start start 2022-07-20T02:26:00.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb4144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb4146050 con 0x7fceb40a7fc0 2022-07-20T02:26:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb4157410 con 0x7fceb40a8c40 2022-07-20T02:26:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40aba00 con 0x7fceb40b5ef0 2022-07-20T02:26:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35922/0 (socket says 172.21.15.26:35922) 2022-07-20T02:26:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 -- 172.21.15.26:0/2211775577 learned_addr learned my addr 172.21.15.26:0/2211775577 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb2ffd700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb4144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb3fff700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 -- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 msgr2=0x7fceb41476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41476b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 -- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 msgr2=0x7fceb4144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb4144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 -- 172.21.15.26:0/2211775577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac00b180 con 0x7fceb40a7fc0 2022-07-20T02:26:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb2ffd700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb4144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb37fe700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fcea400c000 tx=0x7fcea40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea400c6a0 con 0x7fceb40a7fc0 2022-07-20T02:26:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea400c800 con 0x7fceb40a7fc0 2022-07-20T02:26:00.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea400cb60 con 0x7fceb40a7fc0 2022-07-20T02:26:00.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb40abca0 con 0x7fceb40a7fc0 2022-07-20T02:26:00.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.304+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb405c760 con 0x7fceb40a7fc0 2022-07-20T02:26:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb414b2a0 con 0x7fceb40a7fc0 2022-07-20T02:26:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcea4022070 con 0x7fceb40a7fc0 2022-07-20T02:26:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb0ff9700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 0x7fce98055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcea406c750 con 0x7fceb40a7fc0 2022-07-20T02:26:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb2ffd700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 0x7fce98055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.308+0000 7fceb2ffd700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 0x7fce98055fc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fce9c00b000 tx=0x7fce9c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.316+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcea4043510 con 0x7fceb40a7fc0 2022-07-20T02:26:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:00.480+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7fceb40574e0 con 0x7fceb40a7fc0 2022-07-20T02:26:01.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.408+0000 7fceb0ff9700 1 -- 172.21.15.26:0/2211775577 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v45) v1 ==== 142+0+0 (secure 0 0 0) 0x7fcea4046b30 con 0x7fceb40a7fc0 2022-07-20T02:26:01.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 msgr2=0x7fce98055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 0x7fce98055fc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fce9c00b000 tx=0x7fce9c009040).stop 2022-07-20T02:26:01.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 msgr2=0x7fceb40ab2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fcea400c000 tx=0x7fcea40091d0).stop 2022-07-20T02:26:01.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 shutdown_connections 2022-07-20T02:26:01.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fce98053b00 0x7fce98055fc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb40b5ef0 0x7fceb41476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb40a8c40 0x7fceb4144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 --2- 172.21.15.26:0/2211775577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fceb40a7fc0 0x7fceb40ab2f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.412+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 >> 172.21.15.26:0/2211775577 conn(0x7fceb4060310 msgr2=0x7fceb41489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:01.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.416+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 shutdown_connections 2022-07-20T02:26:01.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.416+0000 7fceb893c700 1 -- 172.21.15.26:0/2211775577 wait complete. 2022-07-20T02:26:01.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:26:01.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 Processor -- start 2022-07-20T02:26:01.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 -- start start 2022-07-20T02:26:01.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09a805fa50 0x7f09a8140a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a8143d50 con 0x7f09a80ae8d0 2022-07-20T02:26:01.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a8143fc0 con 0x7f09a805fa50 2022-07-20T02:26:01.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ae8aa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8144260 con 0x7f09a8060760 2022-07-20T02:26:01.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09a805fa50 0x7f09a8140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09a805fa50 0x7f09a8140a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35934/0 (socket says 172.21.15.26:35934) 2022-07-20T02:26:01.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 -- 172.21.15.26:0/3936252180 learned_addr learned my addr 172.21.15.26:0/3936252180 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:01.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f099ffff700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 -- 172.21.15.26:0/3936252180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 msgr2=0x7f09a8140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:01.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 -- 172.21.15.26:0/3936252180 >> v1:172.21.15.26:6789/0 conn(0x7f09a80ae8d0 legacy=0x7f09a80afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 -- 172.21.15.26:0/3936252180 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80af4d0 con 0x7f09a805fa50 2022-07-20T02:26:01.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09a77fe700 1 -- 172.21.15.26:0/3936252180 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248391675 0 0) 0x7f09a8143d50 con 0x7f09a80ae8d0 2022-07-20T02:26:01.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f099ffff700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:01.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09ace47700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a8140a40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f09a00032f0 tx=0x7f09a0009290).ready entity=mon.0 client_cookie=f7b2af88097d2586 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:01.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.568+0000 7f09a77fe700 1 -- 172.21.15.26:0/3936252180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a000b8a0 con 0x7f09a805fa50 2022-07-20T02:26:01.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a77fe700 1 -- 172.21.15.26:0/3936252180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a000ba00 con 0x7f09a805fa50 2022-07-20T02:26:01.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a77fe700 1 -- 172.21.15.26:0/3936252180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a000bb70 con 0x7f09a805fa50 2022-07-20T02:26:01.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- 172.21.15.26:0/3936252180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 msgr2=0x7f09a8140a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a8140a40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f09a00032f0 tx=0x7f09a0009290).stop 2022-07-20T02:26:01.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- 172.21.15.26:0/3936252180 shutdown_connections 2022-07-20T02:26:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a8140a40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/3936252180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- 172.21.15.26:0/3936252180 >> 172.21.15.26:0/3936252180 conn(0x7f09a80b66e0 msgr2=0x7f09a80b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- 172.21.15.26:0/3936252180 shutdown_connections 2022-07-20T02:26:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- 172.21.15.26:0/3936252180 wait complete. 2022-07-20T02:26:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 Processor -- start 2022-07-20T02:26:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- start start 2022-07-20T02:26:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a80ae8d0 0x7f09a8155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80639c0 con 0x7f09a805fa50 2022-07-20T02:26:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8063c60 con 0x7f09a80ae8d0 2022-07-20T02:26:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ae8aa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8063f00 con 0x7f09a8060760 2022-07-20T02:26:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35936/0 (socket says 172.21.15.26:35936) 2022-07-20T02:26:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 -- 172.21.15.26:0/1476138678 learned_addr learned my addr 172.21.15.26:0/1476138678 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09ace47700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a80ae8d0 0x7f09a8155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f099ffff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 -- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 msgr2=0x7f09a8145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 -- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a80ae8d0 msgr2=0x7f09a8155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a80ae8d0 0x7f09a8155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 -- 172.21.15.26:0/1476138678 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0009530 con 0x7f09a805fa50 2022-07-20T02:26:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f099ffff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.572+0000 7f09a7fff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0998000fe0 tx=0x7f0998007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0998012400 con 0x7f09a805fa50 2022-07-20T02:26:01.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998012560 con 0x7f09a805fa50 2022-07-20T02:26:01.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a80641a0 con 0x7f09a805fa50 2022-07-20T02:26:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09980128c0 con 0x7f09a805fa50 2022-07-20T02:26:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a8146190 con 0x7f09a805fa50 2022-07-20T02:26:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a80af770 con 0x7f09a805fa50 2022-07-20T02:26:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0998019030 con 0x7f09a805fa50 2022-07-20T02:26:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 0x7f098c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f099806b380 con 0x7f09a805fa50 2022-07-20T02:26:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.576+0000 7f099ffff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 0x7f098c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.580+0000 7f099ffff700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 0x7f098c055ea0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f099000c000 tx=0x7f0990008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.584+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0998042140 con 0x7f09a805fa50 2022-07-20T02:26:01.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.712+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f09a80574e0 con 0x7f09a805fa50 2022-07-20T02:26:01.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.720+0000 7f09a5ffb700 1 -- 172.21.15.26:0/1476138678 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0998045760 con 0x7f09a805fa50 2022-07-20T02:26:01.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 msgr2=0x7f098c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 0x7f098c055ea0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f099000c000 tx=0x7f0990008040).stop 2022-07-20T02:26:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 msgr2=0x7f09a80b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0998000fe0 tx=0x7f0998007c50).stop 2022-07-20T02:26:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 shutdown_connections 2022-07-20T02:26:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f098c0539e0 0x7f098c055ea0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a80ae8d0 0x7f09a8155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a8060760 0x7f09a8145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 --2- 172.21.15.26:0/1476138678 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09a805fa50 0x7f09a80b4e90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 >> 172.21.15.26:0/1476138678 conn(0x7f09a80b66e0 msgr2=0x7f09a8060b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 shutdown_connections 2022-07-20T02:26:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:01.724+0000 7f09ae8aa700 1 -- 172.21.15.26:0/1476138678 wait complete. 2022-07-20T02:26:01.789 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:26:01.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:26:01.852 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:26:01.920 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:26:01.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:26:01.985 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:26:01.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:01.987 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:01.988 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.006 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:26:02.007 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:26:02.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.009 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.069 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.094 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:26:02.095 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:26:02.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.097 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.098 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.143 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T02:26:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:26:02.146 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T02:26:02.147 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:26:02.148 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:26:02.149 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.184 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.185 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.237 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:26:02.238 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:26:02.239 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:26:02.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:26:02.241 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:02.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.260 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.261 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.272 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:26:02.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:26:02.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.274 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.275 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.283 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:26:02.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:26:02.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.287 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.288 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.338 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2022-07-20T02:26:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:26:02.351 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2022-07-20T02:26:02.353 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:26:02.358 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:26:02.360 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:02.435 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:02.436 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.468 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:26:02.468 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:26:02.469 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:26:02.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:26:02.471 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:26:02.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:26:02.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 Processor -- start 2022-07-20T02:26:02.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 -- start start 2022-07-20T02:26:02.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff53c143090 0x7ff53c098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff53c0b0810 0x7ff53c0950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53c143470 con 0x7ff53c0b0c50 2022-07-20T02:26:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c05fd90 con 0x7ff53c0b0810 2022-07-20T02:26:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff5416a1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c097d50 con 0x7ff53c143090 2022-07-20T02:26:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff53c0b0810 0x7ff53c0950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53b7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff53c0b0c50 0x7ff53c0955d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35226/0 (socket says 172.21.15.26:35226) 2022-07-20T02:26:02.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53b7fe700 1 -- 172.21.15.26:0/2400099334 learned_addr learned my addr 172.21.15.26:0/2400099334 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:02.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53affd700 1 --2- 172.21.15.26:0/2400099334 >> v2:172.21.15.120:3300/0 conn(0x7ff53c143090 0x7ff53c098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 -- 172.21.15.26:0/2400099334 >> v2:172.21.15.120:3300/0 conn(0x7ff53c143090 msgr2=0x7ff53c098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:02.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 --2- 172.21.15.26:0/2400099334 >> v2:172.21.15.120:3300/0 conn(0x7ff53c143090 0x7ff53c098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:02.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 -- 172.21.15.26:0/2400099334 >> v1:172.21.15.26:6789/0 conn(0x7ff53c0b0c50 legacy=0x7ff53c0955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:02.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 -- 172.21.15.26:0/2400099334 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c097ff0 con 0x7ff53c0b0810 2022-07-20T02:26:02.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff53a7fc700 1 --2- 172.21.15.26:0/2400099334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0810 0x7ff53c0950e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524006df0).ready entity=mon.0 client_cookie=5bf3f466ea96bfb1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:02.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff539ffb700 1 -- 172.21.15.26:0/2400099334 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff52400d570 con 0x7ff53c0b0810 2022-07-20T02:26:02.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff539ffb700 1 -- 172.21.15.26:0/2400099334 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524000e90 con 0x7ff53c0b0810 2022-07-20T02:26:02.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.644+0000 7ff539ffb700 1 -- 172.21.15.26:0/2400099334 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff52400d8c0 con 0x7ff53c0b0810 2022-07-20T02:26:02.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- 172.21.15.26:0/2400099334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0810 msgr2=0x7ff53c0950e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:02.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- 172.21.15.26:0/2400099334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0810 0x7ff53c0950e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524006df0).stop 2022-07-20T02:26:02.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- 172.21.15.26:0/2400099334 shutdown_connections 2022-07-20T02:26:02.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- 172.21.15.26:0/2400099334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0810 0x7ff53c0950e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:02.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- 172.21.15.26:0/2400099334 >> v2:172.21.15.120:3300/0 conn(0x7ff53c143090 0x7ff53c098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:02.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- 172.21.15.26:0/2400099334 >> 172.21.15.26:0/2400099334 conn(0x7ff53c0b5e70 msgr2=0x7ff53c0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:02.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- 172.21.15.26:0/2400099334 shutdown_connections 2022-07-20T02:26:02.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- 172.21.15.26:0/2400099334 wait complete. 2022-07-20T02:26:02.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 Processor -- start 2022-07-20T02:26:02.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- start start 2022-07-20T02:26:02.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c0b0810 0x7ff53c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c143090 0x7ff53c094630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0a9d20 con 0x7ff53c0b0c50 2022-07-20T02:26:02.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0afa40 con 0x7ff53c0b0810 2022-07-20T02:26:02.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.648+0000 7ff5416a1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0af7a0 con 0x7ff53c143090 2022-07-20T02:26:02.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35952/0 (socket says 172.21.15.26:35952) 2022-07-20T02:26:02.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 -- 172.21.15.26:0/4270658129 learned_addr learned my addr 172.21.15.26:0/4270658129 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:02.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53affd700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c0b0810 0x7ff53c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53b7fe700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c143090 0x7ff53c094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 -- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c143090 msgr2=0x7ff53c094630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c143090 0x7ff53c094630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:02.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 -- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c0b0810 msgr2=0x7ff53c0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c0b0810 0x7ff53c0b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:02.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 -- 172.21.15.26:0/4270658129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52400b180 con 0x7ff53c0b0c50 2022-07-20T02:26:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff53a7fc700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff524016070 con 0x7ff53c0b0c50 2022-07-20T02:26:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52400d8f0 con 0x7ff53c0b0c50 2022-07-20T02:26:02.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff524013da0 con 0x7ff53c0b0c50 2022-07-20T02:26:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c0a9f80 con 0x7ff53c0b0c50 2022-07-20T02:26:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c094b20 con 0x7ff53c0b0c50 2022-07-20T02:26:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c095ce0 con 0x7ff53c0b0c50 2022-07-20T02:26:02.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff52400f030 con 0x7ff53c0b0c50 2022-07-20T02:26:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.652+0000 7ff533fff700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 0x7ff520055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.656+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff52406cca0 con 0x7ff53c0b0c50 2022-07-20T02:26:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.656+0000 7ff53affd700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 0x7ff520055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:02.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.656+0000 7ff53affd700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 0x7ff520055e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff52c00b000 tx=0x7ff52c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:02.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.660+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff524071070 con 0x7ff53c0b0c50 2022-07-20T02:26:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.788+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff53c0574e0 con 0x7ff53c0b0c50 2022-07-20T02:26:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff533fff700 1 -- 172.21.15.26:0/4270658129 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v24) v1 ==== 81+0+34113 (secure 0 0 0) 0x7ff524023070 con 0x7ff53c0b0c50 2022-07-20T02:26:02.799 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:02.800 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:26:02.801 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 24, 2022-07-20T02:26:02.803 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:26:02.804 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:26:02.804 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:25:35.945703+0000", 2022-07-20T02:26:02.805 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:26:02.805 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:26:02.806 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:26:02.806 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:26:02.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:26:02.807 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:26:02.808 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:26:02.808 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:26:02.809 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:26:02.809 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:26:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:26:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:26:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:26:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:26:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:26:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2022-07-20T02:26:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:26:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:26:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:26:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:26:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:26:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:26:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:26:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:26:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:26:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:26:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:26:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:26:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:26:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:26:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:26:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:26:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:26:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:26:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:26:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:26:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:26:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:26:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:26:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:26:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:26:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:26:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:26:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:26:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:26:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:26:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:26:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:26:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:26:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:26:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:26:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:26:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:26:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:26:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:26:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:26:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:26:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:26:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:26:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:26:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:26:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:26:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:26:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:26:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:26:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:26:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:26:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:26:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:26:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:26:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:26:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:26:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:26:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:26:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:26:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:26:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:26:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:26:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:26:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:26:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:26:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:26:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:26:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:26:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:26:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:26:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:26:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:26:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:26:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:26:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:26:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:26:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:26:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:26:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:26:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:26:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:26:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:26:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:26:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:26:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:26:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:26:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:26:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:26:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:26:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:26:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:26:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:26:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:26:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:26:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:26:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:26:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:26:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:26:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:26:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:26:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:26:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:26:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:26:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:26:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:26:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:26:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:26:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:26:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:26:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:26:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:26:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:26:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:26:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:26:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:26:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:26:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:26:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:26:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:26:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:26:02.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.942 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:26:02.943 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:26:02.943 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:26:02.943 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:26:02.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:26:02.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:26:02.944 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:26:02.945 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:26:02.945 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:26:02.946 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:26:02.946 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:26:02.947 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:26:02.947 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:26:02.951 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:26:02.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:02.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:02.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:02.954 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2022-07-20T02:26:02.955 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:26:02.956 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:31.063095+0000", 2022-07-20T02:26:02.956 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:26:02.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:26:02.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:26:02.958 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:26:02.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:26:02.961 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:26:02.962 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:26:02.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:26:02.967 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:26:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:26:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:26:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:26:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:26:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:26:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:26:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:26:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:26:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:26:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:26:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:26:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "24", 2022-07-20T02:26:03.001 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:26:03.001 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:26:03.001 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:26:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:26:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:26:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:26:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:26:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:26:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:26:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:26:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:26:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:26:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:26:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:26:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:26:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:26:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:26:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:26:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:26:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:26:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:26:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:26:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:26:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:26:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:26:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:26:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:26:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:26:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:26:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:26:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:26:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:26:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:26:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:26:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:26:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:26:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:26:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:26:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:26:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:26:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:26:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:26:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:26:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:26:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2022-07-20T02:26:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:26:03.034 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:33.651615+0000", 2022-07-20T02:26:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 5, 2022-07-20T02:26:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,ec_overwrites", 2022-07-20T02:26:03.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 3, 2022-07-20T02:26:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 4, 2022-07-20T02:26:03.036 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 3, 2022-07-20T02:26:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 1, 2022-07-20T02:26:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:26:03.037 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:26:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:26:03.038 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:26:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:26:03.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:26:03.040 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:26:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:26:03.041 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:26:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:26:03.042 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:26:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:26:03.043 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:26:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:26:03.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "24", 2022-07-20T02:26:03.045 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:26:03.046 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:26:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:26:03.047 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:26:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:26:03.048 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:26:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:26:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:26:03.049 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:26:03.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:26:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:26:03.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:26:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:26:03.053 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:26:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:26:03.054 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:26:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:26:03.059 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:26:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:26:03.060 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:26:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:26:03.061 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:26:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:26:03.062 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "cephfs_data", 2022-07-20T02:26:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:26:03.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:26:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.064 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:26:03.065 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:26:03.066 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:26:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:26:03.068 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:26:03.069 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:26:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:26:03.070 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:26:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 8192, 2022-07-20T02:26:03.071 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:26:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:26:03.072 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:26:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:26:03.073 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:26:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:26:03.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:26:03.076 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:26:03.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:26:03.078 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:26:03.079 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.080 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.082 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 22, 2022-07-20T02:26:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.086 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:26:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:26:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.094 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:26:03.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:26:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:26:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:26:03.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.117 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:26:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:26:03.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:26:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:26:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:26:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:26:03.126 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:26:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.127 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:26:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:26:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.130 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.131 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.132 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 22, 2022-07-20T02:26:03.133 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.134 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:26:03.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:26:03.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:26:03.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:26:03.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:26:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:26:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:26:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:26:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:26:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:26:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:26:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:26:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:26:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:26:03.171 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:26:03.171 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.171 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.172 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.172 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.173 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.173 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.174 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.174 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 16, 2022-07-20T02:26:03.174 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.175 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:26:03.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:26:03.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.182 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:26:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:26:03.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:26:03.190 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:26:03.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.194 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:26:03.196 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:26:03.198 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:26:03.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.201 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:26:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:26:03.202 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:26:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:26:03.203 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:26:03.211 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:26:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.212 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 22, 2022-07-20T02:26:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.214 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:26:03.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:26:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.218 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:26:03.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:26:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:26:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:26:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:26:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:26:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:26:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:26:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:26:03.274 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:26:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:26:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:26:03.276 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:26:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.277 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 22, 2022-07-20T02:26:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.278 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:26:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.280 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:26:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:26:03.288 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:26:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:26:03.295 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:26:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.300 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:26:03.305 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.306 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:26:03.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:26:03.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.308 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:26:03.308 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:26:03.318 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:26:03.318 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:26:03.319 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.319 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.319 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.320 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:26:03.320 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:26:03.320 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.321 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.322 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.322 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 19, 2022-07-20T02:26:03.322 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.322 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.323 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.323 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:26:03.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:26:03.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.326 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:26:03.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:26:03.328 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.329 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:26:03.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:26:03.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.332 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:26:03.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:26:03.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:26:03.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:26:03.336 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:26:03.336 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:26:03.336 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:26:03.337 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.337 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.337 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.338 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:26:03.338 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:26:03.338 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.339 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.339 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.339 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.340 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.340 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 22, 2022-07-20T02:26:03.340 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.341 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:26:03.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:26:03.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.377 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.378 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:26:03.378 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:26:03.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.379 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.380 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:26:03.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:26:03.382 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.383 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:26:03.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:26:03.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:26:03.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:26:03.389 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:26:03.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:26:03.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:26:03.390 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.390 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.390 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:26:03.391 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:26:03.391 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:26:03.392 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:26:03.392 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:26:03.392 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:26:03.392 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:26:03.392 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:26:03.393 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:26:03.393 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 16, 2022-07-20T02:26:03.393 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:26:03.393 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:26:03.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:26:03.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:26:03.406 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:26:03.408 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.409 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:26:03.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:26:03.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:26:03.414 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.415 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.416 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:26:03.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:26:03.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:26:03.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.422 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:26:03.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:26:03.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:26:03.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:26:03.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:26:03.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:26:03.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:26:03.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:26:03.428 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:26:03.429 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:26:03.430 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:26:03.430 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:26:03.431 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:26:03.431 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:26:03.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:03.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.433 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:26:03.433 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:26:03.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.434 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:26:03.435 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.436 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.436 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.437 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.437 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.438 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:26:03.438 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.441 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:26:03.441 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.442 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.442 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.443 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.444 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.444 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:26:03.444 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.448 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:26:03.448 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.449 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.449 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.449 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.451 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.451 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:26:03.451 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.453 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:26:03.453 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.454 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.454 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.462 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.463 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.463 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:26:03.464 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.465 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:26:03.465 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.467 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.468 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.468 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.469 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.469 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:26:03.469 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.471 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:26:03.471 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.473 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.474 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.474 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.475 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:26:03.475 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.477 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:26:03.477 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.478 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.478 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.486 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.487 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:26:03.488 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:03.489 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:26:03.489 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:26:03.490 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:26:03.490 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:26:03.491 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:26:03.491 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:26:03.492 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:26:03.493 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:26:03.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:26:03.494 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:26:03.495 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:26:03.495 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:26:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:26:03.497 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:26:03.497 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:26:03.499 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:26:03.499 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:26:03.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:26:03.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:26:03.501 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:26:03.502 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:26:03.503 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:26:03.504 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:26:03.504 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:26:03.505 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:26:03.505 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:26:03.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.506 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:26:03.507 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:26:03.507 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:26:03.508 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:26:03.509 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:26:03.510 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:26:03.510 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:26:03.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:03.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:03.511 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:26:03.512 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:26:03.512 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:26:03.514 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:26:03.514 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:26:03.515 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:26:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 msgr2=0x7ff520055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 0x7ff520055e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff52c00b000 tx=0x7ff52c006df0).stop 2022-07-20T02:26:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 msgr2=0x7ff53c0ac220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524014000).stop 2022-07-20T02:26:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 shutdown_connections 2022-07-20T02:26:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff53c0b0c50 0x7ff53c0ac220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff520053950 0x7ff520055e10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c143090 0x7ff53c094630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 --2- 172.21.15.26:0/4270658129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c0b0810 0x7ff53c0b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.792+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 >> 172.21.15.26:0/4270658129 conn(0x7ff53c0b5e70 msgr2=0x7ff53c1422e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.796+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 shutdown_connections 2022-07-20T02:26:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:02.796+0000 7ff5416a1700 1 -- 172.21.15.26:0/4270658129 wait complete. 2022-07-20T02:26:03.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:03.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 Processor -- start 2022-07-20T02:26:03.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 -- start start 2022-07-20T02:26:03.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45840b7780 0x7f45840b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4584065810 con 0x7f45840b3290 2022-07-20T02:26:03.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4584066db0 con 0x7f45840b7780 2022-07-20T02:26:03.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.652+0000 7f45893d8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4584066670 con 0x7f45840b2850 2022-07-20T02:26:03.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f4582ffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45840b7780 0x7f45840b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57478/0 (socket says 172.21.15.26:57478) 2022-07-20T02:26:03.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 -- 172.21.15.26:0/2970299147 learned_addr learned my addr 172.21.15.26:0/2970299147 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:03.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45837fe700 1 --1- 172.21.15.26:0/2970299147 >> v1:172.21.15.120:6789/0 conn(0x7f45840b3290 0x7f45841432d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49452/0 (socket says 172.21.15.26:49452) 2022-07-20T02:26:03.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 -- 172.21.15.26:0/2970299147 >> v2:172.21.15.120:3300/0 conn(0x7f45840b7780 msgr2=0x7f45840b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:03.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 --2- 172.21.15.26:0/2970299147 >> v2:172.21.15.120:3300/0 conn(0x7f45840b7780 0x7f45840b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f4581ffb700 1 -- 172.21.15.26:0/2970299147 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818387244 0 0) 0x7f4584065810 con 0x7f45840b3290 2022-07-20T02:26:03.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 -- 172.21.15.26:0/2970299147 >> v1:172.21.15.120:6789/0 conn(0x7f45840b3290 legacy=0x7f45841432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 -- 172.21.15.26:0/2970299147 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840ac3f0 con 0x7f45840b2850 2022-07-20T02:26:03.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45827fc700 1 --2- 172.21.15.26:0/2970299147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f45740032f0 tx=0x7f4574009290).ready entity=mon.2 client_cookie=8da12102e47a70b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:03.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f4581ffb700 1 -- 172.21.15.26:0/2970299147 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f457400b8a0 con 0x7f45840b2850 2022-07-20T02:26:03.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f4581ffb700 1 -- 172.21.15.26:0/2970299147 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400ba00 con 0x7f45840b2850 2022-07-20T02:26:03.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f4581ffb700 1 -- 172.21.15.26:0/2970299147 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f457400bb70 con 0x7f45840b2850 2022-07-20T02:26:03.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 -- 172.21.15.26:0/2970299147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 msgr2=0x7f4584066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 --2- 172.21.15.26:0/2970299147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f45740032f0 tx=0x7f4574009290).stop 2022-07-20T02:26:03.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 -- 172.21.15.26:0/2970299147 shutdown_connections 2022-07-20T02:26:03.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 --2- 172.21.15.26:0/2970299147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b2850 0x7f4584066180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 --2- 172.21.15.26:0/2970299147 >> v2:172.21.15.120:3300/0 conn(0x7f45840b7780 0x7f45840b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 -- 172.21.15.26:0/2970299147 >> 172.21.15.26:0/2970299147 conn(0x7f4584064b10 msgr2=0x7f4584064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:03.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 -- 172.21.15.26:0/2970299147 shutdown_connections 2022-07-20T02:26:03.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.656+0000 7f45893d8700 1 -- 172.21.15.26:0/2970299147 wait complete. 2022-07-20T02:26:03.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 Processor -- start 2022-07-20T02:26:03.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- start start 2022-07-20T02:26:03.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b3290 0x7f4584061170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45840b7780 0x7f458405fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584061660 con 0x7f45840b2850 2022-07-20T02:26:03.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45841532f0 con 0x7f45840b7780 2022-07-20T02:26:03.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840af6a0 con 0x7f45840b3290 2022-07-20T02:26:03.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35964/0 (socket says 172.21.15.26:35964) 2022-07-20T02:26:03.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 -- 172.21.15.26:0/4213676610 learned_addr learned my addr 172.21.15.26:0/4213676610 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45827fc700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b3290 0x7f4584061170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45837fe700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45840b7780 0x7f458405fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 -- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b3290 msgr2=0x7f4584061170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b3290 0x7f4584061170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 -- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45840b7780 msgr2=0x7f458405fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45840b7780 0x7f458405fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 -- 172.21.15.26:0/4213676610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574009530 con 0x7f45840b2850 2022-07-20T02:26:03.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4582ffd700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f457800c000 tx=0x7f4578006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:03.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4578010780 con 0x7f45840b2850 2022-07-20T02:26:03.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45780108e0 con 0x7f45840b2850 2022-07-20T02:26:03.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840af940 con 0x7f45840b2850 2022-07-20T02:26:03.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4578010c40 con 0x7f45840b2850 2022-07-20T02:26:03.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.660+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840aeab0 con 0x7f45840b2850 2022-07-20T02:26:03.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45840b2dd0 con 0x7f45840b2850 2022-07-20T02:26:03.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f457801c460 con 0x7f45840b2850 2022-07-20T02:26:03.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f4573fff700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 0x7f4560055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f457806c500 con 0x7f45840b2850 2022-07-20T02:26:03.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f45827fc700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 0x7f4560055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.664+0000 7f45827fc700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 0x7f4560055f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f457401b000 tx=0x7f4574016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:03.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.672+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4578043150 con 0x7f45840b2850 2022-07-20T02:26:03.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.836+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45840574e0 con 0x7f45840b2850 2022-07-20T02:26:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f4573fff700 1 -- 172.21.15.26:0/4213676610 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2489 (secure 0 0 0) 0x7f4578046770 con 0x7f45840b2850 2022-07-20T02:26:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:03.845 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:25:37.669039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/4183261198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4183261198},{"type":"v1","addr":"172.21.15.26:6837","nonce":4183261198}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:26:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 msgr2=0x7f4560055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 0x7f4560055f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f457401b000 tx=0x7f4574016000).stop 2022-07-20T02:26:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 msgr2=0x7f45840b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f457800c000 tx=0x7f4578006df0).stop 2022-07-20T02:26:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 shutdown_connections 2022-07-20T02:26:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4560053aa0 0x7f4560055f60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45840b7780 0x7f458405fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45840b3290 0x7f4584061170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 --2- 172.21.15.26:0/4213676610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45840b2850 0x7f45840b0e10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 >> 172.21.15.26:0/4213676610 conn(0x7f4584064b10 msgr2=0x7f458405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 shutdown_connections 2022-07-20T02:26:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.840+0000 7f45893d8700 1 -- 172.21.15.26:0/4213676610 wait complete. 2022-07-20T02:26:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2022-07-20T02:26:03.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:26:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.980+0000 7f6b6359e700 1 Processor -- start 2022-07-20T02:26:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.980+0000 7f6b6359e700 1 -- start start 2022-07-20T02:26:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640b1900 0x7f6b640afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6b640ac8e0 0x7f6b640b0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b64140190 con 0x7f6b640b5d50 2022-07-20T02:26:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640af740 con 0x7f6b640ac8e0 2022-07-20T02:26:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b64140430 con 0x7f6b640b1900 2022-07-20T02:26:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6b640ac8e0 0x7f6b640b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6b640ac8e0 0x7f6b640b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:55666/0 (socket says 172.21.15.26:55666) 2022-07-20T02:26:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 -- 172.21.15.26:0/1872876410 learned_addr learned my addr 172.21.15.26:0/1872876410 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b61d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640b1900 0x7f6b640afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6259c700 1 --1- 172.21.15.26:0/1872876410 >> v1:172.21.15.120:6789/0 conn(0x7f6b640b5d50 0x7f6b6413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49464/0 (socket says 172.21.15.26:49464) 2022-07-20T02:26:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 -- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640b1900 msgr2=0x7f6b640afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 --2- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640b1900 0x7f6b640afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 -- 172.21.15.26:0/1872876410 >> v1:172.21.15.120:6789/0 conn(0x7f6b640b5d50 legacy=0x7f6b6413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 -- 172.21.15.26:0/1872876410 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b641406d0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b62d9d700 1 --2- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640ac8e0 0x7f6b640b0330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6b5c0032f0 tx=0x7f6b5c00b530).ready entity=mon.1 client_cookie=83640668aa33a606 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6159a700 1 -- 172.21.15.26:0/1872876410 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b5c00de20 con 0x7f6b640ac8e0 2022-07-20T02:26:04.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6159a700 1 -- 172.21.15.26:0/1872876410 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c010470 con 0x7f6b640ac8e0 2022-07-20T02:26:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6159a700 1 -- 172.21.15.26:0/1872876410 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b5c0107d0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640ac8e0 msgr2=0x7f6b640b0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 --2- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640ac8e0 0x7f6b640b0330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6b5c0032f0 tx=0x7f6b5c00b530).stop 2022-07-20T02:26:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- 172.21.15.26:0/1872876410 shutdown_connections 2022-07-20T02:26:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 --2- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640ac8e0 0x7f6b640b0330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 --2- 172.21.15.26:0/1872876410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640b1900 0x7f6b640afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- 172.21.15.26:0/1872876410 >> 172.21.15.26:0/1872876410 conn(0x7f6b64060310 msgr2=0x7f6b640b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- 172.21.15.26:0/1872876410 shutdown_connections 2022-07-20T02:26:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.984+0000 7f6b6359e700 1 -- 172.21.15.26:0/1872876410 wait complete. 2022-07-20T02:26:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 Processor -- start 2022-07-20T02:26:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 -- start start 2022-07-20T02:26:04.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 0x7f6b64066360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640b5d50 0x7f6b640a8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b64067720 con 0x7f6b640b1900 2022-07-20T02:26:04.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b640679c0 con 0x7f6b640b5d50 2022-07-20T02:26:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b64067c60 con 0x7f6b640ac8e0 2022-07-20T02:26:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b61d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 0x7f6b64066360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b61d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35982/0 (socket says 172.21.15.26:35982) 2022-07-20T02:26:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b61d9b700 1 -- 172.21.15.26:0/3502658875 learned_addr learned my addr 172.21.15.26:0/3502658875 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b62d9d700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640b5d50 0x7f6b640a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 -- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640b5d50 msgr2=0x7f6b640a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640b5d50 0x7f6b640a8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 -- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 msgr2=0x7f6b6413ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 -- 172.21.15.26:0/3502658875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c00b7d0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b61d9b700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6259c700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 0x7f6b64066360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6b640b3040 tx=0x7f6b54007b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b5400ce60 con 0x7f6b640ac8e0 2022-07-20T02:26:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b540070d0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b64066990 con 0x7f6b640ac8e0 2022-07-20T02:26:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.988+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b54007430 con 0x7f6b640ac8e0 2022-07-20T02:26:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b640a38f0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b64067e20 con 0x7f6b640ac8e0 2022-07-20T02:26:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6b54010070 con 0x7f6b640ac8e0 2022-07-20T02:26:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b537fe700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 0x7f6b40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6b5406b430 con 0x7f6b640ac8e0 2022-07-20T02:26:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b61d9b700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 0x7f6b40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:03.992+0000 7f6b61d9b700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 0x7f6b40055f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6b5800b000 tx=0x7f6b58006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.000+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6b540421f0 con 0x7f6b640ac8e0 2022-07-20T02:26:04.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.164+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6b64140970 con 0x7f6b640ac8e0 2022-07-20T02:26:04.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:26:04.420+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:26:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.436+0000 7f6b537fe700 1 -- 172.21.15.26:0/3502658875 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6b54045810 con 0x7f6b640ac8e0 2022-07-20T02:26:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 msgr2=0x7f6b40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 0x7f6b40055f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6b5800b000 tx=0x7f6b58006df0).stop 2022-07-20T02:26:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 msgr2=0x7f6b64066360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 0x7f6b64066360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6b640b3040 tx=0x7f6b54007b80).stop 2022-07-20T02:26:04.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 shutdown_connections 2022-07-20T02:26:04.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b640b1900 0x7f6b6413ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6b40053a70 0x7f6b40055f30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b640b5d50 0x7f6b640a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 --2- 172.21.15.26:0/3502658875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b640ac8e0 0x7f6b64066360 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 >> 172.21.15.26:0/3502658875 conn(0x7f6b64060310 msgr2=0x7f6b640b3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:04.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 shutdown_connections 2022-07-20T02:26:04.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.440+0000 7f6b6359e700 1 -- 172.21.15.26:0/3502658875 wait complete. 2022-07-20T02:26:04.461 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:26:04.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:04.479 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:26:04.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 Processor -- start 2022-07-20T02:26:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 -- start start 2022-07-20T02:26:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f58b40b3260 0x7f58b4067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 0x7f58b4142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b405ff70 con 0x7f58b4061860 2022-07-20T02:26:04.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b40b3640 con 0x7f58b40b3260 2022-07-20T02:26:04.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b895d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b40601a0 con 0x7f58b40980d0 2022-07-20T02:26:04.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f58b40b3260 0x7f58b4067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f58b40b3260 0x7f58b4067900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35998/0 (socket says 172.21.15.26:35998) 2022-07-20T02:26:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 -- 172.21.15.26:0/4074925244 learned_addr learned my addr 172.21.15.26:0/4074925244 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b3fff700 1 --2- 172.21.15.26:0/4074925244 >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 0x7f58b4142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 -- 172.21.15.26:0/4074925244 >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 msgr2=0x7f58b4142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 --2- 172.21.15.26:0/4074925244 >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 0x7f58b4142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 -- 172.21.15.26:0/4074925244 >> v1:172.21.15.120:6789/0 conn(0x7f58b4061860 legacy=0x7f58b40979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 -- 172.21.15.26:0/4074925244 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b40606e0 con 0x7f58b40b3260 2022-07-20T02:26:04.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b3fff700 1 --2- 172.21.15.26:0/4074925244 >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 0x7f58b4142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:04.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b37fe700 1 --2- 172.21.15.26:0/4074925244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b4067900 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f58a4002f70 tx=0x7f58a4006eb0).ready entity=mon.0 client_cookie=d775be588873bc94 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b27fc700 1 -- 172.21.15.26:0/4074925244 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a400d610 con 0x7f58b40b3260 2022-07-20T02:26:04.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b27fc700 1 -- 172.21.15.26:0/4074925244 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4000e90 con 0x7f58b40b3260 2022-07-20T02:26:04.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.584+0000 7f58b27fc700 1 -- 172.21.15.26:0/4074925244 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a400d9a0 con 0x7f58b40b3260 2022-07-20T02:26:04.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- 172.21.15.26:0/4074925244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 msgr2=0x7f58b4067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 --2- 172.21.15.26:0/4074925244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b4067900 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f58a4002f70 tx=0x7f58a4006eb0).stop 2022-07-20T02:26:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- 172.21.15.26:0/4074925244 shutdown_connections 2022-07-20T02:26:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 --2- 172.21.15.26:0/4074925244 >> v2:172.21.15.120:3300/0 conn(0x7f58b40980d0 0x7f58b4142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 --2- 172.21.15.26:0/4074925244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b4067900 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- 172.21.15.26:0/4074925244 >> 172.21.15.26:0/4074925244 conn(0x7f58b40b67f0 msgr2=0x7f58b40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:04.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- 172.21.15.26:0/4074925244 shutdown_connections 2022-07-20T02:26:04.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- 172.21.15.26:0/4074925244 wait complete. 2022-07-20T02:26:04.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 Processor -- start 2022-07-20T02:26:04.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.588+0000 7f58b895d700 1 -- start start 2022-07-20T02:26:04.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58b4061860 0x7f58b4091820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58b40980d0 0x7f58b4153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4152af0 con 0x7f58b40b3260 2022-07-20T02:26:04.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4090370 con 0x7f58b4061860 2022-07-20T02:26:04.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4152d20 con 0x7f58b40980d0 2022-07-20T02:26:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36008/0 (socket says 172.21.15.26:36008) 2022-07-20T02:26:04.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 -- 172.21.15.26:0/3964320374 learned_addr learned my addr 172.21.15.26:0/3964320374 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:04.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b2ffd700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58b40980d0 0x7f58b4153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b37fe700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58b4061860 0x7f58b4091820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 -- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58b40980d0 msgr2=0x7f58b4153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58b40980d0 0x7f58b4153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 -- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58b4061860 msgr2=0x7f58b4091820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58b4061860 0x7f58b4091820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 -- 172.21.15.26:0/3964320374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a400b1e0 con 0x7f58b40b3260 2022-07-20T02:26:04.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b3fff700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f58ac006cc0 tx=0x7f58ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58ac009ad0 con 0x7f58b40b3260 2022-07-20T02:26:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac009c30 con 0x7f58b40b3260 2022-07-20T02:26:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b4152fc0 con 0x7f58b40b3260 2022-07-20T02:26:04.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58ac019410 con 0x7f58b40b3260 2022-07-20T02:26:04.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.592+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b414f970 con 0x7f58b40b3260 2022-07-20T02:26:04.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b4153150 con 0x7f58b40b3260 2022-07-20T02:26:04.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f58ac021070 con 0x7f58b40b3260 2022-07-20T02:26:04.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b0ff9700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 0x7f5894055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f58ac06e780 con 0x7f58b40b3260 2022-07-20T02:26:04.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b37fe700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 0x7f5894055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.596+0000 7f58b37fe700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 0x7f5894055e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f58a401c000 tx=0x7f58a4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:04.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.600+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58ac0453e0 con 0x7f58b40b3260 2022-07-20T02:26:04.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58b40574e0 con 0x7f58b40b3260 2022-07-20T02:26:04.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b0ff9700 1 -- 172.21.15.26:0/3964320374 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+2129 (secure 0 0 0) 0x7f58ac048a00 con 0x7f58b40b3260 2022-07-20T02:26:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:04.792 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1861780653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1861780653},{"type":"v1","addr":"172.21.15.120:6833","nonce":1861780653}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1169333043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1169333043},{"type":"v1","addr":"172.21.15.26:6835","nonce":1169333043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":8,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:25:35.946073+0000","modified":"2022-07-20T02:26:04.423591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T02:26:04.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 msgr2=0x7f5894055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 0x7f5894055e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f58a401c000 tx=0x7f58a4015000).stop 2022-07-20T02:26:04.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 msgr2=0x7f58b414f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.768+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f58ac006cc0 tx=0x7f58ac008040).stop 2022-07-20T02:26:04.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 shutdown_connections 2022-07-20T02:26:04.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58b40b3260 0x7f58b414f480 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5894053980 0x7f5894055e40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58b40980d0 0x7f58b4153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 --2- 172.21.15.26:0/3964320374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58b4061860 0x7f58b4091820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:04.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 >> 172.21.15.26:0/3964320374 conn(0x7f58b40b67f0 msgr2=0x7f58b40b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:04.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 shutdown_connections 2022-07-20T02:26:04.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.772+0000 7f58b895d700 1 -- 172.21.15.26:0/3964320374 wait complete. 2022-07-20T02:26:04.856 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 8 2022-07-20T02:26:04.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:26:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 Processor -- start 2022-07-20T02:26:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 -- start start 2022-07-20T02:26:04.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8700061a20 0x7f87001432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:04.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8700061e00 con 0x7f870012c750 2022-07-20T02:26:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8700147040 con 0x7f87001468c0 2022-07-20T02:26:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f87057c3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87001472e0 con 0x7f8700061a20 2022-07-20T02:26:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f86ff7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f870012c750 0x7f87000b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35296/0 (socket says 172.21.15.26:35296) 2022-07-20T02:26:04.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f86ff7fe700 1 -- 172.21.15.26:0/1947220057 learned_addr learned my addr 172.21.15.26:0/1947220057 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f86feffd700 1 --2- 172.21.15.26:0/1947220057 >> v2:172.21.15.120:3300/0 conn(0x7f8700061a20 0x7f87001432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.980+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602331500 0 0) 0x7f8700061e00 con 0x7f870012c750 2022-07-20T02:26:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e0003260 con 0x7f870012c750 2022-07-20T02:26:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887601599 0 0) 0x7f8700147040 con 0x7f87001468c0 2022-07-20T02:26:04.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8700061e00 con 0x7f87001468c0 2022-07-20T02:26:04.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86feffd700 1 -- 172.21.15.26:0/1947220057 >> v1:172.21.15.120:6789/0 conn(0x7f87001468c0 legacy=0x7f870012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86feffd700 1 -- 172.21.15.26:0/1947220057 >> v1:172.21.15.26:6789/0 conn(0x7f870012c750 legacy=0x7f87000b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:04.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030972880 0 0) 0x7f86e0003260 con 0x7f870012c750 2022-07-20T02:26:04.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86feffd700 1 -- 172.21.15.26:0/1947220057 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8700147580 con 0x7f8700061a20 2022-07-20T02:26:05.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86feffd700 1 --2- 172.21.15.26:0/1947220057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8700061a20 0x7f87001432e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f86f8001490 tx=0x7f86f0006df0).ready entity=mon.1 client_cookie=e0217ac2eb82bbc2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f80019e0 con 0x7f8700061a20 2022-07-20T02:26:05.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0000e90 con 0x7f8700061a20 2022-07-20T02:26:05.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f86fdffb700 1 -- 172.21.15.26:0/1947220057 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f0012460 con 0x7f8700061a20 2022-07-20T02:26:05.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 -- 172.21.15.26:0/1947220057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8700061a20 msgr2=0x7f87001432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 --2- 172.21.15.26:0/1947220057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8700061a20 0x7f87001432e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f86f8001490 tx=0x7f86f0006df0).stop 2022-07-20T02:26:05.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 -- 172.21.15.26:0/1947220057 shutdown_connections 2022-07-20T02:26:05.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 --2- 172.21.15.26:0/1947220057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8700061a20 0x7f87001432e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 -- 172.21.15.26:0/1947220057 >> 172.21.15.26:0/1947220057 conn(0x7f870005faf0 msgr2=0x7f8700066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:05.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 -- 172.21.15.26:0/1947220057 shutdown_connections 2022-07-20T02:26:05.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.984+0000 7f87057c3700 1 -- 172.21.15.26:0/1947220057 wait complete. 2022-07-20T02:26:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 Processor -- start 2022-07-20T02:26:05.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- start start 2022-07-20T02:26:05.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8700061a20 0x7f870012b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 0x7f87000b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87000b7b80 con 0x7f87001468c0 2022-07-20T02:26:05.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87000b7dd0 con 0x7f870012c750 2022-07-20T02:26:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87001526b0 con 0x7f8700061a20 2022-07-20T02:26:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86ff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86ff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36022/0 (socket says 172.21.15.26:36022) 2022-07-20T02:26:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 0x7f87000b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86feffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8700061a20 0x7f870012b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86ff7fe700 1 -- 172.21.15.26:0/2716306361 learned_addr learned my addr 172.21.15.26:0/2716306361 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 -- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8700061a20 msgr2=0x7f870012b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8700061a20 0x7f870012b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 -- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 msgr2=0x7f8700152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 -- 172.21.15.26:0/2716306361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f000b180 con 0x7f870012c750 2022-07-20T02:26:05.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86ff7fe700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:05.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86fe7fc700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 0x7f87000b7690 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f86e8001110 tx=0x7f86e8007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86e800ce90 con 0x7f870012c750 2022-07-20T02:26:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8012400 con 0x7f870012c750 2022-07-20T02:26:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8700152950 con 0x7f870012c750 2022-07-20T02:26:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86e8012760 con 0x7f870012c750 2022-07-20T02:26:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.988+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87001523c0 con 0x7f870012c750 2022-07-20T02:26:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8700152ae0 con 0x7f870012c750 2022-07-20T02:26:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f86e8019030 con 0x7f870012c750 2022-07-20T02:26:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f86f7fff700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 0x7f86e4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f86e806b400 con 0x7f870012c750 2022-07-20T02:26:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f86feffd700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 0x7f86e4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:04.992+0000 7f86feffd700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 0x7f86e4055e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f86f001a000 tx=0x7f86f0014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.000+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86e80420c0 con 0x7f870012c750 2022-07-20T02:26:05.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.172+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f870012cbc0 con 0x7f870012c750 2022-07-20T02:26:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.444+0000 7f86f7fff700 1 -- 172.21.15.26:0/2716306361 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f86e80456e0 con 0x7f870012c750 2022-07-20T02:26:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.444+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 msgr2=0x7f86e4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.444+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 0x7f86e4055e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f86f001a000 tx=0x7f86f0014000).stop 2022-07-20T02:26:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.444+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 msgr2=0x7f87000b7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.444+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 0x7f87000b7690 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f86e8001110 tx=0x7f86e8007ad0).stop 2022-07-20T02:26:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 shutdown_connections 2022-07-20T02:26:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87001468c0 0x7f8700152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f86e4053950 0x7f86e4055e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f870012c750 0x7f87000b7690 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 --2- 172.21.15.26:0/2716306361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8700061a20 0x7f870012b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 >> 172.21.15.26:0/2716306361 conn(0x7f870005faf0 msgr2=0x7f87000b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:05.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 shutdown_connections 2022-07-20T02:26:05.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.448+0000 7f87057c3700 1 -- 172.21.15.26:0/2716306361 wait complete. 2022-07-20T02:26:05.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:26:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 Processor -- start 2022-07-20T02:26:05.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 -- start start 2022-07-20T02:26:05.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1601432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb16012c750 0x7fb1600b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb160061e00 con 0x7fb1601468c0 2022-07-20T02:26:05.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb160147040 con 0x7fb16012c750 2022-07-20T02:26:05.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb164807700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1601472e0 con 0x7fb160061a20 2022-07-20T02:26:05.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15ed9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb16012c750 0x7fb1600b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15dd9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1601468c0 0x7fb16012c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35308/0 (socket says 172.21.15.26:35308) 2022-07-20T02:26:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15dd9b700 1 -- 172.21.15.26:0/3326965219 learned_addr learned my addr 172.21.15.26:0/3326965219 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15e59c700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1601432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15ed9d700 1 -- 172.21.15.26:0/3326965219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 msgr2=0x7fb1601432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15ed9d700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1601432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15ed9d700 1 -- 172.21.15.26:0/3326965219 >> v1:172.21.15.26:6789/0 conn(0x7fb1601468c0 legacy=0x7fb16012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb15ed9d700 1 -- 172.21.15.26:0/3326965219 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160147580 con 0x7fb16012c750 2022-07-20T02:26:05.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.600+0000 7fb15d59a700 1 -- 172.21.15.26:0/3326965219 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257857291 0 0) 0x7fb160061e00 con 0x7fb1601468c0 2022-07-20T02:26:05.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb15ed9d700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb16012c750 0x7fb1600b2720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb1580032f0 tx=0x7fb1580091d0).ready entity=mon.0 client_cookie=a843f011677f1359 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb15d59a700 1 -- 172.21.15.26:0/3326965219 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15800b6c0 con 0x7fb16012c750 2022-07-20T02:26:05.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb15d59a700 1 -- 172.21.15.26:0/3326965219 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15800b820 con 0x7fb16012c750 2022-07-20T02:26:05.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb15d59a700 1 -- 172.21.15.26:0/3326965219 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15800b990 con 0x7fb16012c750 2022-07-20T02:26:05.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- 172.21.15.26:0/3326965219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb16012c750 msgr2=0x7fb1600b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb16012c750 0x7fb1600b2720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb1580032f0 tx=0x7fb1580091d0).stop 2022-07-20T02:26:05.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- 172.21.15.26:0/3326965219 shutdown_connections 2022-07-20T02:26:05.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb16012c750 0x7fb1600b2720 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 --2- 172.21.15.26:0/3326965219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1601432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- 172.21.15.26:0/3326965219 >> 172.21.15.26:0/3326965219 conn(0x7fb16005faf0 msgr2=0x7fb160066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:05.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- 172.21.15.26:0/3326965219 shutdown_connections 2022-07-20T02:26:05.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- 172.21.15.26:0/3326965219 wait complete. 2022-07-20T02:26:05.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 Processor -- start 2022-07-20T02:26:05.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.604+0000 7fb164807700 1 -- start start 2022-07-20T02:26:05.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1600b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 0x7fb160152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600b8e80 con 0x7fb1601468c0 2022-07-20T02:26:05.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600adc80 con 0x7fb16012c750 2022-07-20T02:26:05.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb160153290 con 0x7fb160061a20 2022-07-20T02:26:05.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36036/0 (socket says 172.21.15.26:36036) 2022-07-20T02:26:05.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 -- 172.21.15.26:0/2602198275 learned_addr learned my addr 172.21.15.26:0/2602198275 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:05.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15dd9b700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 0x7fb160152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15e59c700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1600b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 -- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 msgr2=0x7fb1600b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1600b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 -- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 msgr2=0x7fb160152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:05.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 0x7fb160152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:05.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 -- 172.21.15.26:0/2602198275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158009470 con 0x7fb1601468c0 2022-07-20T02:26:05.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15dd9b700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 0x7fb160152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:05.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb15ed9d700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fb1600ba7f0 tx=0x7fb158016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15800b6c0 con 0x7fb1601468c0 2022-07-20T02:26:05.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb158021630 con 0x7fb1601468c0 2022-07-20T02:26:05.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1600b77b0 con 0x7fb1601468c0 2022-07-20T02:26:05.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb158021990 con 0x7fb1601468c0 2022-07-20T02:26:05.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1600b8370 con 0x7fb1601468c0 2022-07-20T02:26:05.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.608+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16012cbc0 con 0x7fb1601468c0 2022-07-20T02:26:05.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.612+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb158027020 con 0x7fb1601468c0 2022-07-20T02:26:05.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.612+0000 7fb1577fe700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 0x7fb144055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:05.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.612+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb15806d490 con 0x7fb1601468c0 2022-07-20T02:26:05.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.612+0000 7fb15e59c700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 0x7fb144055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:05.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.612+0000 7fb15e59c700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 0x7fb144055f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb15000b000 tx=0x7fb150006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:05.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.620+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1580440e0 con 0x7fb1601468c0 2022-07-20T02:26:05.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:05.744+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb1600574e0 con 0x7fb1601468c0 2022-07-20T02:26:06.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.448+0000 7fb1577fe700 1 -- 172.21.15.26:0/2602198275 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb1580150e0 con 0x7fb1601468c0 2022-07-20T02:26:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 msgr2=0x7fb144055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 0x7fb144055f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb15000b000 tx=0x7fb150006df0).stop 2022-07-20T02:26:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 msgr2=0x7fb1600b9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fb1600ba7f0 tx=0x7fb158016000).stop 2022-07-20T02:26:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 shutdown_connections 2022-07-20T02:26:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1601468c0 0x7fb1600b9270 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb144053a70 0x7fb144055f30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb16012c750 0x7fb160152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 --2- 172.21.15.26:0/2602198275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb160061a20 0x7fb1600b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 >> 172.21.15.26:0/2602198275 conn(0x7fb16005faf0 msgr2=0x7fb1600b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 shutdown_connections 2022-07-20T02:26:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.452+0000 7fb164807700 1 -- 172.21.15.26:0/2602198275 wait complete. 2022-07-20T02:26:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:26:06.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:26:06.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.584+0000 7f6b0cd42700 1 Processor -- start 2022-07-20T02:26:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 -- start start 2022-07-20T02:26:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6b08061a20 0x7f6b081432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08061e00 con 0x7f6b081468c0 2022-07-20T02:26:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08147040 con 0x7f6b0812c750 2022-07-20T02:26:06.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b081472e0 con 0x7f6b08061a20 2022-07-20T02:26:06.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6b08061a20 0x7f6b081432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:06.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6afffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6b081468c0 0x7f6b0812c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35322/0 (socket says 172.21.15.26:35322) 2022-07-20T02:26:06.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6afffff700 1 -- 172.21.15.26:0/3702504361 learned_addr learned my addr 172.21.15.26:0/3702504361 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:06.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b05d9b700 1 -- 172.21.15.26:0/3702504361 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475824479 0 0) 0x7f6b08061e00 con 0x7f6b081468c0 2022-07-20T02:26:06.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b05d9b700 1 -- 172.21.15.26:0/3702504361 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8003260 con 0x7f6b081468c0 2022-07-20T02:26:06.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0659c700 1 -- 172.21.15.26:0/3702504361 >> v1:172.21.15.120:6789/0 conn(0x7f6b0812c750 legacy=0x7f6b080b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0659c700 1 -- 172.21.15.26:0/3702504361 >> v1:172.21.15.26:6789/0 conn(0x7f6b081468c0 legacy=0x7f6b0812c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0659c700 1 -- 172.21.15.26:0/3702504361 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08147580 con 0x7f6b08061a20 2022-07-20T02:26:06.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0659c700 1 --2- 172.21.15.26:0/3702504361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b08061a20 0x7f6b081432e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6af8002f70 tx=0x7f6af8006df0).ready entity=mon.0 client_cookie=e596a3b4f354aef5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:06.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b05d9b700 1 -- 172.21.15.26:0/3702504361 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af800d610 con 0x7f6b08061a20 2022-07-20T02:26:06.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b05d9b700 1 -- 172.21.15.26:0/3702504361 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8000e90 con 0x7f6b08061a20 2022-07-20T02:26:06.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b05d9b700 1 -- 172.21.15.26:0/3702504361 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af800d960 con 0x7f6b08061a20 2022-07-20T02:26:06.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 -- 172.21.15.26:0/3702504361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b08061a20 msgr2=0x7f6b081432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.588+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/3702504361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b08061a20 0x7f6b081432e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6af8002f70 tx=0x7f6af8006df0).stop 2022-07-20T02:26:06.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- 172.21.15.26:0/3702504361 shutdown_connections 2022-07-20T02:26:06.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/3702504361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b08061a20 0x7f6b081432e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- 172.21.15.26:0/3702504361 >> 172.21.15.26:0/3702504361 conn(0x7f6b0805faf0 msgr2=0x7f6b08066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:06.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- 172.21.15.26:0/3702504361 shutdown_connections 2022-07-20T02:26:06.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- 172.21.15.26:0/3702504361 wait complete. 2022-07-20T02:26:06.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 Processor -- start 2022-07-20T02:26:06.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- start start 2022-07-20T02:26:06.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b08061a20 0x7f6b080b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 0x7f6b08144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:06.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b0380 con 0x7f6b0812c750 2022-07-20T02:26:06.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080adbe0 con 0x7f6b081468c0 2022-07-20T02:26:06.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.592+0000 7f6b0cd42700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08144640 con 0x7f6b08061a20 2022-07-20T02:26:06.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:06.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b0659c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b08061a20 0x7f6b080b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:06.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b06d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 0x7f6b08144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36048/0 (socket says 172.21.15.26:36048) 2022-07-20T02:26:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 -- 172.21.15.26:0/93299802 learned_addr learned my addr 172.21.15.26:0/93299802 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 -- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b08061a20 msgr2=0x7f6b080b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b08061a20 0x7f6b080b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 -- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 msgr2=0x7f6b08144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 0x7f6b08144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:06.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 -- 172.21.15.26:0/93299802 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af800b180 con 0x7f6b0812c750 2022-07-20T02:26:06.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b06d9d700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 0x7f6b08144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:06.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6afffff700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f6af0001110 tx=0x7f6af0007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:06.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af00070d0 con 0x7f6b0812c750 2022-07-20T02:26:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0007230 con 0x7f6b0812c750 2022-07-20T02:26:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081448e0 con 0x7f6b0812c750 2022-07-20T02:26:06.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af0005610 con 0x7f6b0812c750 2022-07-20T02:26:06.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b080afa50 con 0x7f6b0812c750 2022-07-20T02:26:06.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0812cbc0 con 0x7f6b0812c750 2022-07-20T02:26:06.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.596+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6af001a060 con 0x7f6b0812c750 2022-07-20T02:26:06.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.600+0000 7f6aff7fe700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 0x7f6aec055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:06.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.600+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f6af006b330 con 0x7f6b0812c750 2022-07-20T02:26:06.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.600+0000 7f6b0659c700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 0x7f6aec055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:06.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.600+0000 7f6b0659c700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 0x7f6aec055f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6af801a000 tx=0x7f6af8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:06.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.604+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6af0042140 con 0x7f6b0812c750 2022-07-20T02:26:06.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:06.736+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6b080574e0 con 0x7f6b0812c750 2022-07-20T02:26:07.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.460+0000 7f6aff7fe700 1 -- 172.21.15.26:0/93299802 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6af0018020 con 0x7f6b0812c750 2022-07-20T02:26:07.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 msgr2=0x7f6aec055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 0x7f6aec055f30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6af801a000 tx=0x7f6af8014000).stop 2022-07-20T02:26:07.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 msgr2=0x7f6b081459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f6af0001110 tx=0x7f6af0007b80).stop 2022-07-20T02:26:07.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 shutdown_connections 2022-07-20T02:26:07.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b0812c750 0x7f6b081459c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6aec053a70 0x7f6aec055f30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b081468c0 0x7f6b08144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 --2- 172.21.15.26:0/93299802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b08061a20 0x7f6b080b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 >> 172.21.15.26:0/93299802 conn(0x7f6b0805faf0 msgr2=0x7f6b0812a810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:07.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 shutdown_connections 2022-07-20T02:26:07.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.464+0000 7f6b0cd42700 1 -- 172.21.15.26:0/93299802 wait complete. 2022-07-20T02:26:07.473 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:26:07.482 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:26:07.483 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:26:07.484 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:26:07.484 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:26:07.484 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:26:07.485 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:26:07.485 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:26:07.485 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:26:07.486 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:26:07.486 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:26:07.486 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:26:07.487 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:26:07.495 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:26:07.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:26:07.499 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:26:07.500 DEBUG:teuthology.parallel:result is None 2022-07-20T02:26:07.501 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:26:07.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:26:07.505 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:26:07.506 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:26:07.516 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:26:07.517 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:26:07.518 DEBUG:teuthology.parallel:result is None 2022-07-20T02:26:07.519 DEBUG:teuthology.parallel:result is None 2022-07-20T02:26:07.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:26:07.570 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:26:07.585 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:26:07.591 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:26:07.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 Processor -- start 2022-07-20T02:26:07.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- start start 2022-07-20T02:26:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1640b5d50 0x7fe16413d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe1640b1900 0x7fe16413d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe164140190 con 0x7fe1640ac8e0 2022-07-20T02:26:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1640af740 con 0x7fe1640b5d50 2022-07-20T02:26:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe164140430 con 0x7fe1640b1900 2022-07-20T02:26:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1640b5d50 0x7fe16413d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe162d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1640ac8e0 0x7fe1640afe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35358/0 (socket says 172.21.15.26:35358) 2022-07-20T02:26:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe162d9d700 1 -- 172.21.15.26:0/1167549863 learned_addr learned my addr 172.21.15.26:0/1167549863 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:07.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe15bfff700 1 --2- 172.21.15.26:0/1167549863 >> v2:172.21.15.120:3300/0 conn(0x7fe1640b1900 0x7fe16413d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe161d9b700 1 -- 172.21.15.26:0/1167549863 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058564874 0 0) 0x7fe164140190 con 0x7fe1640ac8e0 2022-07-20T02:26:07.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe161d9b700 1 -- 172.21.15.26:0/1167549863 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003260 con 0x7fe1640ac8e0 2022-07-20T02:26:07.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 -- 172.21.15.26:0/1167549863 >> v2:172.21.15.120:3300/0 conn(0x7fe1640b1900 msgr2=0x7fe16413d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:07.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 --2- 172.21.15.26:0/1167549863 >> v2:172.21.15.120:3300/0 conn(0x7fe1640b1900 0x7fe16413d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 -- 172.21.15.26:0/1167549863 >> v1:172.21.15.26:6789/0 conn(0x7fe1640ac8e0 legacy=0x7fe1640afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 -- 172.21.15.26:0/1167549863 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1641406d0 con 0x7fe1640b5d50 2022-07-20T02:26:07.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16259c700 1 --2- 172.21.15.26:0/1167549863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b5d50 0x7fe16413d2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe14c002f70 tx=0x7fe14c006df0).ready entity=mon.0 client_cookie=ed2b8aba56974ed3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:07.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe161d9b700 1 -- 172.21.15.26:0/1167549863 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe14c00d4f0 con 0x7fe1640b5d50 2022-07-20T02:26:07.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe161d9b700 1 -- 172.21.15.26:0/1167549863 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c000e90 con 0x7fe1640b5d50 2022-07-20T02:26:07.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe161d9b700 1 -- 172.21.15.26:0/1167549863 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe14c00d840 con 0x7fe1640b5d50 2022-07-20T02:26:07.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- 172.21.15.26:0/1167549863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b5d50 msgr2=0x7fe16413d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 --2- 172.21.15.26:0/1167549863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b5d50 0x7fe16413d2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe14c002f70 tx=0x7fe14c006df0).stop 2022-07-20T02:26:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- 172.21.15.26:0/1167549863 shutdown_connections 2022-07-20T02:26:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 --2- 172.21.15.26:0/1167549863 >> v2:172.21.15.120:3300/0 conn(0x7fe1640b1900 0x7fe16413d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 --2- 172.21.15.26:0/1167549863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b5d50 0x7fe16413d2f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- 172.21.15.26:0/1167549863 >> 172.21.15.26:0/1167549863 conn(0x7fe164060310 msgr2=0x7fe1640b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- 172.21.15.26:0/1167549863 shutdown_connections 2022-07-20T02:26:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.644+0000 7fe16359e700 1 -- 172.21.15.26:0/1167549863 wait complete. 2022-07-20T02:26:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 Processor -- start 2022-07-20T02:26:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- start start 2022-07-20T02:26:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1640ac8e0 0x7fe1640659b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 0x7fe164062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164066a60 con 0x7fe1640b1900 2022-07-20T02:26:07.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164063be0 con 0x7fe1640ac8e0 2022-07-20T02:26:07.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe164066280 con 0x7fe1640b5d50 2022-07-20T02:26:07.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36084/0 (socket says 172.21.15.26:36084) 2022-07-20T02:26:07.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 -- 172.21.15.26:0/1883879967 learned_addr learned my addr 172.21.15.26:0/1883879967 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:07.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16259c700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1640ac8e0 0x7fe1640659b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe162d9d700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 0x7fe164062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 -- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 msgr2=0x7fe164062a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 0x7fe164062a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 -- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1640ac8e0 msgr2=0x7fe1640659b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:07.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1640ac8e0 0x7fe1640659b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:07.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 -- 172.21.15.26:0/1883879967 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c00b180 con 0x7fe1640b1900 2022-07-20T02:26:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe162d9d700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 0x7fe164062a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15bfff700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe15400c000 tx=0x7fe154006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe154010780 con 0x7fe1640b1900 2022-07-20T02:26:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1540108e0 con 0x7fe1640b1900 2022-07-20T02:26:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe164066520 con 0x7fe1640b1900 2022-07-20T02:26:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe154010c40 con 0x7fe1640b1900 2022-07-20T02:26:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe164145640 con 0x7fe1640b1900 2022-07-20T02:26:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe164140970 con 0x7fe1640b1900 2022-07-20T02:26:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe15401c460 con 0x7fe1640b1900 2022-07-20T02:26:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 0x7fe13c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe15406c230 con 0x7fe1640b1900 2022-07-20T02:26:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.648+0000 7fe16259c700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 0x7fe13c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.652+0000 7fe16259c700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 0x7fe13c055e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe14c01a000 tx=0x7fe14c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.652+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1540431d0 con 0x7fe1640b1900 2022-07-20T02:26:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:07.788+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fe1640574e0 con 0x7fe1640b1900 2022-07-20T02:26:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.612+0000 7fe15b7fe700 1 -- 172.21.15.26:0/1883879967 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe1540467f0 con 0x7fe1640b1900 2022-07-20T02:26:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 msgr2=0x7fe13c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 0x7fe13c055e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe14c01a000 tx=0x7fe14c014000).stop 2022-07-20T02:26:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 msgr2=0x7fe164143b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe15400c000 tx=0x7fe154006df0).stop 2022-07-20T02:26:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 shutdown_connections 2022-07-20T02:26:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1640b1900 0x7fe164143b60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053950 0x7fe13c055e10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1640b5d50 0x7fe164062a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 --2- 172.21.15.26:0/1883879967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1640ac8e0 0x7fe1640659b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 >> 172.21.15.26:0/1883879967 conn(0x7fe164060310 msgr2=0x7fe16405cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:08.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 shutdown_connections 2022-07-20T02:26:08.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.616+0000 7fe16359e700 1 -- 172.21.15.26:0/1883879967 wait complete. 2022-07-20T02:26:08.648 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:26:08.650 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:26:08.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:26:08.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 Processor -- start 2022-07-20T02:26:08.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 -- start start 2022-07-20T02:26:08.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0080b7d80 0x7fd0080aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd00814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd008064650 con 0x7fd0080608d0 2022-07-20T02:26:08.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0080b4fc0 con 0x7fd0080b7d80 2022-07-20T02:26:08.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00d4d4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080b41a0 con 0x7fd0080b4b80 2022-07-20T02:26:08.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00ccd3700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd0080608d0 0x7fd008060d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35378/0 (socket says 172.21.15.26:35378) 2022-07-20T02:26:08.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd00ccd3700 1 -- 172.21.15.26:0/2782209520 learned_addr learned my addr 172.21.15.26:0/2782209520 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:08.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd007fff700 1 --2- 172.21.15.26:0/2782209520 >> v2:172.21.15.120:3300/0 conn(0x7fd0080b7d80 0x7fd0080aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 --2- 172.21.15.26:0/2782209520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd00814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd006ffd700 1 -- 172.21.15.26:0/2782209520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715541132 0 0) 0x7fd008064650 con 0x7fd0080608d0 2022-07-20T02:26:08.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd006ffd700 1 -- 172.21.15.26:0/2782209520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff4003260 con 0x7fd0080608d0 2022-07-20T02:26:08.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 -- 172.21.15.26:0/2782209520 >> v2:172.21.15.120:3300/0 conn(0x7fd0080b7d80 msgr2=0x7fd0080aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:08.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 --2- 172.21.15.26:0/2782209520 >> v2:172.21.15.120:3300/0 conn(0x7fd0080b7d80 0x7fd0080aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 -- 172.21.15.26:0/2782209520 >> v1:172.21.15.26:6789/0 conn(0x7fd0080608d0 legacy=0x7fd008060d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 -- 172.21.15.26:0/2782209520 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0080b4440 con 0x7fd0080b4b80 2022-07-20T02:26:08.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd0077fe700 1 --2- 172.21.15.26:0/2782209520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd00814ad60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcff8002f70 tx=0x7fcff8006df0).ready entity=mon.2 client_cookie=6c10d9cf5c55db93 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:08.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd006ffd700 1 -- 172.21.15.26:0/2782209520 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcff800d4a0 con 0x7fd0080b4b80 2022-07-20T02:26:08.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd006ffd700 1 -- 172.21.15.26:0/2782209520 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff8000e90 con 0x7fd0080b4b80 2022-07-20T02:26:08.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.808+0000 7fd006ffd700 1 -- 172.21.15.26:0/2782209520 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcff800d7f0 con 0x7fd0080b4b80 2022-07-20T02:26:08.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/2782209520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 msgr2=0x7fd00814ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/2782209520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd00814ad60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcff8002f70 tx=0x7fcff8006df0).stop 2022-07-20T02:26:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/2782209520 shutdown_connections 2022-07-20T02:26:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/2782209520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd00814ad60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/2782209520 >> v2:172.21.15.120:3300/0 conn(0x7fd0080b7d80 0x7fd0080aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/2782209520 >> 172.21.15.26:0/2782209520 conn(0x7fd00805f840 msgr2=0x7fd00805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/2782209520 shutdown_connections 2022-07-20T02:26:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/2782209520 wait complete. 2022-07-20T02:26:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 Processor -- start 2022-07-20T02:26:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- start start 2022-07-20T02:26:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd008159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0080b7d80 0x7fd00815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00814b120 con 0x7fd0080608d0 2022-07-20T02:26:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080ab6d0 con 0x7fd0080b7d80 2022-07-20T02:26:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080ab970 con 0x7fd0080b4b80 2022-07-20T02:26:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36100/0 (socket says 172.21.15.26:36100) 2022-07-20T02:26:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 -- 172.21.15.26:0/25335498 learned_addr learned my addr 172.21.15.26:0/25335498 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd0077fe700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd008159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00ccd3700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0080b7d80 0x7fd00815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 -- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 msgr2=0x7fd008159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd008159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 -- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0080b7d80 msgr2=0x7fd00815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0080b7d80 0x7fd00815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 -- 172.21.15.26:0/25335498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff800b180 con 0x7fd0080608d0 2022-07-20T02:26:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd0077fe700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd008159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd007fff700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fcffc00c000 tx=0x7fcffc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcffc010780 con 0x7fd0080608d0 2022-07-20T02:26:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc0108e0 con 0x7fd0080608d0 2022-07-20T02:26:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd008064e00 con 0x7fd0080608d0 2022-07-20T02:26:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcffc010c40 con 0x7fd0080608d0 2022-07-20T02:26:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.812+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00814ae50 con 0x7fd0080608d0 2022-07-20T02:26:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080b46e0 con 0x7fd0080608d0 2022-07-20T02:26:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcffc01c460 con 0x7fd0080608d0 2022-07-20T02:26:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd0057fa700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 0x7fcfe8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fcffc06c100 con 0x7fd0080608d0 2022-07-20T02:26:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd0077fe700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 0x7fcfe8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.816+0000 7fd0077fe700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 0x7fcfe8055fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcff8018000 tx=0x7fcff8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:08.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.824+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcffc043100 con 0x7fd0080608d0 2022-07-20T02:26:09.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.992+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd00814ae50 con 0x7fd0080608d0 2022-07-20T02:26:09.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.992+0000 7fd0057fa700 1 -- 172.21.15.26:0/25335498 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+5621 (secure 0 0 0) 0x7fd00814ae50 con 0x7fd0080608d0 2022-07-20T02:26:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:09.008 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:26:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:26:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:26:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:26:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:26:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.026 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:26:09.027 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:26:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:26:09.028 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:26:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:26:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.030 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:26:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:26:09.032 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:26:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:26:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:26:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:26:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:26:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:26:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.041 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.042 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:26:09.044 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:26:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.045 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.046 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.049 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:26:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:26:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.050 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.051 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.053 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:26:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:26:09.054 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:26:09.058 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:26:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.059 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.061 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:26:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:26:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:26:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow r", 2022-07-20T02:26:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow rw", 2022-07-20T02:26:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rwx" 2022-07-20T02:26:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:26:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:26:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:26:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow r", 2022-07-20T02:26:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow rw", 2022-07-20T02:26:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rwx" 2022-07-20T02:26:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:26:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:26:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:26:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:26:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:26:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-mds", 2022-07-20T02:26:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9disSORHxAAC7gbdZOfI8FNXDYx8JHk4w==", 2022-07-20T02:26:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-mds" 2022-07-20T02:26:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-mgr", 2022-07-20T02:26:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9dipDWRHxAAXOnpG9WtRSgSQp7G/43VJw==", 2022-07-20T02:26:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-mgr" 2022-07-20T02:26:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-osd", 2022-07-20T02:26:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9diXEWRHxAAZ0Z6Gs1CQkwiOzO9nPAkRA==", 2022-07-20T02:26:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-osd" 2022-07-20T02:26:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-rbd", 2022-07-20T02:26:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9diClWRHxAAMsv0ZOT8S1nmgbdP18Oarw==", 2022-07-20T02:26:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-rbd" 2022-07-20T02:26:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-rbd-mirror", 2022-07-20T02:26:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9diHmaRHxAA5OTM7OfeQ7ukWn1xy8Ew1w==", 2022-07-20T02:26:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2022-07-20T02:26:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.bootstrap-rgw", 2022-07-20T02:26:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCMZ9diF3qRHxAAXiOtjZ3WSMpOsbjmWaQXkw==", 2022-07-20T02:26:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile bootstrap-rgw" 2022-07-20T02:26:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:26:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:26:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:26:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:26:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:26:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:26:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:26:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:26:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:26:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:26:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:26:09.210 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:26:09.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 msgr2=0x7fcfe8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 0x7fcfe8055fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcff8018000 tx=0x7fcff8014000).stop 2022-07-20T02:26:09.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 msgr2=0x7fd008149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fcffc00c000 tx=0x7fcffc006df0).stop 2022-07-20T02:26:09.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 shutdown_connections 2022-07-20T02:26:09.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcfe8053b00 0x7fcfe8055fc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0080b7d80 0x7fd00815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0080b4b80 0x7fd008159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 --2- 172.21.15.26:0/25335498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0080608d0 0x7fd008149970 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 >> 172.21.15.26:0/25335498 conn(0x7fd00805f840 msgr2=0x7fd0080656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:09.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 shutdown_connections 2022-07-20T02:26:09.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:08.996+0000 7fd00d4d4700 1 -- 172.21.15.26:0/25335498 wait complete. 2022-07-20T02:26:09.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2022-07-20T02:26:09.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 Processor -- start 2022-07-20T02:26:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 -- start start 2022-07-20T02:26:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fabd005cfd0 0x7fabd0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fabd0147000 0x7fabd00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd01474e0 con 0x7fabd005cfd0 2022-07-20T02:26:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd005d3b0 con 0x7fabd0147000 2022-07-20T02:26:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd6bf6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01431e0 con 0x7fabd0060f00 2022-07-20T02:26:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd5bf4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fabd005cfd0 0x7fabd0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd5bf4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fabd005cfd0 0x7fabd0143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36108/0 (socket says 172.21.15.26:36108) 2022-07-20T02:26:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd5bf4700 1 -- 172.21.15.26:0/1506156020 learned_addr learned my addr 172.21.15.26:0/1506156020 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd63f5700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.344+0000 7fabd53f3700 1 --2- 172.21.15.26:0/1506156020 >> v2:172.21.15.120:3300/0 conn(0x7fabd0147000 0x7fabd00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 -- 172.21.15.26:0/1506156020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 msgr2=0x7fabd013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 -- 172.21.15.26:0/1506156020 >> v2:172.21.15.120:3300/0 conn(0x7fabd0147000 msgr2=0x7fabd00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/1506156020 >> v2:172.21.15.120:3300/0 conn(0x7fabd0147000 0x7fabd00b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 -- 172.21.15.26:0/1506156020 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0143480 con 0x7fabd005cfd0 2022-07-20T02:26:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd63f5700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd005cfd0 0x7fabd0143bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fabc80032f0 tx=0x7fabc8009290).ready entity=mon.0 client_cookie=10a3b2539392995b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd4bf2700 1 -- 172.21.15.26:0/1506156020 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc800b870 con 0x7fabd005cfd0 2022-07-20T02:26:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd4bf2700 1 -- 172.21.15.26:0/1506156020 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc800b9d0 con 0x7fabd005cfd0 2022-07-20T02:26:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd4bf2700 1 -- 172.21.15.26:0/1506156020 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc800bb40 con 0x7fabd005cfd0 2022-07-20T02:26:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- 172.21.15.26:0/1506156020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd005cfd0 msgr2=0x7fabd0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd005cfd0 0x7fabd0143bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fabc80032f0 tx=0x7fabc8009290).stop 2022-07-20T02:26:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- 172.21.15.26:0/1506156020 shutdown_connections 2022-07-20T02:26:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/1506156020 >> v2:172.21.15.120:3300/0 conn(0x7fabd0147000 0x7fabd00b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/1506156020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd005cfd0 0x7fabd0143bc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- 172.21.15.26:0/1506156020 >> 172.21.15.26:0/1506156020 conn(0x7fabd00620c0 msgr2=0x7fabd005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- 172.21.15.26:0/1506156020 shutdown_connections 2022-07-20T02:26:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- 172.21.15.26:0/1506156020 wait complete. 2022-07-20T02:26:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 Processor -- start 2022-07-20T02:26:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- start start 2022-07-20T02:26:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd0146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd0147000 0x7fabd00afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 0x7fabd0140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0142ad0 con 0x7fabd0147000 2022-07-20T02:26:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0144ec0 con 0x7fabd00affe0 2022-07-20T02:26:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd6bf6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00b7cd0 con 0x7fabd0060f00 2022-07-20T02:26:09.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd53f3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd0147000 0x7fabd00afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd63f5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 0x7fabd0140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd5bf4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd0146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd63f5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 0x7fabd0140f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:55808/0 (socket says 172.21.15.26:55808) 2022-07-20T02:26:09.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.348+0000 7fabd63f5700 1 -- 172.21.15.26:0/2126142597 learned_addr learned my addr 172.21.15.26:0/2126142597 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:09.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 -- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 msgr2=0x7fabd0140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 0x7fabd0140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 -- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd0147000 msgr2=0x7fabd00afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd0147000 0x7fabd00afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 -- 172.21.15.26:0/2126142597 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc8009530 con 0x7fabd0060f00 2022-07-20T02:26:09.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd5bf4700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd0146560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fabd00b0850 tx=0x7fabc8016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc800b870 con 0x7fabd0060f00 2022-07-20T02:26:09.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8011e60 con 0x7fabd0060f00 2022-07-20T02:26:09.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd00b7f70 con 0x7fabd0060f00 2022-07-20T02:26:09.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc8020620 con 0x7fabd0060f00 2022-07-20T02:26:09.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd00b0490 con 0x7fabd0060f00 2022-07-20T02:26:09.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd00b8730 con 0x7fabd0060f00 2022-07-20T02:26:09.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fabc800f070 con 0x7fabd0060f00 2022-07-20T02:26:09.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.352+0000 7fabc37fe700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 0x7fabac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.356+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fabc806cad0 con 0x7fabd0060f00 2022-07-20T02:26:09.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.356+0000 7fabd53f3700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 0x7fabac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.356+0000 7fabd53f3700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 0x7fabac055ea0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fabc400c000 tx=0x7fabc4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.360+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabc8043b40 con 0x7fabd0060f00 2022-07-20T02:26:09.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.528+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fabd00b8100 con 0x7fabd0060f00 2022-07-20T02:26:09.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabc37fe700 1 -- 172.21.15.26:0/2126142597 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7fabc8015070 con 0x7fabd0060f00 2022-07-20T02:26:09.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 msgr2=0x7fabac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 0x7fabac055ea0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fabc400c000 tx=0x7fabc4008040).stop 2022-07-20T02:26:09.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 msgr2=0x7fabd0146560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd0146560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fabd00b0850 tx=0x7fabc8016000).stop 2022-07-20T02:26:09.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 shutdown_connections 2022-07-20T02:26:09.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabd0147000 0x7fabd00afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabac0539e0 0x7fabac055ea0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabd00affe0 0x7fabd0140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 --2- 172.21.15.26:0/2126142597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabd0060f00 0x7fabd0146560 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 >> 172.21.15.26:0/2126142597 conn(0x7fabd005cfd0 msgr2=0x7fabd013a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:09.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 shutdown_connections 2022-07-20T02:26:09.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.544+0000 7fabd6bf6700 1 -- 172.21.15.26:0/2126142597 wait complete. 2022-07-20T02:26:09.558 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:09.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2022-07-20T02:26:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 Processor -- start 2022-07-20T02:26:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- start start 2022-07-20T02:26:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9a640b2ab0 0x7f9a64061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 0x7f9a640afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640aa1e0 con 0x7f9a640add30 2022-07-20T02:26:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640ae110 con 0x7f9a640b2ab0 2022-07-20T02:26:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640a9f40 con 0x7f9a640af6d0 2022-07-20T02:26:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9a640b2ab0 0x7f9a64061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9a640b2ab0 0x7f9a64061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36124/0 (socket says 172.21.15.26:36124) 2022-07-20T02:26:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 -- 172.21.15.26:0/1184199380 learned_addr learned my addr 172.21.15.26:0/1184199380 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 -- 172.21.15.26:0/1184199380 >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 msgr2=0x7f9a640afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a63fff700 1 --2- 172.21.15.26:0/1184199380 >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 0x7f9a640afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 --2- 172.21.15.26:0/1184199380 >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 0x7f9a640afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 -- 172.21.15.26:0/1184199380 >> v1:172.21.15.26:6789/0 conn(0x7f9a640add30 legacy=0x7f9a64060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 -- 172.21.15.26:0/1184199380 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640b1110 con 0x7f9a640b2ab0 2022-07-20T02:26:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a63fff700 1 --2- 172.21.15.26:0/1184199380 >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 0x7f9a640afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a62ffd700 1 --2- 172.21.15.26:0/1184199380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640b2ab0 0x7f9a64061060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9a58002f70 tx=0x7f9a58006eb0).ready entity=mon.0 client_cookie=3d5c77e68e16a004 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a627fc700 1 -- 172.21.15.26:0/1184199380 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5800d610 con 0x7f9a640b2ab0 2022-07-20T02:26:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a627fc700 1 -- 172.21.15.26:0/1184199380 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58000e90 con 0x7f9a640b2ab0 2022-07-20T02:26:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a627fc700 1 -- 172.21.15.26:0/1184199380 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5800d9a0 con 0x7f9a640b2ab0 2022-07-20T02:26:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- 172.21.15.26:0/1184199380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640b2ab0 msgr2=0x7f9a64061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 --2- 172.21.15.26:0/1184199380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640b2ab0 0x7f9a64061060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9a58002f70 tx=0x7f9a58006eb0).stop 2022-07-20T02:26:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- 172.21.15.26:0/1184199380 shutdown_connections 2022-07-20T02:26:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 --2- 172.21.15.26:0/1184199380 >> v2:172.21.15.120:3300/0 conn(0x7f9a640af6d0 0x7f9a640afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 --2- 172.21.15.26:0/1184199380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640b2ab0 0x7f9a64061060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- 172.21.15.26:0/1184199380 >> 172.21.15.26:0/1184199380 conn(0x7f9a640b3050 msgr2=0x7f9a640b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- 172.21.15.26:0/1184199380 shutdown_connections 2022-07-20T02:26:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.688+0000 7f9a68d56700 1 -- 172.21.15.26:0/1184199380 wait complete. 2022-07-20T02:26:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.692+0000 7f9a68d56700 1 Processor -- start 2022-07-20T02:26:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.692+0000 7f9a68d56700 1 -- start start 2022-07-20T02:26:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.692+0000 7f9a68d56700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a640af6d0 0x7f9a6409eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b2ab0 0x7f9a6413f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64064ad0 con 0x7f9a640add30 2022-07-20T02:26:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6414feb0 con 0x7f9a640af6d0 2022-07-20T02:26:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64142ba0 con 0x7f9a640b2ab0 2022-07-20T02:26:09.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36128/0 (socket says 172.21.15.26:36128) 2022-07-20T02:26:09.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 -- 172.21.15.26:0/3868139086 learned_addr learned my addr 172.21.15.26:0/3868139086 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a63fff700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b2ab0 0x7f9a6413f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a62ffd700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a640af6d0 0x7f9a6409eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 -- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b2ab0 msgr2=0x7f9a6413f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b2ab0 0x7f9a6413f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 -- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a640af6d0 msgr2=0x7f9a6409eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a640af6d0 0x7f9a6409eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 -- 172.21.15.26:0/3868139086 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5800b1e0 con 0x7f9a640add30 2022-07-20T02:26:09.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a637fe700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f9a640630a0 tx=0x7f9a54007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a54012400 con 0x7f9a640add30 2022-07-20T02:26:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54012560 con 0x7f9a640add30 2022-07-20T02:26:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6414fc10 con 0x7f9a640add30 2022-07-20T02:26:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a540128c0 con 0x7f9a640add30 2022-07-20T02:26:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640aa770 con 0x7f9a640add30 2022-07-20T02:26:09.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.696+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a640aa930 con 0x7f9a640add30 2022-07-20T02:26:09.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.700+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a54019030 con 0x7f9a640add30 2022-07-20T02:26:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.700+0000 7f9a60ff9700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 0x7f9a44055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.700+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f9a5406b130 con 0x7f9a640add30 2022-07-20T02:26:09.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.700+0000 7f9a62ffd700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 0x7f9a44055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:09.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.700+0000 7f9a62ffd700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 0x7f9a44055f30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9a5801c000 tx=0x7f9a58015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.704+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a54042120 con 0x7f9a640add30 2022-07-20T02:26:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.868+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f9a640b13b0 con 0x7f9a640add30 2022-07-20T02:26:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.876+0000 7f9a60ff9700 1 -- 172.21.15.26:0/3868139086 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9a54045740 con 0x7f9a640add30 2022-07-20T02:26:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 msgr2=0x7f9a44055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 0x7f9a44055f30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9a5801c000 tx=0x7f9a58015000).stop 2022-07-20T02:26:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 msgr2=0x7f9a64142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f9a640630a0 tx=0x7f9a54007c50).stop 2022-07-20T02:26:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 shutdown_connections 2022-07-20T02:26:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a44053a70 0x7f9a44055f30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b2ab0 0x7f9a6413f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a640af6d0 0x7f9a6409eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 --2- 172.21.15.26:0/3868139086 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a640add30 0x7f9a64142310 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 >> 172.21.15.26:0/3868139086 conn(0x7f9a640b3050 msgr2=0x7f9a641444d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 shutdown_connections 2022-07-20T02:26:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:09.880+0000 7f9a68d56700 1 -- 172.21.15.26:0/3868139086 wait complete. 2022-07-20T02:26:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:09.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2022-07-20T02:26:10.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 Processor -- start 2022-07-20T02:26:10.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 -- start start 2022-07-20T02:26:10.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 0x7f31c413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c41474e0 con 0x7f31c405cfd0 2022-07-20T02:26:10.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c40b4e40 con 0x7f31c4060f00 2022-07-20T02:26:10.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.004+0000 7f31c8af6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c405d3b0 con 0x7f31c4147000 2022-07-20T02:26:10.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57654/0 (socket says 172.21.15.26:57654) 2022-07-20T02:26:10.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 -- 172.21.15.26:0/801809558 learned_addr learned my addr 172.21.15.26:0/801809558 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:10.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 0x7f31c413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 -- 172.21.15.26:0/801809558 >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 msgr2=0x7f31c413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:10.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 --2- 172.21.15.26:0/801809558 >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 0x7f31c413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 -- 172.21.15.26:0/801809558 >> v1:172.21.15.120:6789/0 conn(0x7f31c405cfd0 legacy=0x7f31c40b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 -- 172.21.15.26:0/801809558 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c41431e0 con 0x7f31c4147000 2022-07-20T02:26:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c27fc700 1 -- 172.21.15.26:0/801809558 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910514294 0 0) 0x7f31c41474e0 con 0x7f31c405cfd0 2022-07-20T02:26:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c3fff700 1 --2- 172.21.15.26:0/801809558 >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 0x7f31c413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c2ffd700 1 --2- 172.21.15.26:0/801809558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f31b8002f70 tx=0x7f31b8006eb0).ready entity=mon.2 client_cookie=ba56d324cb27c406 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c27fc700 1 -- 172.21.15.26:0/801809558 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31b800d570 con 0x7f31c4147000 2022-07-20T02:26:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c27fc700 1 -- 172.21.15.26:0/801809558 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b8000e90 con 0x7f31c4147000 2022-07-20T02:26:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c27fc700 1 -- 172.21.15.26:0/801809558 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31b800d900 con 0x7f31c4147000 2022-07-20T02:26:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 -- 172.21.15.26:0/801809558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 msgr2=0x7f31c4143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 --2- 172.21.15.26:0/801809558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f31b8002f70 tx=0x7f31b8006eb0).stop 2022-07-20T02:26:10.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 -- 172.21.15.26:0/801809558 shutdown_connections 2022-07-20T02:26:10.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 --2- 172.21.15.26:0/801809558 >> v2:172.21.15.120:3300/0 conn(0x7f31c4060f00 0x7f31c413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 --2- 172.21.15.26:0/801809558 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4147000 0x7f31c4143bc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 -- 172.21.15.26:0/801809558 >> 172.21.15.26:0/801809558 conn(0x7f31c40620c0 msgr2=0x7f31c405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 -- 172.21.15.26:0/801809558 shutdown_connections 2022-07-20T02:26:10.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.008+0000 7f31c8af6700 1 -- 172.21.15.26:0/801809558 wait complete. 2022-07-20T02:26:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 Processor -- start 2022-07-20T02:26:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- start start 2022-07-20T02:26:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4060f00 0x7f31c41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c4147000 0x7f31c4155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4146990 con 0x7f31c4144d20 2022-07-20T02:26:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4142ed0 con 0x7f31c4147000 2022-07-20T02:26:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40ad730 con 0x7f31c4060f00 2022-07-20T02:26:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36146/0 (socket says 172.21.15.26:36146) 2022-07-20T02:26:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 -- 172.21.15.26:0/3508173472 learned_addr learned my addr 172.21.15.26:0/3508173472 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c2ffd700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c4147000 0x7f31c4155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c37fe700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4060f00 0x7f31c41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 -- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4060f00 msgr2=0x7f31c41419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4060f00 0x7f31c41419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 -- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c4147000 msgr2=0x7f31c4155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c4147000 0x7f31c4155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 -- 172.21.15.26:0/3508173472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc009470 con 0x7f31c4144d20 2022-07-20T02:26:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c3fff700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f31bc00c000 tx=0x7f31bc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31bc00ee60 con 0x7f31c4144d20 2022-07-20T02:26:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc00c850 con 0x7f31c4144d20 2022-07-20T02:26:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b800b1e0 con 0x7f31c4144d20 2022-07-20T02:26:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31bc00cbb0 con 0x7f31c4144d20 2022-07-20T02:26:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.012+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c40b37e0 con 0x7f31c4144d20 2022-07-20T02:26:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c40ad8a0 con 0x7f31c4144d20 2022-07-20T02:26:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f31bc022070 con 0x7f31c4144d20 2022-07-20T02:26:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c0ff9700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 0x7f31a4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f31bc06c3d0 con 0x7f31c4144d20 2022-07-20T02:26:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c37fe700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 0x7f31a4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.016+0000 7f31c37fe700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 0x7f31a4055ff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f31b400d000 tx=0x7f31b4007b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.024+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f31bc0434d0 con 0x7f31c4144d20 2022-07-20T02:26:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.188+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f31c4143480 con 0x7f31c4144d20 2022-07-20T02:26:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.196+0000 7f31c0ff9700 1 -- 172.21.15.26:0/3508173472 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f31bc046af0 con 0x7f31c4144d20 2022-07-20T02:26:10.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 msgr2=0x7f31a4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 0x7f31a4055ff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f31b400d000 tx=0x7f31b4007b30).stop 2022-07-20T02:26:10.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 msgr2=0x7f31c4155d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f31bc00c000 tx=0x7f31bc0091d0).stop 2022-07-20T02:26:10.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 shutdown_connections 2022-07-20T02:26:10.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31c4144d20 0x7f31c4155d60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31a4053b30 0x7f31a4055ff0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31c4147000 0x7f31c4155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 --2- 172.21.15.26:0/3508173472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31c4060f00 0x7f31c41419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 >> 172.21.15.26:0/3508173472 conn(0x7f31c405cfd0 msgr2=0x7f31c40b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 shutdown_connections 2022-07-20T02:26:10.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.200+0000 7f31c8af6700 1 -- 172.21.15.26:0/3508173472 wait complete. 2022-07-20T02:26:10.236 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:10.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2022-07-20T02:26:10.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 Processor -- start 2022-07-20T02:26:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- start start 2022-07-20T02:26:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb880b2ab0 0x7fcb88061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880af6d0 0x7fcb880afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb880aa1e0 con 0x7fcb880add30 2022-07-20T02:26:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb880ae110 con 0x7fcb880b2ab0 2022-07-20T02:26:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880a9f40 con 0x7fcb880af6d0 2022-07-20T02:26:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb880b2ab0 0x7fcb88061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb880b2ab0 0x7fcb88061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36154/0 (socket says 172.21.15.26:36154) 2022-07-20T02:26:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 -- 172.21.15.26:0/498729628 learned_addr learned my addr 172.21.15.26:0/498729628 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb86d9d700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880af6d0 0x7fcb880afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 -- 172.21.15.26:0/498729628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880af6d0 msgr2=0x7fcb880afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880af6d0 0x7fcb880afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 -- 172.21.15.26:0/498729628 >> v1:172.21.15.120:6789/0 conn(0x7fcb880add30 legacy=0x7fcb88060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 -- 172.21.15.26:0/498729628 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880b1110 con 0x7fcb880b2ab0 2022-07-20T02:26:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8559a700 1 -- 172.21.15.26:0/498729628 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674339558 0 0) 0x7fcb880aa1e0 con 0x7fcb880add30 2022-07-20T02:26:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb85d9b700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880b2ab0 0x7fcb88061060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c006eb0).ready entity=mon.0 client_cookie=2c69256da48c8b01 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8559a700 1 -- 172.21.15.26:0/498729628 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00d570 con 0x7fcb880b2ab0 2022-07-20T02:26:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8559a700 1 -- 172.21.15.26:0/498729628 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c000e90 con 0x7fcb880b2ab0 2022-07-20T02:26:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8559a700 1 -- 172.21.15.26:0/498729628 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00d8c0 con 0x7fcb880b2ab0 2022-07-20T02:26:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- 172.21.15.26:0/498729628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880b2ab0 msgr2=0x7fcb88061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880b2ab0 0x7fcb88061060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c006eb0).stop 2022-07-20T02:26:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- 172.21.15.26:0/498729628 shutdown_connections 2022-07-20T02:26:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880af6d0 0x7fcb880afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 --2- 172.21.15.26:0/498729628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880b2ab0 0x7fcb88061060 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- 172.21.15.26:0/498729628 >> 172.21.15.26:0/498729628 conn(0x7fcb880b3050 msgr2=0x7fcb880b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- 172.21.15.26:0/498729628 shutdown_connections 2022-07-20T02:26:10.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.348+0000 7fcb8759e700 1 -- 172.21.15.26:0/498729628 wait complete. 2022-07-20T02:26:10.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 Processor -- start 2022-07-20T02:26:10.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 -- start start 2022-07-20T02:26:10.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880add30 0x7fcb8812d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880b2ab0 0x7fcb880b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88129bd0 con 0x7fcb880af6d0 2022-07-20T02:26:10.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8812a8a0 con 0x7fcb880add30 2022-07-20T02:26:10.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8812c1b0 con 0x7fcb880b2ab0 2022-07-20T02:26:10.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb86d9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880b2ab0 0x7fcb880b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36158/0 (socket says 172.21.15.26:36158) 2022-07-20T02:26:10.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 -- 172.21.15.26:0/3071502651 learned_addr learned my addr 172.21.15.26:0/3071502651 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb8659c700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880add30 0x7fcb8812d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 -- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880b2ab0 msgr2=0x7fcb880b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880b2ab0 0x7fcb880b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 -- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880add30 msgr2=0x7fcb8812d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880add30 0x7fcb8812d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 -- 172.21.15.26:0/3071502651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c00b1e0 con 0x7fcb880af6d0 2022-07-20T02:26:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.352+0000 7fcb85d9b700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00d8f0 con 0x7fcb880af6d0 2022-07-20T02:26:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c011b80 con 0x7fcb880af6d0 2022-07-20T02:26:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8812ccb0 con 0x7fcb880af6d0 2022-07-20T02:26:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c01e460 con 0x7fcb880af6d0 2022-07-20T02:26:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8812d1f0 con 0x7fcb880af6d0 2022-07-20T02:26:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb880b13b0 con 0x7fcb880af6d0 2022-07-20T02:26:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb7c025030 con 0x7fcb880af6d0 2022-07-20T02:26:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 0x7fcb64055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fcb7c06cee0 con 0x7fcb880af6d0 2022-07-20T02:26:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.356+0000 7fcb8659c700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 0x7fcb64055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.360+0000 7fcb8659c700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 0x7fcb64055e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcb7800d000 tx=0x7fcb78007a90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.364+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb7c043fe0 con 0x7fcb880af6d0 2022-07-20T02:26:10.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.532+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fcb880574e0 con 0x7fcb880af6d0 2022-07-20T02:26:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.536+0000 7fcb777fe700 1 -- 172.21.15.26:0/3071502651 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcb7c047600 con 0x7fcb880af6d0 2022-07-20T02:26:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 msgr2=0x7fcb64055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 0x7fcb64055e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcb7800d000 tx=0x7fcb78007a90).stop 2022-07-20T02:26:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 msgr2=0x7fcb8812fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c015000).stop 2022-07-20T02:26:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 shutdown_connections 2022-07-20T02:26:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb880af6d0 0x7fcb8812fa40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb64053950 0x7fcb64055e10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb880b2ab0 0x7fcb880b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 --2- 172.21.15.26:0/3071502651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb880add30 0x7fcb8812d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 >> 172.21.15.26:0/3071502651 conn(0x7fcb880b3050 msgr2=0x7fcb8805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 shutdown_connections 2022-07-20T02:26:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.540+0000 7fcb8759e700 1 -- 172.21.15.26:0/3071502651 wait complete. 2022-07-20T02:26:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:10.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2022-07-20T02:26:10.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.692+0000 7f02bd014700 1 Processor -- start 2022-07-20T02:26:10.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.692+0000 7f02bd014700 1 -- start start 2022-07-20T02:26:10.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02b80b24c0 0x7f02b80604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02b805f840 0x7f02b80609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80b1ae0 con 0x7f02b81465b0 2022-07-20T02:26:10.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b80b2e20 con 0x7f02b80b24c0 2022-07-20T02:26:10.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b80b1d80 con 0x7f02b805f840 2022-07-20T02:26:10.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02b80b24c0 0x7f02b80604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02b7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f02b81465b0 0x7f02b8066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35444/0 (socket says 172.21.15.26:35444) 2022-07-20T02:26:10.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02b7fff700 1 -- 172.21.15.26:0/2266292233 learned_addr learned my addr 172.21.15.26:0/2266292233 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:10.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bc813700 1 --2- 172.21.15.26:0/2266292233 >> v2:172.21.15.120:3300/0 conn(0x7f02b805f840 0x7f02b80609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 -- 172.21.15.26:0/2266292233 >> v2:172.21.15.120:3300/0 conn(0x7f02b805f840 msgr2=0x7f02b80609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:10.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 --2- 172.21.15.26:0/2266292233 >> v2:172.21.15.120:3300/0 conn(0x7f02b805f840 0x7f02b80609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 -- 172.21.15.26:0/2266292233 >> v1:172.21.15.26:6789/0 conn(0x7f02b81465b0 legacy=0x7f02b8066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 -- 172.21.15.26:0/2266292233 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80b2020 con 0x7f02b80b24c0 2022-07-20T02:26:10.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02affff700 1 --2- 172.21.15.26:0/2266292233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b80b24c0 0x7f02b80604e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f02a0002f70 tx=0x7f02a0006df0).ready entity=mon.0 client_cookie=4cf4e8ea11fc672d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02b77fe700 1 -- 172.21.15.26:0/2266292233 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a000d570 con 0x7f02b80b24c0 2022-07-20T02:26:10.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02b77fe700 1 -- 172.21.15.26:0/2266292233 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0000e90 con 0x7f02b80b24c0 2022-07-20T02:26:10.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02b77fe700 1 -- 172.21.15.26:0/2266292233 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a000d8c0 con 0x7f02b80b24c0 2022-07-20T02:26:10.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- 172.21.15.26:0/2266292233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b80b24c0 msgr2=0x7f02b80604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 --2- 172.21.15.26:0/2266292233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b80b24c0 0x7f02b80604e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f02a0002f70 tx=0x7f02a0006df0).stop 2022-07-20T02:26:10.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- 172.21.15.26:0/2266292233 shutdown_connections 2022-07-20T02:26:10.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 --2- 172.21.15.26:0/2266292233 >> v2:172.21.15.120:3300/0 conn(0x7f02b805f840 0x7f02b80609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 --2- 172.21.15.26:0/2266292233 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b80b24c0 0x7f02b80604e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- 172.21.15.26:0/2266292233 >> 172.21.15.26:0/2266292233 conn(0x7f02b80b3050 msgr2=0x7f02b80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- 172.21.15.26:0/2266292233 shutdown_connections 2022-07-20T02:26:10.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.696+0000 7f02bd014700 1 -- 172.21.15.26:0/2266292233 wait complete. 2022-07-20T02:26:10.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 Processor -- start 2022-07-20T02:26:10.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 -- start start 2022-07-20T02:26:10.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 0x7f02b8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 0x7f02b8155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b8063800 0x7f02b8066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80abb10 con 0x7f02b81465b0 2022-07-20T02:26:10.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80abd40 con 0x7f02b805f840 2022-07-20T02:26:10.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80abfe0 con 0x7f02b8063800 2022-07-20T02:26:10.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02affff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 0x7f02b8155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 0x7f02b8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02affff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 0x7f02b8155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36172/0 (socket says 172.21.15.26:36172) 2022-07-20T02:26:10.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02affff700 1 -- 172.21.15.26:0/795501785 learned_addr learned my addr 172.21.15.26:0/795501785 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:10.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bc813700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b8063800 0x7f02b8066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 -- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b8063800 msgr2=0x7f02b8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b8063800 0x7f02b8066180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 -- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 msgr2=0x7f02b8155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 0x7f02b8155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 -- 172.21.15.26:0/795501785 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a000b180 con 0x7f02b805f840 2022-07-20T02:26:10.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b7fff700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 0x7f02b8145b40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f02b8064f30 tx=0x7f02a8007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a8007e50 con 0x7f02b805f840 2022-07-20T02:26:10.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80070d0 con 0x7f02b805f840 2022-07-20T02:26:10.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.700+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b8065ce0 con 0x7f02b805f840 2022-07-20T02:26:10.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a8007430 con 0x7f02b805f840 2022-07-20T02:26:10.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b8066670 con 0x7f02b805f840 2022-07-20T02:26:10.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b80ab860 con 0x7f02b805f840 2022-07-20T02:26:10.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f02a8010070 con 0x7f02b805f840 2022-07-20T02:26:10.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02b5ffb700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 0x7f029c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f02a806b270 con 0x7f02b805f840 2022-07-20T02:26:10.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02affff700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 0x7f029c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.704+0000 7f02affff700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 0x7f029c055f30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f02a001a000 tx=0x7f02a0014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:10.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.716+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02b80ab860 con 0x7f02b805f840 2022-07-20T02:26:10.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.888+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f02b80ab860 con 0x7f02b805f840 2022-07-20T02:26:10.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.896+0000 7f02b5ffb700 1 -- 172.21.15.26:0/795501785 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02b80ab860 con 0x7f02b805f840 2022-07-20T02:26:10.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 msgr2=0x7f029c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 0x7f029c055f30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f02a001a000 tx=0x7f02a0014000).stop 2022-07-20T02:26:10.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 msgr2=0x7f02b8145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:10.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 0x7f02b8145b40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f02b8064f30 tx=0x7f02a8007be0).stop 2022-07-20T02:26:10.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 shutdown_connections 2022-07-20T02:26:10.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02b81465b0 0x7f02b8155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f029c053a70 0x7f029c055f30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b8063800 0x7f02b8066180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 --2- 172.21.15.26:0/795501785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b805f840 0x7f02b8145b40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 >> 172.21.15.26:0/795501785 conn(0x7f02b80b3050 msgr2=0x7f02b80b55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 shutdown_connections 2022-07-20T02:26:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:10.900+0000 7f02bd014700 1 -- 172.21.15.26:0/795501785 wait complete. 2022-07-20T02:26:10.937 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:10.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2022-07-20T02:26:11.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.080+0000 7fbc2b59e700 1 Processor -- start 2022-07-20T02:26:11.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.080+0000 7fbc2b59e700 1 -- start start 2022-07-20T02:26:11.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbc2c067440 0x7fbc2c0950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c147080 con 0x7fbc2c0955d0 2022-07-20T02:26:11.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c143000 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c1472b0 con 0x7fbc2c067440 2022-07-20T02:26:11.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36178/0 (socket says 172.21.15.26:36178) 2022-07-20T02:26:11.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 -- 172.21.15.26:0/2620925297 learned_addr learned my addr 172.21.15.26:0/2620925297 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:11.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc23fff700 1 --2- 172.21.15.26:0/2620925297 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c067440 0x7fbc2c0950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 -- 172.21.15.26:0/2620925297 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c067440 msgr2=0x7fbc2c0950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 --2- 172.21.15.26:0/2620925297 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c067440 0x7fbc2c0950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 -- 172.21.15.26:0/2620925297 >> v1:172.21.15.120:6789/0 conn(0x7fbc2c0955d0 legacy=0x7fbc2c1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 -- 172.21.15.26:0/2620925297 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0a9920 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2a59c700 1 --2- 172.21.15.26:0/2620925297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fbc1c002f70 tx=0x7fbc1c006eb0).ready entity=mon.0 client_cookie=ed5c5d1460582b09 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc29d9b700 1 -- 172.21.15.26:0/2620925297 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc1c00d610 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc29d9b700 1 -- 172.21.15.26:0/2620925297 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c000e90 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc29d9b700 1 -- 172.21.15.26:0/2620925297 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc1c00d9a0 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- 172.21.15.26:0/2620925297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0b2f20 msgr2=0x7fbc2c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/2620925297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fbc1c002f70 tx=0x7fbc1c006eb0).stop 2022-07-20T02:26:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- 172.21.15.26:0/2620925297 shutdown_connections 2022-07-20T02:26:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/2620925297 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c067440 0x7fbc2c0950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/2620925297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0b2f20 0x7fbc2c0b3300 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- 172.21.15.26:0/2620925297 >> 172.21.15.26:0/2620925297 conn(0x7fbc2c0b67f0 msgr2=0x7fbc2c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- 172.21.15.26:0/2620925297 shutdown_connections 2022-07-20T02:26:11.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.084+0000 7fbc2b59e700 1 -- 172.21.15.26:0/2620925297 wait complete. 2022-07-20T02:26:11.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 Processor -- start 2022-07-20T02:26:11.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 -- start start 2022-07-20T02:26:11.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c067440 0x7fbc2c09b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0b2f20 0x7fbc2c0911a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0986c0 con 0x7fbc2c0955d0 2022-07-20T02:26:11.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0aff80 con 0x7fbc2c067440 2022-07-20T02:26:11.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c090810 con 0x7fbc2c0b2f20 2022-07-20T02:26:11.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc23fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc23fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36186/0 (socket says 172.21.15.26:36186) 2022-07-20T02:26:11.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc23fff700 1 -- 172.21.15.26:0/3149020346 learned_addr learned my addr 172.21.15.26:0/3149020346 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:11.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.088+0000 7fbc2a59c700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c067440 0x7fbc2c09b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc2ad9d700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0b2f20 0x7fbc2c0911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 -- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0b2f20 msgr2=0x7fbc2c0911a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0b2f20 0x7fbc2c0911a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 -- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c067440 msgr2=0x7fbc2c09b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c067440 0x7fbc2c09b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 -- 172.21.15.26:0/3149020346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c00b1e0 con 0x7fbc2c0955d0 2022-07-20T02:26:11.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc23fff700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbc140030b0 tx=0x7fbc14008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc14011780 con 0x7fbc2c0955d0 2022-07-20T02:26:11.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc140118e0 con 0x7fbc2c0955d0 2022-07-20T02:26:11.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c09aa20 con 0x7fbc2c0955d0 2022-07-20T02:26:11.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc14011c40 con 0x7fbc2c0955d0 2022-07-20T02:26:11.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c09adb0 con 0x7fbc2c0955d0 2022-07-20T02:26:11.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c0909d0 con 0x7fbc2c0955d0 2022-07-20T02:26:11.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbc1401c460 con 0x7fbc2c0955d0 2022-07-20T02:26:11.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 0x7fbc10055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.092+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fbc1406be90 con 0x7fbc2c0955d0 2022-07-20T02:26:11.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.096+0000 7fbc2a59c700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 0x7fbc10055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.096+0000 7fbc2a59c700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 0x7fbc10055e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbc1c01c000 tx=0x7fbc1c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.100+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc14042f30 con 0x7fbc2c0955d0 2022-07-20T02:26:11.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.268+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fbc2c0b0140 con 0x7fbc2c0955d0 2022-07-20T02:26:11.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.276+0000 7fbc237fe700 1 -- 172.21.15.26:0/3149020346 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v12) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbc14046550 con 0x7fbc2c0955d0 2022-07-20T02:26:11.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 msgr2=0x7fbc10055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 0x7fbc10055e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbc1c01c000 tx=0x7fbc1c015000).stop 2022-07-20T02:26:11.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 msgr2=0x7fbc2c090cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbc140030b0 tx=0x7fbc14008040).stop 2022-07-20T02:26:11.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 shutdown_connections 2022-07-20T02:26:11.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbc2c0955d0 0x7fbc2c090cb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbc10053950 0x7fbc10055e10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0b2f20 0x7fbc2c0911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 --2- 172.21.15.26:0/3149020346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c067440 0x7fbc2c09b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 >> 172.21.15.26:0/3149020346 conn(0x7fbc2c0b67f0 msgr2=0x7fbc2c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 shutdown_connections 2022-07-20T02:26:11.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.280+0000 7fbc2b59e700 1 -- 172.21.15.26:0/3149020346 wait complete. 2022-07-20T02:26:11.296 INFO:teuthology.orchestra.run.smithi026.stderr:updated 2022-07-20T02:26:11.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:26:11.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 Processor -- start 2022-07-20T02:26:11.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 -- start start 2022-07-20T02:26:11.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5400b06e0 0x7fe54014a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5400aa1e0 con 0x7fe5400add30 2022-07-20T02:26:11.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5400ae110 con 0x7fe5400b2ab0 2022-07-20T02:26:11.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54754e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5400a9f40 con 0x7fe5400b06e0 2022-07-20T02:26:11.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54654c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe5400add30 0x7fe540060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35472/0 (socket says 172.21.15.26:35472) 2022-07-20T02:26:11.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54654c700 1 -- 172.21.15.26:0/320970994 learned_addr learned my addr 172.21.15.26:0/320970994 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:11.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe546d4d700 1 --2- 172.21.15.26:0/320970994 >> v2:172.21.15.120:3300/0 conn(0x7fe5400b06e0 0x7fe54014a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724987384 0 0) 0x7fe5400aa1e0 con 0x7fe5400add30 2022-07-20T02:26:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528003260 con 0x7fe5400add30 2022-07-20T02:26:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634366761 0 0) 0x7fe5400ae110 con 0x7fe5400b2ab0 2022-07-20T02:26:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5400aa1e0 con 0x7fe5400b2ab0 2022-07-20T02:26:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe546d4d700 1 -- 172.21.15.26:0/320970994 >> v1:172.21.15.120:6789/0 conn(0x7fe5400b2ab0 legacy=0x7fe540061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe546d4d700 1 -- 172.21.15.26:0/320970994 >> v1:172.21.15.26:6789/0 conn(0x7fe5400add30 legacy=0x7fe540060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe546d4d700 1 -- 172.21.15.26:0/320970994 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400b1110 con 0x7fe5400b06e0 2022-07-20T02:26:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe546d4d700 1 --2- 172.21.15.26:0/320970994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe54014a9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe5380032f0 tx=0x7fe5380091d0).ready entity=mon.1 client_cookie=210cf74f304d8f09 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe53800b600 con 0x7fe5400b06e0 2022-07-20T02:26:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53800b760 con 0x7fe5400b06e0 2022-07-20T02:26:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.420+0000 7fe54554a700 1 -- 172.21.15.26:0/320970994 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe53800b8d0 con 0x7fe5400b06e0 2022-07-20T02:26:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- 172.21.15.26:0/320970994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 msgr2=0x7fe54014a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 --2- 172.21.15.26:0/320970994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe54014a9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe5380032f0 tx=0x7fe5380091d0).stop 2022-07-20T02:26:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- 172.21.15.26:0/320970994 shutdown_connections 2022-07-20T02:26:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 --2- 172.21.15.26:0/320970994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe54014a9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- 172.21.15.26:0/320970994 >> 172.21.15.26:0/320970994 conn(0x7fe5400b3050 msgr2=0x7fe5400b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- 172.21.15.26:0/320970994 shutdown_connections 2022-07-20T02:26:11.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- 172.21.15.26:0/320970994 wait complete. 2022-07-20T02:26:11.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 Processor -- start 2022-07-20T02:26:11.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- start start 2022-07-20T02:26:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe5400aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5400b2ab0 0x7fe5400aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400ace90 con 0x7fe5400add30 2022-07-20T02:26:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400a3950 con 0x7fe5400b06e0 2022-07-20T02:26:11.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54754e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400ab6e0 con 0x7fe5400b2ab0 2022-07-20T02:26:11.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54654c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54654c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36198/0 (socket says 172.21.15.26:36198) 2022-07-20T02:26:11.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe54654c700 1 -- 172.21.15.26:0/1151442462 learned_addr learned my addr 172.21.15.26:0/1151442462 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:11.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe545d4b700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe5400aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.424+0000 7fe546d4d700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5400b2ab0 0x7fe5400aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 -- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5400b2ab0 msgr2=0x7fe5400aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5400b2ab0 0x7fe5400aa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 -- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 msgr2=0x7fe54005fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 -- 172.21.15.26:0/1151442462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538009470 con 0x7fe5400b06e0 2022-07-20T02:26:11.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe54654c700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:11.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe545d4b700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe5400aa3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe530001110 tx=0x7fe530007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe53000ce90 con 0x7fe5400b06e0 2022-07-20T02:26:11.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530012400 con 0x7fe5400b06e0 2022-07-20T02:26:11.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe530012760 con 0x7fe5400b06e0 2022-07-20T02:26:11.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5400ab980 con 0x7fe5400b06e0 2022-07-20T02:26:11.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5400ac550 con 0x7fe5400b06e0 2022-07-20T02:26:11.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5400b13b0 con 0x7fe5400b06e0 2022-07-20T02:26:11.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe530019030 con 0x7fe5400b06e0 2022-07-20T02:26:11.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.428+0000 7fe52f7fe700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 0x7fe520055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.432+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe53006b1a0 con 0x7fe5400b06e0 2022-07-20T02:26:11.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.432+0000 7fe54654c700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 0x7fe520055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.432+0000 7fe54654c700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 0x7fe520055f30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe53c00d000 tx=0x7fe53c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.440+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe530042110 con 0x7fe5400b06e0 2022-07-20T02:26:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.568+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5400574e0 con 0x7fe5400b06e0 2022-07-20T02:26:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe52f7fe700 1 -- 172.21.15.26:0/1151442462 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13462 (secure 0 0 0) 0x7fe530045730 con 0x7fe5400b06e0 2022-07-20T02:26:11.582 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:11.583 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":29,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:26:08.605748+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:26:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 msgr2=0x7fe520055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 0x7fe520055f30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe53c00d000 tx=0x7fe53c009040).stop 2022-07-20T02:26:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 msgr2=0x7fe5400aa3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe5400aa3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe530001110 tx=0x7fe530007ad0).stop 2022-07-20T02:26:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 shutdown_connections 2022-07-20T02:26:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe520053a70 0x7fe520055f30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5400b2ab0 0x7fe5400aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5400b06e0 0x7fe5400aa3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 --2- 172.21.15.26:0/1151442462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5400add30 0x7fe54005fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 >> 172.21.15.26:0/1151442462 conn(0x7fe5400b3050 msgr2=0x7fe5400afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.572+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 shutdown_connections 2022-07-20T02:26:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.576+0000 7fe54754e700 1 -- 172.21.15.26:0/1151442462 wait complete. 2022-07-20T02:26:11.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:26:11.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:26:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 Processor -- start 2022-07-20T02:26:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 -- start start 2022-07-20T02:26:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f9090098750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90900b2430 0x7f9090098fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90900b55d0 con 0x7f90900a95b0 2022-07-20T02:26:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900b1f10 con 0x7f90900b2430 2022-07-20T02:26:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f909869c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9090097300 con 0x7f90900b51f0 2022-07-20T02:26:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90900b2430 0x7f9090098fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9096438700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f9090098750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9096c39700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f90900a95b0 0x7f9090095cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49702/0 (socket says 172.21.15.26:49702) 2022-07-20T02:26:11.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9096c39700 1 -- 172.21.15.26:0/1164487556 learned_addr learned my addr 172.21.15.26:0/1164487556 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:11.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90900b2430 0x7f9090098fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:55900/0 (socket says 172.21.15.26:55900) 2022-07-20T02:26:11.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 -- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 msgr2=0x7f9090098750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:11.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 --2- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f9090098750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 -- 172.21.15.26:0/1164487556 >> v1:172.21.15.120:6789/0 conn(0x7f90900a95b0 legacy=0x7f9090095cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.736+0000 7f9095c37700 1 -- 172.21.15.26:0/1164487556 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900b19d0 con 0x7f90900b2430 2022-07-20T02:26:11.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f9095c37700 1 --2- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900b2430 0x7f9090098fa0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9090097ec0 tx=0x7f9080006df0).ready entity=mon.1 client_cookie=90554f9560645e1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f9095436700 1 -- 172.21.15.26:0/1164487556 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908000d570 con 0x7f90900b2430 2022-07-20T02:26:11.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f9095436700 1 -- 172.21.15.26:0/1164487556 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080000e90 con 0x7f90900b2430 2022-07-20T02:26:11.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f9095436700 1 -- 172.21.15.26:0/1164487556 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908000d8c0 con 0x7f90900b2430 2022-07-20T02:26:11.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 -- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900b2430 msgr2=0x7f9090098fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 --2- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900b2430 0x7f9090098fa0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9090097ec0 tx=0x7f9080006df0).stop 2022-07-20T02:26:11.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 -- 172.21.15.26:0/1164487556 shutdown_connections 2022-07-20T02:26:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 --2- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900b2430 0x7f9090098fa0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 --2- 172.21.15.26:0/1164487556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f9090098750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 -- 172.21.15.26:0/1164487556 >> 172.21.15.26:0/1164487556 conn(0x7f90900b5ea0 msgr2=0x7f90900b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 -- 172.21.15.26:0/1164487556 shutdown_connections 2022-07-20T02:26:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.740+0000 7f909869c700 1 -- 172.21.15.26:0/1164487556 wait complete. 2022-07-20T02:26:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 Processor -- start 2022-07-20T02:26:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- start start 2022-07-20T02:26:11.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 0x7f9090066500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f90900abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090094e60 con 0x7f90900b2430 2022-07-20T02:26:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090093100 con 0x7f90900a95b0 2022-07-20T02:26:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90900ac190 con 0x7f90900b51f0 2022-07-20T02:26:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36216/0 (socket says 172.21.15.26:36216) 2022-07-20T02:26:11.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 -- 172.21.15.26:0/880156625 learned_addr learned my addr 172.21.15.26:0/880156625 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:11.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9096c39700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f90900abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9096438700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 0x7f9090066500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 -- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 msgr2=0x7f90900abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f90900abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 -- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 msgr2=0x7f9090066500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 0x7f9090066500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 -- 172.21.15.26:0/880156625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908000b180 con 0x7f90900b2430 2022-07-20T02:26:11.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9096438700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 0x7f9090066500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:11.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f9095c37700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9090097ec0 tx=0x7f9080014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9080013e90 con 0x7f90900b2430 2022-07-20T02:26:11.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080002f70 con 0x7f90900b2430 2022-07-20T02:26:11.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9090094bc0 con 0x7f90900b2430 2022-07-20T02:26:11.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908001e5f0 con 0x7f90900b2430 2022-07-20T02:26:11.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.744+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90900941a0 con 0x7f90900b2430 2022-07-20T02:26:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90900932d0 con 0x7f90900b2430 2022-07-20T02:26:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9080025030 con 0x7f90900b2430 2022-07-20T02:26:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f907f7fe700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 0x7f9070055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f908006ca00 con 0x7f90900b2430 2022-07-20T02:26:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f9096438700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 0x7f9070055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.748+0000 7f9096438700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 0x7f9070055f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f908c00b000 tx=0x7f908c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.752+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9080043b00 con 0x7f90900b2430 2022-07-20T02:26:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f90900574e0 con 0x7f90900b2430 2022-07-20T02:26:11.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f907f7fe700 1 -- 172.21.15.26:0/880156625 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v30) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9080023070 con 0x7f90900b2430 2022-07-20T02:26:11.990 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:26:11.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 msgr2=0x7f9070055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 0x7f9070055f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f908c00b000 tx=0x7f908c008040).stop 2022-07-20T02:26:11.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 msgr2=0x7f90900ab7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:11.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9090097ec0 tx=0x7f9080014000).stop 2022-07-20T02:26:11.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 shutdown_connections 2022-07-20T02:26:11.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90900b2430 0x7f90900ab7b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9070053a70 0x7f9070055f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90900b51f0 0x7f90900abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 --2- 172.21.15.26:0/880156625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90900a95b0 0x7f9090066500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:11.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.884+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 >> 172.21.15.26:0/880156625 conn(0x7f90900b5ea0 msgr2=0x7f90900610c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:11.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.888+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 shutdown_connections 2022-07-20T02:26:11.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:11.888+0000 7f909869c700 1 -- 172.21.15.26:0/880156625 wait complete. 2022-07-20T02:26:12.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:26:12.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.128+0000 7f075e2f5700 1 Processor -- start 2022-07-20T02:26:12.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.128+0000 7f075e2f5700 1 -- start start 2022-07-20T02:26:12.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.128+0000 7f075e2f5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f07580b2ac0 0x7f0758060a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.128+0000 7f075e2f5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b0680 0x7f0758141460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.128+0000 7f075e2f5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07580ac430 con 0x7f0758061010 2022-07-20T02:26:12.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07580b1d00 con 0x7f07580b2ac0 2022-07-20T02:26:12.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07580b2ea0 con 0x7f07580b0680 2022-07-20T02:26:12.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f07577fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0758061010 0x7f07580aff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35500/0 (socket says 172.21.15.26:35500) 2022-07-20T02:26:12.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f07577fe700 1 -- 172.21.15.26:0/3627556863 learned_addr learned my addr 172.21.15.26:0/3627556863 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:12.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 --2- 172.21.15.26:0/3627556863 >> v2:172.21.15.26:3300/0 conn(0x7f07580b2ac0 0x7f0758060a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075c892700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b0680 0x7f0758141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658035070 0 0) 0x7f07580ac430 con 0x7f0758061010 2022-07-20T02:26:12.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0738003260 con 0x7f0758061010 2022-07-20T02:26:12.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 -- 172.21.15.26:0/3627556863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b0680 msgr2=0x7f0758141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b0680 0x7f0758141460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867316404 0 0) 0x7f0738003260 con 0x7f0758061010 2022-07-20T02:26:12.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 -- 172.21.15.26:0/3627556863 >> v1:172.21.15.26:6789/0 conn(0x7f0758061010 legacy=0x7f07580aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 -- 172.21.15.26:0/3627556863 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580ac190 con 0x7f07580b2ac0 2022-07-20T02:26:12.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0757fff700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07580b2ac0 0x7f0758060a60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0748001b80 tx=0x7f0740006df0).ready entity=mon.0 client_cookie=bf7285c6f194e2a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07480019e0 con 0x7f07580b2ac0 2022-07-20T02:26:12.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0740000e90 con 0x7f07580b2ac0 2022-07-20T02:26:12.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f0756ffd700 1 -- 172.21.15.26:0/3627556863 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0740012460 con 0x7f07580b2ac0 2022-07-20T02:26:12.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- 172.21.15.26:0/3627556863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07580b2ac0 msgr2=0x7f0758060a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07580b2ac0 0x7f0758060a60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0748001b80 tx=0x7f0740006df0).stop 2022-07-20T02:26:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- 172.21.15.26:0/3627556863 shutdown_connections 2022-07-20T02:26:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b0680 0x7f0758141460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 --2- 172.21.15.26:0/3627556863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07580b2ac0 0x7f0758060a60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- 172.21.15.26:0/3627556863 >> 172.21.15.26:0/3627556863 conn(0x7f07580b60a0 msgr2=0x7f07580b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- 172.21.15.26:0/3627556863 shutdown_connections 2022-07-20T02:26:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- 172.21.15.26:0/3627556863 wait complete. 2022-07-20T02:26:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 Processor -- start 2022-07-20T02:26:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.132+0000 7f075e2f5700 1 -- start start 2022-07-20T02:26:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07580b0680 0x7f0758065ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b2ac0 0x7f07580663d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580668c0 con 0x7f0758061010 2022-07-20T02:26:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0758064c20 con 0x7f07580b0680 2022-07-20T02:26:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0758065160 con 0x7f07580b2ac0 2022-07-20T02:26:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36226/0 (socket says 172.21.15.26:36226) 2022-07-20T02:26:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 -- 172.21.15.26:0/1962454693 learned_addr learned my addr 172.21.15.26:0/1962454693 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075c892700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b2ac0 0x7f07580663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f07577fe700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07580b0680 0x7f0758065ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 -- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b2ac0 msgr2=0x7f07580663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b2ac0 0x7f07580663d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 -- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07580b0680 msgr2=0x7f0758065ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07580b0680 0x7f0758065ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 -- 172.21.15.26:0/1962454693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074000b180 con 0x7f0758061010 2022-07-20T02:26:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f0757fff700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0748001b80 tx=0x7f0740014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0740016070 con 0x7f0758061010 2022-07-20T02:26:12.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074000bec0 con 0x7f0758061010 2022-07-20T02:26:12.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0758065400 con 0x7f0758061010 2022-07-20T02:26:12.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074000ddf0 con 0x7f0758061010 2022-07-20T02:26:12.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0758064910 con 0x7f0758061010 2022-07-20T02:26:12.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0740025030 con 0x7f0758061010 2022-07-20T02:26:12.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.136+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07580b13a0 con 0x7f0758061010 2022-07-20T02:26:12.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.140+0000 7f07557fa700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 0x7f073c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.140+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f074006ccc0 con 0x7f0758061010 2022-07-20T02:26:12.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.140+0000 7f07577fe700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 0x7f073c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.140+0000 7f07577fe700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 0x7f073c055e40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f074800d000 tx=0x7f0748000f40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.144+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0740043dc0 con 0x7f0758061010 2022-07-20T02:26:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f07580574e0 con 0x7f0758061010 2022-07-20T02:26:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f07557fa700 1 -- 172.21.15.26:0/1962454693 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v30) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0740023070 con 0x7f0758061010 2022-07-20T02:26:12.280 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:26:12.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 msgr2=0x7f073c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 0x7f073c055e40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f074800d000 tx=0x7f0748000f40).stop 2022-07-20T02:26:12.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 msgr2=0x7f0758143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0748001b80 tx=0x7f0740014000).stop 2022-07-20T02:26:12.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 shutdown_connections 2022-07-20T02:26:12.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f073c053980 0x7f073c055e40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07580b2ac0 0x7f07580663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07580b0680 0x7f0758065ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 --2- 172.21.15.26:0/1962454693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0758061010 0x7f0758143d20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 >> 172.21.15.26:0/1962454693 conn(0x7f07580b60a0 msgr2=0x7f07580af530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:12.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 shutdown_connections 2022-07-20T02:26:12.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.272+0000 7f075e2f5700 1 -- 172.21.15.26:0/1962454693 wait complete. 2022-07-20T02:26:12.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:26:12.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.424+0000 7f2c844b7700 1 Processor -- start 2022-07-20T02:26:12.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.424+0000 7f2c844b7700 1 -- start start 2022-07-20T02:26:12.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2c7c0b2920 0x7f2c7c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c7c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0b1bc0 con 0x7f2c7c066a80 2022-07-20T02:26:12.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c14b040 con 0x7f2c7c0b2920 2022-07-20T02:26:12.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0a8860 con 0x7f2c7c061050 2022-07-20T02:26:12.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82a54700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2c7c066a80 0x7f2c7c066ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35518/0 (socket says 172.21.15.26:35518) 2022-07-20T02:26:12.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82a54700 1 -- 172.21.15.26:0/2549829021 learned_addr learned my addr 172.21.15.26:0/2549829021 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:12.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81a52700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c7c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 --2- 172.21.15.26:0/2549829021 >> v2:172.21.15.120:3300/0 conn(0x7f2c7c0b2920 0x7f2c7c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357713463 0 0) 0x7f2c7c0b1bc0 con 0x7f2c7c066a80 2022-07-20T02:26:12.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5c003260 con 0x7f2c7c066a80 2022-07-20T02:26:12.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208404653 0 0) 0x7f2c5c003260 con 0x7f2c7c066a80 2022-07-20T02:26:12.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 msgr2=0x7f2c7c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c7c148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 >> v2:172.21.15.120:3300/0 conn(0x7f2c7c0b2920 msgr2=0x7f2c7c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 --2- 172.21.15.26:0/2549829021 >> v2:172.21.15.120:3300/0 conn(0x7f2c7c0b2920 0x7f2c7c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c147700 con 0x7f2c7c066a80 2022-07-20T02:26:12.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f2c74001660 con 0x7f2c7c066a80 2022-07-20T02:26:12.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 >> v1:172.21.15.26:6789/0 conn(0x7f2c7c066a80 legacy=0x7f2c7c066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c5c003750 0x7f2c5c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c5c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c5c0049f0 0x7f2c5c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c14b040 con 0x7f2c5c003750 2022-07-20T02:26:12.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0a8860 con 0x7f2c5c0049f0 2022-07-20T02:26:12.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c00b750 con 0x7f2c7c061050 2022-07-20T02:26:12.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c5c003750 0x7f2c5c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81a52700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c5c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 -- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 msgr2=0x7f2c5c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c7c061050 0x7f2c5c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82a54700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c5c0049f0 0x7f2c5c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 -- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c5c0049f0 msgr2=0x7f2c5c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c5c0049f0 0x7f2c5c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 -- 172.21.15.26:0/2549829021 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c003260 con 0x7f2c5c003750 2022-07-20T02:26:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82a54700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c5c0049f0 0x7f2c5c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:12.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82253700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c5c003750 0x7f2c5c003b70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f2c6c00c000 tx=0x7f2c6c0091d0).ready entity=mon.0 client_cookie=c5bce0394f5956cb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c6c00e7e0 con 0x7f2c5c003750 2022-07-20T02:26:12.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c00e940 con 0x7f2c5c003750 2022-07-20T02:26:12.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c81251700 1 -- 172.21.15.26:0/2549829021 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c6c00eca0 con 0x7f2c5c003750 2022-07-20T02:26:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 -- 172.21.15.26:0/2549829021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c5c003750 msgr2=0x7f2c5c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2549829021 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c5c003750 0x7f2c5c003b70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f2c6c00c000 tx=0x7f2c6c0091d0).stop 2022-07-20T02:26:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.428+0000 7f2c82a54700 1 -- 172.21.15.26:0/2549829021 reap_dead start 2022-07-20T02:26:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- 172.21.15.26:0/2549829021 shutdown_connections 2022-07-20T02:26:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- 172.21.15.26:0/2549829021 >> 172.21.15.26:0/2549829021 conn(0x7f2c7c0b60a0 msgr2=0x7f2c7c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:12.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- 172.21.15.26:0/2549829021 shutdown_connections 2022-07-20T02:26:12.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- 172.21.15.26:0/2549829021 wait complete. 2022-07-20T02:26:12.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 Processor -- start 2022-07-20T02:26:12.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- start start 2022-07-20T02:26:12.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c5c003750 0x7f2c7c145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 0x7f2c7c143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c144500 con 0x7f2c7c146970 2022-07-20T02:26:12.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c146420 con 0x7f2c7c145a30 2022-07-20T02:26:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c844b7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c1466c0 con 0x7f2c5c003750 2022-07-20T02:26:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c82253700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c5c003750 0x7f2c7c145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36248/0 (socket says 172.21.15.26:36248) 2022-07-20T02:26:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 -- 172.21.15.26:0/2955961073 learned_addr learned my addr 172.21.15.26:0/2955961073 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:12.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c82a54700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 0x7f2c7c143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 -- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c5c003750 msgr2=0x7f2c7c145250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c5c003750 0x7f2c7c145250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 -- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 msgr2=0x7f2c7c143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 0x7f2c7c143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 -- 172.21.15.26:0/2955961073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c156810 con 0x7f2c7c146970 2022-07-20T02:26:12.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c82a54700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 0x7f2c7c143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:12.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.432+0000 7f2c81a52700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f2c6400b000 tx=0x7f2c64008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c64014680 con 0x7f2c7c146970 2022-07-20T02:26:12.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c640147e0 con 0x7f2c7c146970 2022-07-20T02:26:12.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c156ab0 con 0x7f2c7c146970 2022-07-20T02:26:12.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c64014b40 con 0x7f2c7c146970 2022-07-20T02:26:12.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c063c20 con 0x7f2c7c146970 2022-07-20T02:26:12.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c156c40 con 0x7f2c7c146970 2022-07-20T02:26:12.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2c64024070 con 0x7f2c7c146970 2022-07-20T02:26:12.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 0x7f2c60055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f2c6406c5b0 con 0x7f2c7c146970 2022-07-20T02:26:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.436+0000 7f2c82253700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 0x7f2c60055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.440+0000 7f2c82253700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 0x7f2c60055fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2c6c01e000 tx=0x7f2c6c01a000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.444+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c64043650 con 0x7f2c7c146970 2022-07-20T02:26:12.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2c7c14ada0 con 0x7f2c7c146970 2022-07-20T02:26:12.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c7b7fe700 1 -- 172.21.15.26:0/2955961073 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+143 (secure 0 0 0) 0x7f2c64046c70 con 0x7f2c7c146970 2022-07-20T02:26:12.589 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:12.590 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:26:12.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:12.591 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:26:12.592 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:26:12.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:26:12.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:26:12.593 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:26:12.593 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:26:12.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:26:12.594 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:26:12.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 msgr2=0x7f2c60055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 0x7f2c60055fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2c6c01e000 tx=0x7f2c6c01a000).stop 2022-07-20T02:26:12.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 msgr2=0x7f2c7c156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f2c6400b000 tx=0x7f2c64008040).stop 2022-07-20T02:26:12.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 shutdown_connections 2022-07-20T02:26:12.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c7c146970 0x7f2c7c156f50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2c60053b00 0x7f2c60055fc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c7c145a30 0x7f2c7c143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 --2- 172.21.15.26:0/2955961073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2c5c003750 0x7f2c7c145250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 >> 172.21.15.26:0/2955961073 conn(0x7f2c7c0b60a0 msgr2=0x7f2c7c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:12.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 shutdown_connections 2022-07-20T02:26:12.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.580+0000 7f2c844b7700 1 -- 172.21.15.26:0/2955961073 wait complete. 2022-07-20T02:26:12.611 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:26:12.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:26:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.764+0000 7f3d39fad700 1 Processor -- start 2022-07-20T02:26:12.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.764+0000 7f3d39fad700 1 -- start start 2022-07-20T02:26:12.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d39fad700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d341407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d39fad700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d34147a00 con 0x7f3d340b1160 2022-07-20T02:26:12.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d39fad700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3405d3b0 con 0x7f3d34146e80 2022-07-20T02:26:12.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d39fad700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d341435b0 con 0x7f3d3405cfd0 2022-07-20T02:26:12.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d32ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3d340b1160 0x7f3d34140cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35538/0 (socket says 172.21.15.26:35538) 2022-07-20T02:26:12.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d32ffd700 1 -- 172.21.15.26:0/1870680924 learned_addr learned my addr 172.21.15.26:0/1870680924 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:12.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d341407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433004101 0 0) 0x7f3d34147a00 con 0x7f3d340b1160 2022-07-20T02:26:12.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003260 con 0x7f3d340b1160 2022-07-20T02:26:12.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898131426 0 0) 0x7f3d3405d3b0 con 0x7f3d34146e80 2022-07-20T02:26:12.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34147a00 con 0x7f3d34146e80 2022-07-20T02:26:12.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095348093 0 0) 0x7f3d20003260 con 0x7f3d340b1160 2022-07-20T02:26:12.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 msgr2=0x7f3d341407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d341407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 >> v1:172.21.15.120:6789/0 conn(0x7f3d34146e80 legacy=0x7f3d341472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340b4bc0 con 0x7f3d340b1160 2022-07-20T02:26:12.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f3d24001660 con 0x7f3d340b1160 2022-07-20T02:26:12.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 >> v1:172.21.15.26:6789/0 conn(0x7f3d340b1160 legacy=0x7f3d34140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d200049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d20004e90 0x7f3d200034a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20003990 0x7f3d20003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0015b0 con 0x7f3d20004e90 2022-07-20T02:26:12.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000b780 con 0x7f3d20003990 2022-07-20T02:26:12.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000b940 con 0x7f3d3405cfd0 2022-07-20T02:26:12.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d32ffd700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d20004e90 0x7f3d200034a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d200049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d33fff700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20003990 0x7f3d20003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 -- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20003990 msgr2=0x7f3d20003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20003990 0x7f3d20003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 -- 172.21.15.26:0/1870680924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d20004e90 msgr2=0x7f3d200034a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:12.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d20004e90 0x7f3d200034a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 -- 172.21.15.26:0/1870680924 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20003260 con 0x7f3d3405cfd0 2022-07-20T02:26:12.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d32ffd700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d20004e90 0x7f3d200034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:12.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d200049a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3d2800c000 tx=0x7f3d28006df0).ready entity=mon.2 client_cookie=105f758f14c07ac8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d2800e830 con 0x7f3d3405cfd0 2022-07-20T02:26:12.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2800e990 con 0x7f3d3405cfd0 2022-07-20T02:26:12.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.768+0000 7f3d327fc700 1 -- 172.21.15.26:0/1870680924 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d2800ecf0 con 0x7f3d3405cfd0 2022-07-20T02:26:12.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 msgr2=0x7f3d200049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1870680924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3405cfd0 0x7f3d200049a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3d2800c000 tx=0x7f3d28006df0).stop 2022-07-20T02:26:12.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d33fff700 1 -- 172.21.15.26:0/1870680924 reap_dead start 2022-07-20T02:26:12.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- 172.21.15.26:0/1870680924 shutdown_connections 2022-07-20T02:26:12.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- 172.21.15.26:0/1870680924 >> 172.21.15.26:0/1870680924 conn(0x7f3d34066f70 msgr2=0x7f3d3405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:12.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- 172.21.15.26:0/1870680924 shutdown_connections 2022-07-20T02:26:12.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- 172.21.15.26:0/1870680924 wait complete. 2022-07-20T02:26:12.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 Processor -- start 2022-07-20T02:26:12.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- start start 2022-07-20T02:26:12.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 0x7f3d341530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d340ad8a0 0x7f3d340b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34152510 con 0x7f3d340afbe0 2022-07-20T02:26:12.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34152700 con 0x7f3d3405cfd0 2022-07-20T02:26:12.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d39fad700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d341529a0 con 0x7f3d340ad8a0 2022-07-20T02:26:12.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36268/0 (socket says 172.21.15.26:36268) 2022-07-20T02:26:12.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d337fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 0x7f3d341530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 -- 172.21.15.26:0/1175784125 learned_addr learned my addr 172.21.15.26:0/1175784125 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d33fff700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d340ad8a0 0x7f3d340b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 -- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d340ad8a0 msgr2=0x7f3d340b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d340ad8a0 0x7f3d340b7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 -- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 msgr2=0x7f3d341530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 0x7f3d341530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d32ffd700 1 -- 172.21.15.26:0/1175784125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34152c40 con 0x7f3d340afbe0 2022-07-20T02:26:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.772+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 0x7f3d341530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d32ffd700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3d2400d000 tx=0x7f3d24001d50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d24011780 con 0x7f3d340afbe0 2022-07-20T02:26:12.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d240118e0 con 0x7f3d340afbe0 2022-07-20T02:26:12.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d340b9110 con 0x7f3d340afbe0 2022-07-20T02:26:12.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d24011c40 con 0x7f3d340afbe0 2022-07-20T02:26:12.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d340b7d00 con 0x7f3d340afbe0 2022-07-20T02:26:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d340adcb0 con 0x7f3d340afbe0 2022-07-20T02:26:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3d2401b030 con 0x7f3d340afbe0 2022-07-20T02:26:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 0x7f3d14055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f3d2406b820 con 0x7f3d340afbe0 2022-07-20T02:26:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.776+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 0x7f3d14055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.780+0000 7f3d337fe700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 0x7f3d14055fc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3d2801d000 tx=0x7f3d28019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:12.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.784+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d24042920 con 0x7f3d340afbe0 2022-07-20T02:26:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:12.912+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f3d340574e0 con 0x7f3d340afbe0 2022-07-20T02:26:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.492+0000 7f3d30ff9700 1 -- 172.21.15.26:0/1175784125 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v31) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3d24019030 con 0x7f3d340afbe0 2022-07-20T02:26:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 msgr2=0x7f3d14055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 0x7f3d14055fc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3d2801d000 tx=0x7f3d28019000).stop 2022-07-20T02:26:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 msgr2=0x7f3d34146610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:13.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3d2400d000 tx=0x7f3d24001d50).stop 2022-07-20T02:26:13.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 shutdown_connections 2022-07-20T02:26:13.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d340afbe0 0x7f3d34146610 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d14053b00 0x7f3d14055fc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d340ad8a0 0x7f3d340b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 --2- 172.21.15.26:0/1175784125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3405cfd0 0x7f3d341530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.496+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 >> 172.21.15.26:0/1175784125 conn(0x7f3d34066f70 msgr2=0x7f3d3413c010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:13.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.500+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 shutdown_connections 2022-07-20T02:26:13.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.500+0000 7f3d39fad700 1 -- 172.21.15.26:0/1175784125 wait complete. 2022-07-20T02:26:13.519 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:26:13.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:26:13.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 Processor -- start 2022-07-20T02:26:13.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 -- start start 2022-07-20T02:26:13.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 0x7f85a80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805f840 0x7f85a8144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f85a80a9ed0 0x7f85a81450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a80b27a0 con 0x7f85a80a9ed0 2022-07-20T02:26:13.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a805d3b0 con 0x7f85a805cfd0 2022-07-20T02:26:13.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85ae417700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a80b29d0 con 0x7f85a805f840 2022-07-20T02:26:13.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.636+0000 7f85adc16700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f85a80a9ed0 0x7f85a81450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f85a80a9ed0 0x7f85a81450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36278/0 (socket says 172.21.15.26:36278) 2022-07-20T02:26:13.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 -- 172.21.15.26:0/2548062614 learned_addr learned my addr 172.21.15.26:0/2548062614 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:13.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85acc14700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805f840 0x7f85a8144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ad415700 1 --2- 172.21.15.26:0/2548062614 >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 0x7f85a80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 -- 172.21.15.26:0/2548062614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805f840 msgr2=0x7f85a8144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:13.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805f840 0x7f85a8144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 -- 172.21.15.26:0/2548062614 >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 msgr2=0x7f85a80b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:13.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 --2- 172.21.15.26:0/2548062614 >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 0x7f85a80b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 -- 172.21.15.26:0/2548062614 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80b2c70 con 0x7f85a80a9ed0 2022-07-20T02:26:13.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ad415700 1 --2- 172.21.15.26:0/2548062614 >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 0x7f85a80b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:13.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85adc16700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a80a9ed0 0x7f85a81450c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f85a00032f0 tx=0x7f85a0009290).ready entity=mon.0 client_cookie=fe8d9d670c7f4daa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:13.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f8597fff700 1 -- 172.21.15.26:0/2548062614 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a000b870 con 0x7f85a80a9ed0 2022-07-20T02:26:13.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f8597fff700 1 -- 172.21.15.26:0/2548062614 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a000b9d0 con 0x7f85a80a9ed0 2022-07-20T02:26:13.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f8597fff700 1 -- 172.21.15.26:0/2548062614 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a000bb40 con 0x7f85a80a9ed0 2022-07-20T02:26:13.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 -- 172.21.15.26:0/2548062614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a80a9ed0 msgr2=0x7f85a81450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a80a9ed0 0x7f85a81450c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f85a00032f0 tx=0x7f85a0009290).stop 2022-07-20T02:26:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 -- 172.21.15.26:0/2548062614 shutdown_connections 2022-07-20T02:26:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a80a9ed0 0x7f85a81450c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 --2- 172.21.15.26:0/2548062614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805f840 0x7f85a8144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 --2- 172.21.15.26:0/2548062614 >> v2:172.21.15.120:3300/0 conn(0x7f85a805cfd0 0x7f85a80b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 -- 172.21.15.26:0/2548062614 >> 172.21.15.26:0/2548062614 conn(0x7f85a80b7090 msgr2=0x7f85a805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 -- 172.21.15.26:0/2548062614 shutdown_connections 2022-07-20T02:26:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.640+0000 7f85ae417700 1 -- 172.21.15.26:0/2548062614 wait complete. 2022-07-20T02:26:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 Processor -- start 2022-07-20T02:26:13.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- start start 2022-07-20T02:26:13.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805cfd0 0x7f85a80a9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85a80a9ed0 0x7f85a809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8150a40 con 0x7f85a805f840 2022-07-20T02:26:13.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8150c70 con 0x7f85a80a9ed0 2022-07-20T02:26:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8150f10 con 0x7f85a805cfd0 2022-07-20T02:26:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36282/0 (socket says 172.21.15.26:36282) 2022-07-20T02:26:13.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 -- 172.21.15.26:0/4108947848 learned_addr learned my addr 172.21.15.26:0/4108947848 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85adc16700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85a80a9ed0 0x7f85a809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ad415700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805cfd0 0x7f85a80a9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 -- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805cfd0 msgr2=0x7f85a80a9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:13.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805cfd0 0x7f85a80a9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 -- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85a80a9ed0 msgr2=0x7f85a809f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:13.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85a80a9ed0 0x7f85a809f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:13.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 -- 172.21.15.26:0/4108947848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0009530 con 0x7f85a805f840 2022-07-20T02:26:13.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85acc14700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f859800c000 tx=0x7f8598006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:13.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598010780 con 0x7f85a805f840 2022-07-20T02:26:13.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85980108e0 con 0x7f85a805f840 2022-07-20T02:26:13.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598010c40 con 0x7f85a805f840 2022-07-20T02:26:13.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a8064660 con 0x7f85a805f840 2022-07-20T02:26:13.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.644+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a8063800 con 0x7f85a805f840 2022-07-20T02:26:13.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a80647f0 con 0x7f85a805f840 2022-07-20T02:26:13.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f859801c460 con 0x7f85a805f840 2022-07-20T02:26:13.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85967fc700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 0x7f8588055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:13.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f859806c330 con 0x7f85a805f840 2022-07-20T02:26:13.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85ad415700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 0x7f8588055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:13.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.648+0000 7f85ad415700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 0x7f8588055e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f85a400c000 tx=0x7f85a4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:13.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.656+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85980431c0 con 0x7f85a805f840 2022-07-20T02:26:13.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:13.776+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f85a8144730 con 0x7f85a805f840 2022-07-20T02:26:14.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.496+0000 7f85967fc700 1 -- 172.21.15.26:0/4108947848 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v32) v1 ==== 125+0+0 (secure 0 0 0) 0x7f85980467e0 con 0x7f85a805f840 2022-07-20T02:26:14.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 msgr2=0x7f8588055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 0x7f8588055e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f85a400c000 tx=0x7f85a4008040).stop 2022-07-20T02:26:14.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 msgr2=0x7f85a80a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f859800c000 tx=0x7f8598006df0).stop 2022-07-20T02:26:14.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 shutdown_connections 2022-07-20T02:26:14.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85a805f840 0x7f85a80a9680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8588053950 0x7f8588055e10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85a80a9ed0 0x7f85a809f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 --2- 172.21.15.26:0/4108947848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85a805cfd0 0x7f85a80a9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 >> 172.21.15.26:0/4108947848 conn(0x7f85a80b7090 msgr2=0x7f85a80ac770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:14.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 shutdown_connections 2022-07-20T02:26:14.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.500+0000 7f85ae417700 1 -- 172.21.15.26:0/4108947848 wait complete. 2022-07-20T02:26:14.527 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:26:14.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:26:14.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.636+0000 7f63e7134700 1 Processor -- start 2022-07-20T02:26:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.636+0000 7f63e7134700 1 -- start start 2022-07-20T02:26:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63e01465b0 0x7f63e0066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00b1ae0 con 0x7f63e00b24c0 2022-07-20T02:26:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e00b2e20 con 0x7f63e01465b0 2022-07-20T02:26:14.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00b1d80 con 0x7f63e00672a0 2022-07-20T02:26:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6132700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63e01465b0 0x7f63e0066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e5931700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f63e00b24c0 0x7f63e00604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49780/0 (socket says 172.21.15.26:49780) 2022-07-20T02:26:14.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57806/0 (socket says 172.21.15.26:57806) 2022-07-20T02:26:14.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e5931700 1 -- 172.21.15.26:0/3701014846 learned_addr learned my addr 172.21.15.26:0/3701014846 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:14.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 -- 172.21.15.26:0/3701014846 >> v2:172.21.15.120:3300/0 conn(0x7f63e01465b0 msgr2=0x7f63e0066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:14.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 --2- 172.21.15.26:0/3701014846 >> v2:172.21.15.120:3300/0 conn(0x7f63e01465b0 0x7f63e0066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 -- 172.21.15.26:0/3701014846 >> v1:172.21.15.120:6789/0 conn(0x7f63e00b24c0 legacy=0x7f63e00604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 -- 172.21.15.26:0/3701014846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00b2020 con 0x7f63e00672a0 2022-07-20T02:26:14.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e6933700 1 --2- 172.21.15.26:0/3701014846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f63d80032f0 tx=0x7f63d80091d0).ready entity=mon.2 client_cookie=3d1f6505bbc607bb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e5130700 1 -- 172.21.15.26:0/3701014846 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63d800b6c0 con 0x7f63e00672a0 2022-07-20T02:26:14.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e5130700 1 -- 172.21.15.26:0/3701014846 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d800b820 con 0x7f63e00672a0 2022-07-20T02:26:14.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e5130700 1 -- 172.21.15.26:0/3701014846 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63d800b990 con 0x7f63e00672a0 2022-07-20T02:26:14.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- 172.21.15.26:0/3701014846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 msgr2=0x7f63e0146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 --2- 172.21.15.26:0/3701014846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f63d80032f0 tx=0x7f63d80091d0).stop 2022-07-20T02:26:14.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- 172.21.15.26:0/3701014846 shutdown_connections 2022-07-20T02:26:14.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 --2- 172.21.15.26:0/3701014846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e0146990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 --2- 172.21.15.26:0/3701014846 >> v2:172.21.15.120:3300/0 conn(0x7f63e01465b0 0x7f63e0066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- 172.21.15.26:0/3701014846 >> 172.21.15.26:0/3701014846 conn(0x7f63e00b3050 msgr2=0x7f63e00b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:14.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- 172.21.15.26:0/3701014846 shutdown_connections 2022-07-20T02:26:14.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.640+0000 7f63e7134700 1 -- 172.21.15.26:0/3701014846 wait complete. 2022-07-20T02:26:14.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 Processor -- start 2022-07-20T02:26:14.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 -- start start 2022-07-20T02:26:14.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e00acf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 0x7f63e01415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064300 con 0x7f63e00ac870 2022-07-20T02:26:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0152350 con 0x7f63e01465b0 2022-07-20T02:26:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e7134700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0152540 con 0x7f63e00672a0 2022-07-20T02:26:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e6933700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e6933700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36296/0 (socket says 172.21.15.26:36296) 2022-07-20T02:26:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e6933700 1 -- 172.21.15.26:0/2299409810 learned_addr learned my addr 172.21.15.26:0/2299409810 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e6132700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e00acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 0x7f63e01415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 -- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 msgr2=0x7f63e00acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e00acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 -- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 msgr2=0x7f63e0141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.644+0000 7f63e5931700 1 -- 172.21.15.26:0/2299409810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8009470 con 0x7f63e01465b0 2022-07-20T02:26:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e6933700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e5931700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 0x7f63e01415e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f63e0143ef0 tx=0x7f63dc007b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63dc0070d0 con 0x7f63e01465b0 2022-07-20T02:26:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc007230 con 0x7f63e01465b0 2022-07-20T02:26:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e7134700 1 -- 172.21.15.26:0/2299409810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01527e0 con 0x7f63e01465b0 2022-07-20T02:26:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63dc005610 con 0x7f63e01465b0 2022-07-20T02:26:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e7134700 1 -- 172.21.15.26:0/2299409810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e0152d20 con 0x7f63e01465b0 2022-07-20T02:26:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e7134700 1 -- 172.21.15.26:0/2299409810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e0152ee0 con 0x7f63e01465b0 2022-07-20T02:26:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f63dc010070 con 0x7f63e01465b0 2022-07-20T02:26:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 0x7f63c0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f63dc06b420 con 0x7f63e01465b0 2022-07-20T02:26:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e6132700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 0x7f63c0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.648+0000 7f63e6132700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 0x7f63c0055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f63d000b000 tx=0x7f63d0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.660+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63dc042260 con 0x7f63e01465b0 2022-07-20T02:26:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.828+0000 7f63e7134700 1 -- 172.21.15.26:0/2299409810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f63e0060bf0 con 0x7f63e01465b0 2022-07-20T02:26:14.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.840+0000 7f63cf7fe700 1 -- 172.21.15.26:0/2299409810 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 187+0+0 (secure 0 0 0) 0x7f63dc045880 con 0x7f63e01465b0 2022-07-20T02:26:14.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 msgr2=0x7f63c0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 0x7f63c0055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f63d000b000 tx=0x7f63d0006df0).stop 2022-07-20T02:26:14.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 msgr2=0x7f63e01415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 0x7f63e01415e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f63e0143ef0 tx=0x7f63dc007b80).stop 2022-07-20T02:26:14.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 shutdown_connections 2022-07-20T02:26:14.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63e00ac870 0x7f63e0141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63c0053a70 0x7f63c0055f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e01465b0 0x7f63e01415e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 --2- 172.21.15.26:0/2299409810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e00672a0 0x7f63e00acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 >> 172.21.15.26:0/2299409810 conn(0x7f63e00b3050 msgr2=0x7f63e013ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:14.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 shutdown_connections 2022-07-20T02:26:14.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.852+0000 7f63bf7fe700 1 -- 172.21.15.26:0/2299409810 wait complete. 2022-07-20T02:26:14.862 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 5 and data pool 6 2022-07-20T02:26:14.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:26:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 Processor -- start 2022-07-20T02:26:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 -- start start 2022-07-20T02:26:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 0x7fd8fc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0b62d0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc14aac0 con 0x7fd8fc147be0 2022-07-20T02:26:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9025ad700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc14ad60 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9015ab700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd8fc0b5ef0 0x7fd8fc1474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35592/0 (socket says 172.21.15.26:35592) 2022-07-20T02:26:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd9015ab700 1 -- 172.21.15.26:0/2795578613 learned_addr learned my addr 172.21.15.26:0/2795578613 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd900daa700 1 --2- 172.21.15.26:0/2795578613 >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 0x7fd8fc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.960+0000 7fd901dac700 1 --2- 172.21.15.26:0/2795578613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519521435 0 0) 0x7fd8fc0b62d0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e0003260 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd901dac700 1 -- 172.21.15.26:0/2795578613 >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 msgr2=0x7fd8fc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd901dac700 1 --2- 172.21.15.26:0/2795578613 >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 0x7fd8fc0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd901dac700 1 -- 172.21.15.26:0/2795578613 >> v1:172.21.15.26:6789/0 conn(0x7fd8fc0b5ef0 legacy=0x7fd8fc1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946048863 0 0) 0x7fd8e0003260 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd901dac700 1 -- 172.21.15.26:0/2795578613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc14b000 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd900daa700 1 --2- 172.21.15.26:0/2795578613 >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 0x7fd8fc0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd901dac700 1 --2- 172.21.15.26:0/2795578613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0af330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd8f4001b80 tx=0x7fd8f8006df0).ready entity=mon.2 client_cookie=79132059c9172c74 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f40019e0 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f8000e90 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd8f3fff700 1 -- 172.21.15.26:0/2795578613 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f8012460 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- 172.21.15.26:0/2795578613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 msgr2=0x7fd8fc0af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2795578613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0af330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd8f4001b80 tx=0x7fd8f8006df0).stop 2022-07-20T02:26:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- 172.21.15.26:0/2795578613 shutdown_connections 2022-07-20T02:26:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2795578613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0af330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2795578613 >> v2:172.21.15.120:3300/0 conn(0x7fd8fc147be0 0x7fd8fc0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- 172.21.15.26:0/2795578613 >> 172.21.15.26:0/2795578613 conn(0x7fd8fc060310 msgr2=0x7fd8fc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- 172.21.15.26:0/2795578613 shutdown_connections 2022-07-20T02:26:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- 172.21.15.26:0/2795578613 wait complete. 2022-07-20T02:26:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 Processor -- start 2022-07-20T02:26:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.964+0000 7fd9025ad700 1 -- start start 2022-07-20T02:26:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc147be0 0x7fd8fc0b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0aacc0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0aaf60 con 0x7fd8fc0a7fc0 2022-07-20T02:26:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0ab200 con 0x7fd8fc147be0 2022-07-20T02:26:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36322/0 (socket says 172.21.15.26:36322) 2022-07-20T02:26:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 -- 172.21.15.26:0/2236294736 learned_addr learned my addr 172.21.15.26:0/2236294736 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9015ab700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd901dac700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc147be0 0x7fd8fc0b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 -- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc147be0 msgr2=0x7fd8fc0b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc147be0 0x7fd8fc0b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 -- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8fc0a7fc0 msgr2=0x7fd8fc0abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 -- 172.21.15.26:0/2236294736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec00b180 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd900daa700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd8ec00c000 tx=0x7fd8ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec010780 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0108e0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f800b180 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec010c40 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.968+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc0ac230 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc14b2a0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd8ec01c460 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd8f27fc700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 0x7fd8e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd8ec06c620 con 0x7fd8fc0b5ef0 2022-07-20T02:26:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd9015ab700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 0x7fd8e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.972+0000 7fd9015ab700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 0x7fd8e4055f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd8f400d000 tx=0x7fd8f4001740).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:14.980+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8ec0431f0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:15.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.104+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8fc0574e0 con 0x7fd8fc0b5ef0 2022-07-20T02:26:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.108+0000 7fd8f27fc700 1 -- 172.21.15.26:0/2236294736 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16332 (secure 0 0 0) 0x7fd8ec046810 con 0x7fd8fc0b5ef0 2022-07-20T02:26:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:15.112 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:26:14.832810+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:26:12.967166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T02:26:13.835040+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:26:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 msgr2=0x7fd8e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 0x7fd8e4055f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd8f400d000 tx=0x7fd8f4001740).stop 2022-07-20T02:26:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 msgr2=0x7fd8fc15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd8ec00c000 tx=0x7fd8ec006df0).stop 2022-07-20T02:26:15.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 shutdown_connections 2022-07-20T02:26:15.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8fc0b5ef0 0x7fd8fc15a2e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd8e4053a70 0x7fd8e4055f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8fc147be0 0x7fd8fc0b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 --2- 172.21.15.26:0/2236294736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8fc0a7fc0 0x7fd8fc0abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 >> 172.21.15.26:0/2236294736 conn(0x7fd8fc060310 msgr2=0x7fd8fc05c870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:15.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 shutdown_connections 2022-07-20T02:26:15.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.112+0000 7fd9025ad700 1 -- 172.21.15.26:0/2236294736 wait complete. 2022-07-20T02:26:15.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:26:15.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 Processor -- start 2022-07-20T02:26:15.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 -- start start 2022-07-20T02:26:15.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc0c0b3260 0x7fdc0c0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c0609e0 0x7fdc0c0b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0b3640 con 0x7fdc0c097c50 2022-07-20T02:26:15.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c060e90 con 0x7fdc0c0b3260 2022-07-20T02:26:15.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c095300 con 0x7fdc0c0609e0 2022-07-20T02:26:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.240+0000 7fdc0a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc0c0b3260 0x7fdc0c0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc0c0b3260 0x7fdc0c0604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36328/0 (socket says 172.21.15.26:36328) 2022-07-20T02:26:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 -- 172.21.15.26:0/1306092714 learned_addr learned my addr 172.21.15.26:0/1306092714 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0ad9d700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c0609e0 0x7fdc0c0b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 -- 172.21.15.26:0/1306092714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c0609e0 msgr2=0x7fdc0c0b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c0609e0 0x7fdc0c0b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 -- 172.21.15.26:0/1306092714 >> v1:172.21.15.120:6789/0 conn(0x7fdc0c097c50 legacy=0x7fdc0c0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 -- 172.21.15.26:0/1306092714 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0955a0 con 0x7fdc0c0b3260 2022-07-20T02:26:15.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0a59c700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0604f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fdbfc002f70 tx=0x7fdbfc006eb0).ready entity=mon.0 client_cookie=a6e48307ad1d6f0c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc09d9b700 1 -- 172.21.15.26:0/1306092714 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbfc00d570 con 0x7fdc0c0b3260 2022-07-20T02:26:15.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc09d9b700 1 -- 172.21.15.26:0/1306092714 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc000e90 con 0x7fdc0c0b3260 2022-07-20T02:26:15.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc09d9b700 1 -- 172.21.15.26:0/1306092714 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbfc00d900 con 0x7fdc0c0b3260 2022-07-20T02:26:15.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- 172.21.15.26:0/1306092714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0b3260 msgr2=0x7fdc0c0604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0604f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fdbfc002f70 tx=0x7fdbfc006eb0).stop 2022-07-20T02:26:15.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- 172.21.15.26:0/1306092714 shutdown_connections 2022-07-20T02:26:15.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c0609e0 0x7fdc0c0b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/1306092714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0604f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- 172.21.15.26:0/1306092714 >> 172.21.15.26:0/1306092714 conn(0x7fdc0c0b67f0 msgr2=0x7fdc0c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:15.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- 172.21.15.26:0/1306092714 shutdown_connections 2022-07-20T02:26:15.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- 172.21.15.26:0/1306092714 wait complete. 2022-07-20T02:26:15.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 Processor -- start 2022-07-20T02:26:15.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.244+0000 7fdc0b59e700 1 -- start start 2022-07-20T02:26:15.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 0x7fdc0c0640d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0645c0 con 0x7fdc0c0609e0 2022-07-20T02:26:15.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0ad5c0 con 0x7fdc0c0b3260 2022-07-20T02:26:15.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c150f90 con 0x7fdc0c097c50 2022-07-20T02:26:15.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36332/0 (socket says 172.21.15.26:36332) 2022-07-20T02:26:15.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 -- 172.21.15.26:0/3675932412 learned_addr learned my addr 172.21.15.26:0/3675932412 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc03fff700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 0x7fdc0c0640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0ad9d700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 -- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 msgr2=0x7fdc0c0640d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 0x7fdc0c0640d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 -- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc0c0b3260 msgr2=0x7fdc0c0aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 -- 172.21.15.26:0/3675932412 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc00b1e0 con 0x7fdc0c0609e0 2022-07-20T02:26:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc03fff700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 0x7fdc0c0640d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0a59c700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fdbfc002f70 tx=0x7fdbfc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbfc0119d0 con 0x7fdc0c0609e0 2022-07-20T02:26:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc011b30 con 0x7fdc0c0609e0 2022-07-20T02:26:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbfc011e90 con 0x7fdc0c0609e0 2022-07-20T02:26:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c151230 con 0x7fdc0c0609e0 2022-07-20T02:26:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.248+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0ad120 con 0x7fdc0c0609e0 2022-07-20T02:26:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c0ad730 con 0x7fdc0c0609e0 2022-07-20T02:26:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdbfc025030 con 0x7fdc0c0609e0 2022-07-20T02:26:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc037fe700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 0x7fdbf0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdbfc06d420 con 0x7fdc0c0609e0 2022-07-20T02:26:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc03fff700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 0x7fdbf0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.252+0000 7fdc03fff700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 0x7fdbf0055e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdbf400b000 tx=0x7fdbf4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.260+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc0c0ad730 con 0x7fdc0c0609e0 2022-07-20T02:26:15.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.380+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc0c0ad730 con 0x7fdc0c0609e0 2022-07-20T02:26:15.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.380+0000 7fdc037fe700 1 -- 172.21.15.26:0/3675932412 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16332 (secure 0 0 0) 0x7fdc0c0ad730 con 0x7fdc0c0609e0 2022-07-20T02:26:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:15.384 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:26:14.832810+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:26:12.967166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T02:26:13.835040+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:26:15.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 msgr2=0x7fdbf0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 0x7fdbf0055e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdbf400b000 tx=0x7fdbf4009040).stop 2022-07-20T02:26:15.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 msgr2=0x7fdc0c063be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fdbfc002f70 tx=0x7fdbfc015000).stop 2022-07-20T02:26:15.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 shutdown_connections 2022-07-20T02:26:15.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbf0053950 0x7fdbf0055e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc0c0b3260 0x7fdc0c0aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0c097c50 0x7fdc0c0640d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 --2- 172.21.15.26:0/3675932412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc0c0609e0 0x7fdc0c063be0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 >> 172.21.15.26:0/3675932412 conn(0x7fdc0c0b67f0 msgr2=0x7fdc0c144b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:15.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 shutdown_connections 2022-07-20T02:26:15.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.384+0000 7fdc0b59e700 1 -- 172.21.15.26:0/3675932412 wait complete. 2022-07-20T02:26:15.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:26:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 Processor -- start 2022-07-20T02:26:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 -- start start 2022-07-20T02:26:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc4e00b2f20 0x7fc4e00b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e0147080 con 0x7fc4e0067440 2022-07-20T02:26:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e01472b0 con 0x7fc4e00957f0 2022-07-20T02:26:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4e5282700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e0143000 con 0x7fc4e00b2f20 2022-07-20T02:26:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4df7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc4e0067440 0x7fc4e00950e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35622/0 (socket says 172.21.15.26:35622) 2022-07-20T02:26:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4df7fe700 1 -- 172.21.15.26:0/760761937 learned_addr learned my addr 172.21.15.26:0/760761937 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4dffff700 1 --2- 172.21.15.26:0/760761937 >> v2:172.21.15.26:3300/0 conn(0x7fc4e00b2f20 0x7fc4e00b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034184340 0 0) 0x7fc4e0147080 con 0x7fc4e0067440 2022-07-20T02:26:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4cc003260 con 0x7fc4e0067440 2022-07-20T02:26:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375467183 0 0) 0x7fc4e01472b0 con 0x7fc4e00957f0 2022-07-20T02:26:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e0147080 con 0x7fc4e00957f0 2022-07-20T02:26:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4dffff700 1 -- 172.21.15.26:0/760761937 >> v1:172.21.15.120:6789/0 conn(0x7fc4e00957f0 legacy=0x7fc4e01433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.512+0000 7fc4dffff700 1 -- 172.21.15.26:0/760761937 >> v1:172.21.15.26:6789/0 conn(0x7fc4e0067440 legacy=0x7fc4e00950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4dffff700 1 -- 172.21.15.26:0/760761937 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00a9920 con 0x7fc4e00b2f20 2022-07-20T02:26:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4dffff700 1 --2- 172.21.15.26:0/760761937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e00b3300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc4d4002f70 tx=0x7fc4d4006df0).ready entity=mon.0 client_cookie=79fbb3358f77f355 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d400d540 con 0x7fc4e00b2f20 2022-07-20T02:26:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d4000e90 con 0x7fc4e00b2f20 2022-07-20T02:26:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4deffd700 1 -- 172.21.15.26:0/760761937 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d400d890 con 0x7fc4e00b2f20 2022-07-20T02:26:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 -- 172.21.15.26:0/760761937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 msgr2=0x7fc4e00b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 --2- 172.21.15.26:0/760761937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e00b3300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc4d4002f70 tx=0x7fc4d4006df0).stop 2022-07-20T02:26:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 -- 172.21.15.26:0/760761937 shutdown_connections 2022-07-20T02:26:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 --2- 172.21.15.26:0/760761937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e00b3300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 -- 172.21.15.26:0/760761937 >> 172.21.15.26:0/760761937 conn(0x7fc4e00b67f0 msgr2=0x7fc4e00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 -- 172.21.15.26:0/760761937 shutdown_connections 2022-07-20T02:26:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.516+0000 7fc4e5282700 1 -- 172.21.15.26:0/760761937 wait complete. 2022-07-20T02:26:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 Processor -- start 2022-07-20T02:26:15.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 -- start start 2022-07-20T02:26:15.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e0067440 0x7fc4e00ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00957f0 0x7fc4e009b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e005f9c0 con 0x7fc4e00b2f20 2022-07-20T02:26:15.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e005fc60 con 0x7fc4e0067440 2022-07-20T02:26:15.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e005ff00 con 0x7fc4e00957f0 2022-07-20T02:26:15.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36350/0 (socket says 172.21.15.26:36350) 2022-07-20T02:26:15.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 -- 172.21.15.26:0/4170688259 learned_addr learned my addr 172.21.15.26:0/4170688259 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:15.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4dffff700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e0067440 0x7fc4e00ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4df7fe700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00957f0 0x7fc4e009b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 -- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00957f0 msgr2=0x7fc4e009b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00957f0 0x7fc4e009b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 -- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e0067440 msgr2=0x7fc4e00ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e0067440 0x7fc4e00ac680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 -- 172.21.15.26:0/4170688259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d400b180 con 0x7fc4e00b2f20 2022-07-20T02:26:15.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e4a81700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc4d8001110 tx=0x7fc4d8007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d8012400 con 0x7fc4e00b2f20 2022-07-20T02:26:15.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d8012560 con 0x7fc4e00b2f20 2022-07-20T02:26:15.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e0152270 con 0x7fc4e00b2f20 2022-07-20T02:26:15.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.520+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d80128c0 con 0x7fc4e00b2f20 2022-07-20T02:26:15.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e0098de0 con 0x7fc4e00b2f20 2022-07-20T02:26:15.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0152400 con 0x7fc4e00b2f20 2022-07-20T02:26:15.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc4d8010070 con 0x7fc4e00b2f20 2022-07-20T02:26:15.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4dd7fa700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 0x7fc4c0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc4d806b180 con 0x7fc4e00b2f20 2022-07-20T02:26:15.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.524+0000 7fc4dffff700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 0x7fc4c0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.528+0000 7fc4dffff700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 0x7fc4c0055e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc4d401a000 tx=0x7fc4d4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.532+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4d8041f40 con 0x7fc4e00b2f20 2022-07-20T02:26:15.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.696+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc4e00a95a0 con 0x7fc4e00b2f20 2022-07-20T02:26:15.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.848+0000 7fc4dd7fa700 1 -- 172.21.15.26:0/4170688259 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc4d8045560 con 0x7fc4e00b2f20 2022-07-20T02:26:15.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 msgr2=0x7fc4c0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 0x7fc4c0055e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc4d401a000 tx=0x7fc4d4014000).stop 2022-07-20T02:26:15.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 msgr2=0x7fc4e0152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc4d8001110 tx=0x7fc4d8007ae0).stop 2022-07-20T02:26:15.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 shutdown_connections 2022-07-20T02:26:15.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4e00b2f20 0x7fc4e0152710 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4c0053980 0x7fc4c0055e40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4e00957f0 0x7fc4e009b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 --2- 172.21.15.26:0/4170688259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4e0067440 0x7fc4e00ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 >> 172.21.15.26:0/4170688259 conn(0x7fc4e00b67f0 msgr2=0x7fc4e0060630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 shutdown_connections 2022-07-20T02:26:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.852+0000 7fc4e5282700 1 -- 172.21.15.26:0/4170688259 wait complete. 2022-07-20T02:26:15.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 Processor -- start 2022-07-20T02:26:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- start start 2022-07-20T02:26:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2d54144660 0x7f2d54144a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d5405c580 0x7f2d54095e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 0x7f2d5413e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d540b1330 con 0x7f2d54144660 2022-07-20T02:26:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d54145280 con 0x7f2d540b1a30 2022-07-20T02:26:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54096300 con 0x7f2d5405c580 2022-07-20T02:26:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2d54144660 0x7f2d54144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2d54144660 0x7f2d54144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36354/0 (socket says 172.21.15.26:36354) 2022-07-20T02:26:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 -- 172.21.15.26:0/1645399323 learned_addr learned my addr 172.21.15.26:0/1645399323 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5aaef700 1 --2- 172.21.15.26:0/1645399323 >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 0x7f2d5413e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d59aed700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d5405c580 0x7f2d54095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 -- 172.21.15.26:0/1645399323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d5405c580 msgr2=0x7f2d54095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d5405c580 0x7f2d54095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 -- 172.21.15.26:0/1645399323 >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 msgr2=0x7f2d5413e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 --2- 172.21.15.26:0/1645399323 >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 0x7f2d5413e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 -- 172.21.15.26:0/1645399323 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d54144fe0 con 0x7f2d54144660 2022-07-20T02:26:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5aaef700 1 --2- 172.21.15.26:0/1645399323 >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 0x7f2d5413e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5a2ee700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d54144a40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f2d500032f0 tx=0x7f2d50009290).ready entity=mon.0 client_cookie=945f19d70db8e0e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d592ec700 1 -- 172.21.15.26:0/1645399323 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d5000b870 con 0x7f2d54144660 2022-07-20T02:26:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d592ec700 1 -- 172.21.15.26:0/1645399323 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5000b9d0 con 0x7f2d54144660 2022-07-20T02:26:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d592ec700 1 -- 172.21.15.26:0/1645399323 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d5000bb40 con 0x7f2d54144660 2022-07-20T02:26:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- 172.21.15.26:0/1645399323 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 msgr2=0x7f2d54144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d54144a40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f2d500032f0 tx=0x7f2d50009290).stop 2022-07-20T02:26:15.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- 172.21.15.26:0/1645399323 shutdown_connections 2022-07-20T02:26:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- 172.21.15.26:0/1645399323 >> v2:172.21.15.120:3300/0 conn(0x7f2d540b1a30 0x7f2d5413e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d5405c580 0x7f2d54095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 --2- 172.21.15.26:0/1645399323 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d54144a40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- 172.21.15.26:0/1645399323 >> 172.21.15.26:0/1645399323 conn(0x7f2d540b5e70 msgr2=0x7f2d540b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- 172.21.15.26:0/1645399323 shutdown_connections 2022-07-20T02:26:16.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.980+0000 7f2d5c552700 1 -- 172.21.15.26:0/1645399323 wait complete. 2022-07-20T02:26:16.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:16.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 Processor -- start 2022-07-20T02:26:16.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 -- start start 2022-07-20T02:26:16.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d5405c580 0x7f2d540651f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d540b1a30 0x7f2d5409a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54151220 con 0x7f2d54144660 2022-07-20T02:26:16.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54099f00 con 0x7f2d5405c580 2022-07-20T02:26:16.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5c552700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5409a160 con 0x7f2d540b1a30 2022-07-20T02:26:16.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5aaef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5aaef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36364/0 (socket says 172.21.15.26:36364) 2022-07-20T02:26:16.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5a2ee700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d5405c580 0x7f2d540651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d5aaef700 1 -- 172.21.15.26:0/327747920 learned_addr learned my addr 172.21.15.26:0/327747920 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.984+0000 7f2d59aed700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d540b1a30 0x7f2d5409a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 -- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d540b1a30 msgr2=0x7f2d5409a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d540b1a30 0x7f2d5409a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 -- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d5405c580 msgr2=0x7f2d540651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d5405c580 0x7f2d540651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 -- 172.21.15.26:0/327747920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d50009530 con 0x7f2d54144660 2022-07-20T02:26:16.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5aaef700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f2d4c006940 tx=0x7f2d4c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d4c011780 con 0x7f2d54144660 2022-07-20T02:26:16.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c0118e0 con 0x7f2d54144660 2022-07-20T02:26:16.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5409a400 con 0x7f2d54144660 2022-07-20T02:26:16.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d4c011c40 con 0x7f2d54144660 2022-07-20T02:26:16.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d540952f0 con 0x7f2d54144660 2022-07-20T02:26:16.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d540955a0 con 0x7f2d54144660 2022-07-20T02:26:16.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d4c012420 con 0x7f2d54144660 2022-07-20T02:26:16.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 0x7f2d34055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2d4c06beb0 con 0x7f2d54144660 2022-07-20T02:26:16.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.988+0000 7f2d5a2ee700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 0x7f2d34055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.992+0000 7f2d5a2ee700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 0x7f2d34055e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2d5001a000 tx=0x7f2d50016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:15.996+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d4c042b00 con 0x7f2d54144660 2022-07-20T02:26:16.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:16.054 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:16.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d540b1590 con 0x7f2d54144660 2022-07-20T02:26:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d437fe700 1 -- 172.21.15.26:0/327747920 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7f2d4c046120 con 0x7f2d54144660 2022-07-20T02:26:16.169 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:16.169 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 msgr2=0x7f2d34055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 0x7f2d34055e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2d5001a000 tx=0x7f2d50016000).stop 2022-07-20T02:26:16.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 msgr2=0x7f2d5409ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f2d4c006940 tx=0x7f2d4c008040).stop 2022-07-20T02:26:16.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 shutdown_connections 2022-07-20T02:26:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d54144660 0x7f2d5409ad90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d34053980 0x7f2d34055e40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d540b1a30 0x7f2d5409a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.164+0000 7f2d5c552700 1 --2- 172.21.15.26:0/327747920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d5405c580 0x7f2d540651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.168+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 >> 172.21.15.26:0/327747920 conn(0x7f2d540b5e70 msgr2=0x7f2d540b54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.168+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 shutdown_connections 2022-07-20T02:26:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.168+0000 7f2d5c552700 1 -- 172.21.15.26:0/327747920 wait complete. 2022-07-20T02:26:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:16.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:26:16.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 Processor -- start 2022-07-20T02:26:16.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 -- start start 2022-07-20T02:26:16.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 0x7f987c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f987c060850 0x7f987c0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c0610e0 con 0x7f987c0ac260 2022-07-20T02:26:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c0b2ea0 con 0x7f987c060850 2022-07-20T02:26:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f9882008700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c0a8e20 con 0x7f987c0b2ac0 2022-07-20T02:26:16.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f987bfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f987c0ac260 0x7f987c1468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35652/0 (socket says 172.21.15.26:35652) 2022-07-20T02:26:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f987bfff700 1 -- 172.21.15.26:0/1751663931 learned_addr learned my addr 172.21.15.26:0/1751663931 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f987affd700 1 --2- 172.21.15.26:0/1751663931 >> v2:172.21.15.26:3300/0 conn(0x7f987c060850 0x7f987c0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.284+0000 7f987b7fe700 1 --2- 172.21.15.26:0/1751663931 >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 0x7f987c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987a7fc700 1 -- 172.21.15.26:0/1751663931 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572477825 0 0) 0x7f987c0610e0 con 0x7f987c0ac260 2022-07-20T02:26:16.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987a7fc700 1 -- 172.21.15.26:0/1751663931 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9868003260 con 0x7f987c0ac260 2022-07-20T02:26:16.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987affd700 1 -- 172.21.15.26:0/1751663931 >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 msgr2=0x7f987c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:16.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987affd700 1 --2- 172.21.15.26:0/1751663931 >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 0x7f987c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987affd700 1 -- 172.21.15.26:0/1751663931 >> v1:172.21.15.26:6789/0 conn(0x7f987c0ac260 legacy=0x7f987c1468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987affd700 1 -- 172.21.15.26:0/1751663931 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0a95e0 con 0x7f987c060850 2022-07-20T02:26:16.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987b7fe700 1 --2- 172.21.15.26:0/1751663931 >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 0x7f987c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:16.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987affd700 1 --2- 172.21.15.26:0/1751663931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c060850 0x7f987c0abd70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9870002f70 tx=0x7f9870006df0).ready entity=mon.0 client_cookie=281246956f67941e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987a7fc700 1 -- 172.21.15.26:0/1751663931 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f987000d570 con 0x7f987c060850 2022-07-20T02:26:16.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987a7fc700 1 -- 172.21.15.26:0/1751663931 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9870000e90 con 0x7f987c060850 2022-07-20T02:26:16.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f987a7fc700 1 -- 172.21.15.26:0/1751663931 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f987000d8c0 con 0x7f987c060850 2022-07-20T02:26:16.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 -- 172.21.15.26:0/1751663931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c060850 msgr2=0x7f987c0abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 --2- 172.21.15.26:0/1751663931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c060850 0x7f987c0abd70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9870002f70 tx=0x7f9870006df0).stop 2022-07-20T02:26:16.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 -- 172.21.15.26:0/1751663931 shutdown_connections 2022-07-20T02:26:16.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 --2- 172.21.15.26:0/1751663931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c060850 0x7f987c0abd70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 --2- 172.21.15.26:0/1751663931 >> v2:172.21.15.120:3300/0 conn(0x7f987c0b2ac0 0x7f987c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 -- 172.21.15.26:0/1751663931 >> 172.21.15.26:0/1751663931 conn(0x7f987c0b60a0 msgr2=0x7f987c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 -- 172.21.15.26:0/1751663931 shutdown_connections 2022-07-20T02:26:16.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.288+0000 7f9882008700 1 -- 172.21.15.26:0/1751663931 wait complete. 2022-07-20T02:26:16.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 Processor -- start 2022-07-20T02:26:16.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 -- start start 2022-07-20T02:26:16.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c060850 0x7f987c152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c0ac260 0x7f987c0ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c155ac0 con 0x7f987c0b2ac0 2022-07-20T02:26:16.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0b4d00 con 0x7f987c060850 2022-07-20T02:26:16.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f9882008700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c155cf0 con 0x7f987c0ac260 2022-07-20T02:26:16.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36378/0 (socket says 172.21.15.26:36378) 2022-07-20T02:26:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 -- 172.21.15.26:0/3637531256 learned_addr learned my addr 172.21.15.26:0/3637531256 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c060850 0x7f987c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987affd700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c0ac260 0x7f987c0ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 -- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c0ac260 msgr2=0x7f987c0ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c0ac260 0x7f987c0ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 -- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c060850 msgr2=0x7f987c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c060850 0x7f987c152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 -- 172.21.15.26:0/3637531256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987000b180 con 0x7f987c0b2ac0 2022-07-20T02:26:16.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.292+0000 7f987bfff700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9874001490 tx=0x7f9874007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9874005ba0 con 0x7f987c0b2ac0 2022-07-20T02:26:16.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9874005d00 con 0x7f987c0b2ac0 2022-07-20T02:26:16.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c0b5280 con 0x7f987c0b2ac0 2022-07-20T02:26:16.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98740072c0 con 0x7f987c0b2ac0 2022-07-20T02:26:16.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0a2650 con 0x7f987c0b2ac0 2022-07-20T02:26:16.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c155eb0 con 0x7f987c0b2ac0 2022-07-20T02:26:16.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9874010070 con 0x7f987c0b2ac0 2022-07-20T02:26:16.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.296+0000 7f9878ff9700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 0x7f985c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.300+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f987406b8c0 con 0x7f987c0b2ac0 2022-07-20T02:26:16.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.300+0000 7f987b7fe700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 0x7f985c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.300+0000 7f987b7fe700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 0x7f985c055e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f986c00c000 tx=0x7f986c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.308+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9874042680 con 0x7f987c0b2ac0 2022-07-20T02:26:16.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.436+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f987c060d60 con 0x7f987c0b2ac0 2022-07-20T02:26:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.436+0000 7f9878ff9700 1 -- 172.21.15.26:0/3637531256 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f9874018070 con 0x7f987c0b2ac0 2022-07-20T02:26:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:16.441 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":34,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:26:15.835998+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:26:12.967166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T02:26:13.835040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:26:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 msgr2=0x7f985c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 0x7f985c055e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f986c00c000 tx=0x7f986c008040).stop 2022-07-20T02:26:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 msgr2=0x7f987c156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9874001490 tx=0x7f9874007e20).stop 2022-07-20T02:26:16.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 shutdown_connections 2022-07-20T02:26:16.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f987c0b2ac0 0x7f987c156190 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f985c053950 0x7f985c055e10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c0ac260 0x7f987c0ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 --2- 172.21.15.26:0/3637531256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c060850 0x7f987c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 >> 172.21.15.26:0/3637531256 conn(0x7f987c0b60a0 msgr2=0x7f987c143010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 shutdown_connections 2022-07-20T02:26:16.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.440+0000 7f9882008700 1 -- 172.21.15.26:0/3637531256 wait complete. 2022-07-20T02:26:16.459 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:26:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 Processor -- start 2022-07-20T02:26:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 -- start start 2022-07-20T02:26:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40add30 0x7fbfc40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfc4060950 0x7fbfc4060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc40aa1e0 con 0x7fbfc4061280 2022-07-20T02:26:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40ae110 con 0x7fbfc4060950 2022-07-20T02:26:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfca27f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40a9f40 con 0x7fbfc40add30 2022-07-20T02:26:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfc4060950 0x7fbfc4060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfc4060950 0x7fbfc4060d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36386/0 (socket says 172.21.15.26:36386) 2022-07-20T02:26:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 -- 172.21.15.26:0/2163962711 learned_addr learned my addr 172.21.15.26:0/2163962711 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc927d700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40add30 0x7fbfc40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 -- 172.21.15.26:0/2163962711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40add30 msgr2=0x7fbfc40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40add30 0x7fbfc40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 -- 172.21.15.26:0/2163962711 >> v1:172.21.15.120:6789/0 conn(0x7fbfc4061280 legacy=0x7fbfc4143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfbbfff700 1 -- 172.21.15.26:0/2163962711 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977958601 0 0) 0x7fbfc40aa1e0 con 0x7fbfc4061280 2022-07-20T02:26:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.588+0000 7fbfc8a7c700 1 -- 172.21.15.26:0/2163962711 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40b1110 con 0x7fbfc4060950 2022-07-20T02:26:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfc8a7c700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc4060950 0x7fbfc4060d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fbfb4002f70 tx=0x7fbfb4006eb0).ready entity=mon.0 client_cookie=f4709ab8c7bd3159 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfbbfff700 1 -- 172.21.15.26:0/2163962711 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb400d570 con 0x7fbfc4060950 2022-07-20T02:26:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfbbfff700 1 -- 172.21.15.26:0/2163962711 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb4000e90 con 0x7fbfc4060950 2022-07-20T02:26:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfbbfff700 1 -- 172.21.15.26:0/2163962711 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb400d8c0 con 0x7fbfc4060950 2022-07-20T02:26:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- 172.21.15.26:0/2163962711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc4060950 msgr2=0x7fbfc4060d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc4060950 0x7fbfc4060d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fbfb4002f70 tx=0x7fbfb4006eb0).stop 2022-07-20T02:26:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- 172.21.15.26:0/2163962711 shutdown_connections 2022-07-20T02:26:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc4060950 0x7fbfc4060d90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 --2- 172.21.15.26:0/2163962711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc40add30 0x7fbfc40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- 172.21.15.26:0/2163962711 >> 172.21.15.26:0/2163962711 conn(0x7fbfc40b3050 msgr2=0x7fbfc40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- 172.21.15.26:0/2163962711 shutdown_connections 2022-07-20T02:26:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- 172.21.15.26:0/2163962711 wait complete. 2022-07-20T02:26:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 Processor -- start 2022-07-20T02:26:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.592+0000 7fbfca27f700 1 -- start start 2022-07-20T02:26:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc4060950 0x7fbfc40aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 0x7fbfc414e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4152050 con 0x7fbfc40add30 2022-07-20T02:26:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc41522f0 con 0x7fbfc4060950 2022-07-20T02:26:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc413d3c0 con 0x7fbfc4061280 2022-07-20T02:26:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36392/0 (socket says 172.21.15.26:36392) 2022-07-20T02:26:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 -- 172.21.15.26:0/4143699924 learned_addr learned my addr 172.21.15.26:0/4143699924 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc927d700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc4060950 0x7fbfc40aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc8a7c700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 0x7fbfc414e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 -- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 msgr2=0x7fbfc414e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 0x7fbfc414e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 -- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc4060950 msgr2=0x7fbfc40aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc4060950 0x7fbfc40aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 -- 172.21.15.26:0/4143699924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb400b1e0 con 0x7fbfc40add30 2022-07-20T02:26:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc8a7c700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 0x7fbfc414e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:16.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfc9a7e700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fbfbc000f40 tx=0x7fbfbc007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfbc00e4a0 con 0x7fbfc40add30 2022-07-20T02:26:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc00e600 con 0x7fbfc40add30 2022-07-20T02:26:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc413d660 con 0x7fbfc40add30 2022-07-20T02:26:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfbc00e960 con 0x7fbfc40add30 2022-07-20T02:26:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc413dd60 con 0x7fbfc40add30 2022-07-20T02:26:16.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.596+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc413d7f0 con 0x7fbfc40add30 2022-07-20T02:26:16.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.600+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbfbc019030 con 0x7fbfc40add30 2022-07-20T02:26:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.600+0000 7fbfba7fc700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 0x7fbfac055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.600+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbfbc06ba60 con 0x7fbfc40add30 2022-07-20T02:26:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.600+0000 7fbfc927d700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 0x7fbfac055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.600+0000 7fbfc927d700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 0x7fbfac055e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbfc000d000 tx=0x7fbfc000b060).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.608+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfbc042820 con 0x7fbfc40add30 2022-07-20T02:26:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.772+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbfc41524b0 con 0x7fbfc40add30 2022-07-20T02:26:16.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.776+0000 7fbfba7fc700 1 -- 172.21.15.26:0/4143699924 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbfbc045e40 con 0x7fbfc40add30 2022-07-20T02:26:16.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 msgr2=0x7fbfac055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 0x7fbfac055e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbfc000d000 tx=0x7fbfc000b060).stop 2022-07-20T02:26:16.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 msgr2=0x7fbfc405bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fbfbc000f40 tx=0x7fbfbc007e00).stop 2022-07-20T02:26:16.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 shutdown_connections 2022-07-20T02:26:16.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfc40add30 0x7fbfc405bf30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfac053950 0x7fbfac055e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfc4061280 0x7fbfc414e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 --2- 172.21.15.26:0/4143699924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfc4060950 0x7fbfc40aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 >> 172.21.15.26:0/4143699924 conn(0x7fbfc40b3050 msgr2=0x7fbfc40b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 shutdown_connections 2022-07-20T02:26:16.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.780+0000 7fbfca27f700 1 -- 172.21.15.26:0/4143699924 wait complete. 2022-07-20T02:26:16.792 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:26:16.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:26:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 Processor -- start 2022-07-20T02:26:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 -- start start 2022-07-20T02:26:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4148061860 0x7f4148067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414805ff70 con 0x7f41480b3260 2022-07-20T02:26:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41480606e0 con 0x7f41480980d0 2022-07-20T02:26:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414e1dd700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41480b3640 con 0x7f4148061860 2022-07-20T02:26:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414d1db700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41480b3260 0x7f41480979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35676/0 (socket says 172.21.15.26:35676) 2022-07-20T02:26:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414d1db700 1 -- 172.21.15.26:0/1089935695 learned_addr learned my addr 172.21.15.26:0/1089935695 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684106918 0 0) 0x7f414805ff70 con 0x7f41480b3260 2022-07-20T02:26:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f414c9da700 1 --2- 172.21.15.26:0/1089935695 >> v2:172.21.15.120:3300/0 conn(0x7f4148061860 0x7f4148067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4130003260 con 0x7f41480b3260 2022-07-20T02:26:16.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402112262 0 0) 0x7f4130003260 con 0x7f41480b3260 2022-07-20T02:26:16.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.916+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 >> v2:172.21.15.120:3300/0 conn(0x7f4148061860 msgr2=0x7f4148067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:16.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 --2- 172.21.15.26:0/1089935695 >> v2:172.21.15.120:3300/0 conn(0x7f4148061860 0x7f4148067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 >> v1:172.21.15.120:6789/0 conn(0x7f41480980d0 legacy=0x7f4148142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480601a0 con 0x7f41480b3260 2022-07-20T02:26:16.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f4144001660 con 0x7f41480b3260 2022-07-20T02:26:16.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 >> v1:172.21.15.26:6789/0 conn(0x7f41480b3260 legacy=0x7f41480979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130003750 0x7f4130003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4130004500 0x7f4130004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 0x7f413000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480b3640 con 0x7f4130004e70 2022-07-20T02:26:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413000cd80 con 0x7f4130003750 2022-07-20T02:26:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413000cf40 con 0x7f4130004500 2022-07-20T02:26:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 0x7f413000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d1db700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130003750 0x7f4130003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414c9da700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4130004500 0x7f4130004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 -- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4130004500 msgr2=0x7f4130004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:16.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4130004500 0x7f4130004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 -- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130003750 msgr2=0x7f4130003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:16.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130003750 0x7f4130003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 -- 172.21.15.26:0/1089935695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130003260 con 0x7f4130004e70 2022-07-20T02:26:16.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 -- 172.21.15.26:0/1089935695 reap_dead start 2022-07-20T02:26:16.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414d9dc700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 0x7f413000b750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f41400013a0 tx=0x7f4140007d60).ready entity=mon.0 client_cookie=aee368d06c61be5a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41400016c0 con 0x7f4130004e70 2022-07-20T02:26:16.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4140006890 con 0x7f4130004e70 2022-07-20T02:26:16.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f4137fff700 1 -- 172.21.15.26:0/1089935695 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4140006bf0 con 0x7f4130004e70 2022-07-20T02:26:16.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 msgr2=0x7f413000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 0x7f413000b750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f41400013a0 tx=0x7f4140007d60).stop 2022-07-20T02:26:16.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- 172.21.15.26:0/1089935695 shutdown_connections 2022-07-20T02:26:16.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 --2- 172.21.15.26:0/1089935695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4130004e70 0x7f413000b750 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- 172.21.15.26:0/1089935695 >> 172.21.15.26:0/1089935695 conn(0x7f41480b67f0 msgr2=0x7f41480b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:16.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- 172.21.15.26:0/1089935695 shutdown_connections 2022-07-20T02:26:16.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- 172.21.15.26:0/1089935695 wait complete. 2022-07-20T02:26:16.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 Processor -- start 2022-07-20T02:26:16.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.920+0000 7f414e1dd700 1 -- start start 2022-07-20T02:26:16.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130004e70 0x7f41480b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f414809ac10 0x7f414809b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480b5a50 con 0x7f4148098de0 2022-07-20T02:26:16.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414809a9d0 con 0x7f4130004e70 2022-07-20T02:26:16.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41480b5300 con 0x7f414809ac10 2022-07-20T02:26:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36412/0 (socket says 172.21.15.26:36412) 2022-07-20T02:26:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 -- 172.21.15.26:0/3132557026 learned_addr learned my addr 172.21.15.26:0/3132557026 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414c9da700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f414809ac10 0x7f414809b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d1db700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130004e70 0x7f41480b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 -- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f414809ac10 msgr2=0x7f414809b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f414809ac10 0x7f414809b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 -- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130004e70 msgr2=0x7f41480b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:16.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130004e70 0x7f41480b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:16.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 -- 172.21.15.26:0/3132557026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148064280 con 0x7f4148098de0 2022-07-20T02:26:16.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414d9dc700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f41400013a0 tx=0x7f4140015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41400016c0 con 0x7f4148098de0 2022-07-20T02:26:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414000ca10 con 0x7f4148098de0 2022-07-20T02:26:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4140006ea0 con 0x7f4148098de0 2022-07-20T02:26:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4148064520 con 0x7f4148098de0 2022-07-20T02:26:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4148064c60 con 0x7f4148098de0 2022-07-20T02:26:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41480574e0 con 0x7f4148098de0 2022-07-20T02:26:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.924+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4140027070 con 0x7f4148098de0 2022-07-20T02:26:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.928+0000 7f41367fc700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 0x7f4128055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.928+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f414006cd20 con 0x7f4148098de0 2022-07-20T02:26:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.928+0000 7f414d1db700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 0x7f4128055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.928+0000 7f414d1db700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 0x7f4128055ff0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4144005000 tx=0x7f4148060330).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:16.932+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4140043ae0 con 0x7f4148098de0 2022-07-20T02:26:17.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.096+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f41480b5a50 con 0x7f4148098de0 2022-07-20T02:26:17.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f41367fc700 1 -- 172.21.15.26:0/3132557026 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4140047100 con 0x7f4148098de0 2022-07-20T02:26:17.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 msgr2=0x7f4128055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 0x7f4128055ff0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4144005000 tx=0x7f4148060330).stop 2022-07-20T02:26:17.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 msgr2=0x7f4148099260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f41400013a0 tx=0x7f4140015000).stop 2022-07-20T02:26:17.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 shutdown_connections 2022-07-20T02:26:17.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4148098de0 0x7f4148099260 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4128053b30 0x7f4128055ff0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f414809ac10 0x7f414809b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.100+0000 7f414e1dd700 1 --2- 172.21.15.26:0/3132557026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4130004e70 0x7f41480b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.104+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 >> 172.21.15.26:0/3132557026 conn(0x7f41480b67f0 msgr2=0x7f4148140bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:17.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.104+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 shutdown_connections 2022-07-20T02:26:17.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.104+0000 7f414e1dd700 1 -- 172.21.15.26:0/3132557026 wait complete. 2022-07-20T02:26:17.119 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:26:17.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:17.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.228+0000 7ff84b0a4700 1 Processor -- start 2022-07-20T02:26:17.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.228+0000 7ff84b0a4700 1 -- start start 2022-07-20T02:26:17.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.228+0000 7ff84b0a4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff8440b5ef0 0x7ff8441474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.228+0000 7ff84b0a4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 0x7ff8441479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440b62d0 con 0x7ff8440a8c40 2022-07-20T02:26:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff844147eb0 con 0x7ff8440b5ef0 2022-07-20T02:26:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84414aac0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff8440b5ef0 0x7ff8441474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff8440b5ef0 0x7ff8441474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36418/0 (socket says 172.21.15.26:36418) 2022-07-20T02:26:17.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 -- 172.21.15.26:0/4121450760 learned_addr learned my addr 172.21.15.26:0/4121450760 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:17.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff8498a1700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 0x7ff8441479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 -- 172.21.15.26:0/4121450760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 msgr2=0x7ff8441479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:17.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 0x7ff8441479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 -- 172.21.15.26:0/4121450760 >> v1:172.21.15.120:6789/0 conn(0x7ff8440a8c40 legacy=0x7ff8440aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 -- 172.21.15.26:0/4121450760 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84414ad60 con 0x7ff8440b5ef0 2022-07-20T02:26:17.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff8498a1700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 0x7ff8441479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:17.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84a0a2700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440b5ef0 0x7ff8441474d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff834002f70 tx=0x7ff834006eb0).ready entity=mon.0 client_cookie=b5ad72ee45f60bfd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:17.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff8490a0700 1 -- 172.21.15.26:0/4121450760 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83400d5c0 con 0x7ff8440b5ef0 2022-07-20T02:26:17.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff8490a0700 1 -- 172.21.15.26:0/4121450760 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834000e90 con 0x7ff8440b5ef0 2022-07-20T02:26:17.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff8490a0700 1 -- 172.21.15.26:0/4121450760 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83400d950 con 0x7ff8440b5ef0 2022-07-20T02:26:17.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- 172.21.15.26:0/4121450760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440b5ef0 msgr2=0x7ff8441474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440b5ef0 0x7ff8441474d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff834002f70 tx=0x7ff834006eb0).stop 2022-07-20T02:26:17.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- 172.21.15.26:0/4121450760 shutdown_connections 2022-07-20T02:26:17.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a7fc0 0x7ff8441479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/4121450760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440b5ef0 0x7ff8441474d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- 172.21.15.26:0/4121450760 >> 172.21.15.26:0/4121450760 conn(0x7ff844060310 msgr2=0x7ff8440b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- 172.21.15.26:0/4121450760 shutdown_connections 2022-07-20T02:26:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- 172.21.15.26:0/4121450760 wait complete. 2022-07-20T02:26:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 Processor -- start 2022-07-20T02:26:17.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 -- start start 2022-07-20T02:26:17.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 0x7ff8440ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.232+0000 7ff84b0a4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8440b5ef0 0x7ff84411d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440643b0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff844064650 con 0x7ff8440b5ef0 2022-07-20T02:26:17.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff844063890 con 0x7ff8440a8c40 2022-07-20T02:26:17.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8498a1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 0x7ff8440ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8498a1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 0x7ff8440ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:57940/0 (socket says 172.21.15.26:57940) 2022-07-20T02:26:17.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a8a3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8440b5ef0 0x7ff84411d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36422/0 (socket says 172.21.15.26:36422) 2022-07-20T02:26:17.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8498a1700 1 -- 172.21.15.26:0/855435681 learned_addr learned my addr 172.21.15.26:0/855435681 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:17.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 -- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 msgr2=0x7ff8440ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 0x7ff8440ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 -- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8440b5ef0 msgr2=0x7ff84411d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8440b5ef0 0x7ff84411d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 -- 172.21.15.26:0/855435681 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83400b1e0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84a0a2700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff834002f70 tx=0x7ff834015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:17.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83400d940 con 0x7ff8440a7fc0 2022-07-20T02:26:17.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff844063b30 con 0x7ff8440a7fc0 2022-07-20T02:26:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8440649b0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834006a40 con 0x7ff8440a7fc0 2022-07-20T02:26:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834011b40 con 0x7ff8440a7fc0 2022-07-20T02:26:17.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.236+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff820003410 con 0x7ff8440a7fc0 2022-07-20T02:26:17.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.240+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff83401a040 con 0x7ff8440a7fc0 2022-07-20T02:26:17.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.240+0000 7ff8337fe700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 0x7ff824055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:17.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.240+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff83406ce70 con 0x7ff8440a7fc0 2022-07-20T02:26:17.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.240+0000 7ff8498a1700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 0x7ff824055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:17.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.240+0000 7ff8498a1700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 0x7ff824055f60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff84000d000 tx=0x7ff840008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:17.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.244+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff834043ae0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.408+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff820003c40 con 0x7ff8440a7fc0 2022-07-20T02:26:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.408+0000 7ff8337fe700 1 -- 172.21.15.26:0/855435681 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7ff8340470c0 con 0x7ff8440a7fc0 2022-07-20T02:26:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:17.415 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:17.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 msgr2=0x7ff824055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 0x7ff824055f60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff84000d000 tx=0x7ff840008040).stop 2022-07-20T02:26:17.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 msgr2=0x7ff844157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:17.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff834002f70 tx=0x7ff834015000).stop 2022-07-20T02:26:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 shutdown_connections 2022-07-20T02:26:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff824053aa0 0x7ff824055f60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8440b5ef0 0x7ff84411d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8440a8c40 0x7ff8440ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 --2- 172.21.15.26:0/855435681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8440a7fc0 0x7ff844157050 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 >> 172.21.15.26:0/855435681 conn(0x7ff844060310 msgr2=0x7ff844067950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 shutdown_connections 2022-07-20T02:26:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:17.412+0000 7ff84b0a4700 1 -- 172.21.15.26:0/855435681 wait complete. 2022-07-20T02:26:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:17.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:26:14.833033+0000', 'modified': '2022-07-20T02:26:15.835906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4881}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4881': {'gid': 4881, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6835/2813102162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6834', 'nonce': 2813102162}, {'type': 'v1', 'addr': '172.21.15.26:6835', 'nonce': 2813102162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:26:17.784 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:26:17.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:26:17.904 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":4881,"want_state":"up:active","state":"up:active","rank_uptime":3.0572042580000001,"mdsmap_epoch":13,"osdmap_epoch":34,"osdmap_epoch_barrier":33,"uptime":10.326911436} 2022-07-20T02:26:17.905 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 4881, "mdsmap_epoch": 13, "osdmap_epoch": 34, "osdmap_epoch_barrier": 33, "rank_uptime": 3.057204258, "state": "up:active", "uptime": 10.326911436, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:26:17.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:26:18.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 Processor -- start 2022-07-20T02:26:18.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- start start 2022-07-20T02:26:18.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4018060850 0x7f40180abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 0x7f40181468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40180610e0 con 0x7f4018060850 2022-07-20T02:26:18.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40180b2ea0 con 0x7f40180ac260 2022-07-20T02:26:18.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40180a8e20 con 0x7f40180b2ac0 2022-07-20T02:26:18.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4018060850 0x7f40180abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4018060850 0x7f40180abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36432/0 (socket says 172.21.15.26:36432) 2022-07-20T02:26:18.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 -- 172.21.15.26:0/2473759017 learned_addr learned my addr 172.21.15.26:0/2473759017 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4017fff700 1 --2- 172.21.15.26:0/2473759017 >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 0x7f40181468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f40177fe700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 -- 172.21.15.26:0/2473759017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 msgr2=0x7f40180ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 -- 172.21.15.26:0/2473759017 >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 msgr2=0x7f40181468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 --2- 172.21.15.26:0/2473759017 >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 0x7f40181468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 -- 172.21.15.26:0/2473759017 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180a95e0 con 0x7f4018060850 2022-07-20T02:26:18.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4017fff700 1 --2- 172.21.15.26:0/2473759017 >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 0x7f40181468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:18.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f4016ffd700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180abd70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f4000002f70 tx=0x7f4000006eb0).ready entity=mon.0 client_cookie=68fc73610805b0f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f40167fc700 1 -- 172.21.15.26:0/2473759017 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400000d570 con 0x7f4018060850 2022-07-20T02:26:18.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f40167fc700 1 -- 172.21.15.26:0/2473759017 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4000000e90 con 0x7f4018060850 2022-07-20T02:26:18.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f40167fc700 1 -- 172.21.15.26:0/2473759017 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400000d900 con 0x7f4018060850 2022-07-20T02:26:18.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- 172.21.15.26:0/2473759017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 msgr2=0x7f40180abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180abd70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f4000002f70 tx=0x7f4000006eb0).stop 2022-07-20T02:26:18.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.008+0000 7f401dad8700 1 -- 172.21.15.26:0/2473759017 shutdown_connections 2022-07-20T02:26:18.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- 172.21.15.26:0/2473759017 >> v2:172.21.15.120:3300/0 conn(0x7f40180ac260 0x7f40181468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180abd70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- 172.21.15.26:0/2473759017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- 172.21.15.26:0/2473759017 >> 172.21.15.26:0/2473759017 conn(0x7f40180b60a0 msgr2=0x7f40180b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- 172.21.15.26:0/2473759017 shutdown_connections 2022-07-20T02:26:18.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- 172.21.15.26:0/2473759017 wait complete. 2022-07-20T02:26:18.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 Processor -- start 2022-07-20T02:26:18.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- start start 2022-07-20T02:26:18.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40180ac260 0x7f401813f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180ae980 con 0x7f4018060850 2022-07-20T02:26:18.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401809f470 con 0x7f40180ac260 2022-07-20T02:26:18.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f401dad8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401809ef30 con 0x7f40180b2ac0 2022-07-20T02:26:18.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36436/0 (socket says 172.21.15.26:36436) 2022-07-20T02:26:18.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 -- 172.21.15.26:0/3734632650 learned_addr learned my addr 172.21.15.26:0/3734632650 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f4017fff700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f4016ffd700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40180ac260 0x7f401813f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 -- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 msgr2=0x7f40180ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 -- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40180ac260 msgr2=0x7f401813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40180ac260 0x7f401813f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.012+0000 7f40177fe700 1 -- 172.21.15.26:0/3734632650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4010009470 con 0x7f4018060850 2022-07-20T02:26:18.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f40177fe700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f40180ad3c0 tx=0x7f4008006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4008010780 con 0x7f4018060850 2022-07-20T02:26:18.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40080108e0 con 0x7f4018060850 2022-07-20T02:26:18.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400000b1e0 con 0x7f4018060850 2022-07-20T02:26:18.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4008010c40 con 0x7f4018060850 2022-07-20T02:26:18.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401809eb00 con 0x7f4018060850 2022-07-20T02:26:18.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401809f0f0 con 0x7f4018060850 2022-07-20T02:26:18.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f400801c460 con 0x7f4018060850 2022-07-20T02:26:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 0x7f3ffc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f400806c3f0 con 0x7f4018060850 2022-07-20T02:26:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4016ffd700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 0x7f3ffc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.016+0000 7f4016ffd700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 0x7f3ffc055e40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f400001c000 tx=0x7f4000015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.020+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f401809f0f0 con 0x7f4018060850 2022-07-20T02:26:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.148+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f401809f0f0 con 0x7f4018060850 2022-07-20T02:26:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.148+0000 7f4014ff9700 1 -- 172.21.15.26:0/3734632650 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f401809f0f0 con 0x7f4018060850 2022-07-20T02:26:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:18.157 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":34,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:26:15.835998+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:26:12.967166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T02:26:13.835040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:26:18.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 msgr2=0x7f3ffc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 0x7f3ffc055e40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f400001c000 tx=0x7f4000015000).stop 2022-07-20T02:26:18.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 msgr2=0x7f40180ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f40180ad3c0 tx=0x7f4008006df0).stop 2022-07-20T02:26:18.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 shutdown_connections 2022-07-20T02:26:18.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ffc053980 0x7f3ffc055e40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40180b2ac0 0x7f40180ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40180ac260 0x7f401813f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 --2- 172.21.15.26:0/3734632650 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4018060850 0x7f40180ac870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 >> 172.21.15.26:0/3734632650 conn(0x7f40180b60a0 msgr2=0x7f40180afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 shutdown_connections 2022-07-20T02:26:18.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.152+0000 7f401dad8700 1 -- 172.21.15.26:0/3734632650 wait complete. 2022-07-20T02:26:18.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:26:12.967166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:26:13.835040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:26:18.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:26:18.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 Processor -- start 2022-07-20T02:26:18.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 -- start start 2022-07-20T02:26:18.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68180a9520 0x7f681805c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f6818142c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 0x7f6818143120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68180b1090 con 0x7f68180a9520 2022-07-20T02:26:18.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68180b0bd0 con 0x7f68180ac8e0 2022-07-20T02:26:18.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681eb17700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180b12c0 con 0x7f68180b06d0 2022-07-20T02:26:18.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68180a9520 0x7f681805c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68180a9520 0x7f681805c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36444/0 (socket says 172.21.15.26:36444) 2022-07-20T02:26:18.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 -- 172.21.15.26:0/1102263884 learned_addr learned my addr 172.21.15.26:0/1102263884 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681e316700 1 --2- 172.21.15.26:0/1102263884 >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 0x7f6818143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681d314700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f6818142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 -- 172.21.15.26:0/1102263884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 msgr2=0x7f6818142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f6818142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 -- 172.21.15.26:0/1102263884 >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 msgr2=0x7f6818143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 --2- 172.21.15.26:0/1102263884 >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 0x7f6818143120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681db15700 1 -- 172.21.15.26:0/1102263884 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68180b1560 con 0x7f68180a9520 2022-07-20T02:26:18.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.296+0000 7f681e316700 1 --2- 172.21.15.26:0/1102263884 >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 0x7f6818143120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:18.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681db15700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681805c780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6808002f70 tx=0x7f6808006f40).ready entity=mon.0 client_cookie=8d79ae9929871fe3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681cb13700 1 -- 172.21.15.26:0/1102263884 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680800da80 con 0x7f68180a9520 2022-07-20T02:26:18.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681cb13700 1 -- 172.21.15.26:0/1102263884 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808000e90 con 0x7f68180a9520 2022-07-20T02:26:18.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681cb13700 1 -- 172.21.15.26:0/1102263884 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680800de10 con 0x7f68180a9520 2022-07-20T02:26:18.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- 172.21.15.26:0/1102263884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 msgr2=0x7f681805c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681805c780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6808002f70 tx=0x7f6808006f40).stop 2022-07-20T02:26:18.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- 172.21.15.26:0/1102263884 shutdown_connections 2022-07-20T02:26:18.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 --2- 172.21.15.26:0/1102263884 >> v2:172.21.15.120:3300/0 conn(0x7f68180ac8e0 0x7f6818143120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f6818142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 --2- 172.21.15.26:0/1102263884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681805c780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- 172.21.15.26:0/1102263884 >> 172.21.15.26:0/1102263884 conn(0x7f6818060310 msgr2=0x7f68180b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- 172.21.15.26:0/1102263884 shutdown_connections 2022-07-20T02:26:18.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- 172.21.15.26:0/1102263884 wait complete. 2022-07-20T02:26:18.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 Processor -- start 2022-07-20T02:26:18.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.300+0000 7f681eb17700 1 -- start start 2022-07-20T02:26:18.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68180ac8e0 0x7f6818152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f68180acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681813f630 con 0x7f68180a9520 2022-07-20T02:26:18.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6818151770 con 0x7f68180ac8e0 2022-07-20T02:26:18.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6818151a10 con 0x7f68180b06d0 2022-07-20T02:26:18.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36450/0 (socket says 172.21.15.26:36450) 2022-07-20T02:26:18.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 -- 172.21.15.26:0/3511767611 learned_addr learned my addr 172.21.15.26:0/3511767611 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681d314700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68180ac8e0 0x7f6818152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681e316700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f68180acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 -- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 msgr2=0x7f68180acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f68180acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 -- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68180ac8e0 msgr2=0x7f6818152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68180ac8e0 0x7f6818152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 -- 172.21.15.26:0/3511767611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680800b460 con 0x7f68180a9520 2022-07-20T02:26:18.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681db15700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6808002f70 tx=0x7f6808006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680800de40 con 0x7f68180a9520 2022-07-20T02:26:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808006c00 con 0x7f68180a9520 2022-07-20T02:26:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6818151cb0 con 0x7f68180a9520 2022-07-20T02:26:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6808018800 con 0x7f68180a9520 2022-07-20T02:26:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.304+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68180ad2b0 con 0x7f68180a9520 2022-07-20T02:26:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6818151e40 con 0x7f68180a9520 2022-07-20T02:26:18.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6808021020 con 0x7f68180a9520 2022-07-20T02:26:18.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f6806ffd700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 0x7f67f8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f680806fcf0 con 0x7f68180a9520 2022-07-20T02:26:18.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f681d314700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 0x7f67f8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.308+0000 7f681d314700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 0x7f67f8055e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f681400b000 tx=0x7f6814006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.316+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6808046af0 con 0x7f68180a9520 2022-07-20T02:26:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f681805cc70 con 0x7f68180a9520 2022-07-20T02:26:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f6806ffd700 1 -- 172.21.15.26:0/3511767611 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v34) v1 ==== 110+0+10 (secure 0 0 0) 0x7f680804a0d0 con 0x7f68180a9520 2022-07-20T02:26:18.440 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:26:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 msgr2=0x7f67f8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 0x7f67f8055e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f681400b000 tx=0x7f6814006df0).stop 2022-07-20T02:26:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 msgr2=0x7f681813ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6808002f70 tx=0x7f6808006f40).stop 2022-07-20T02:26:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 shutdown_connections 2022-07-20T02:26:18.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67f8053950 0x7f67f8055e10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68180b06d0 0x7f68180acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68180ac8e0 0x7f6818152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 --2- 172.21.15.26:0/3511767611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68180a9520 0x7f681813ffc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 >> 172.21.15.26:0/3511767611 conn(0x7f6818060310 msgr2=0x7f68180b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 shutdown_connections 2022-07-20T02:26:18.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.436+0000 7f681eb17700 1 -- 172.21.15.26:0/3511767611 wait complete. 2022-07-20T02:26:18.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:26:18.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 Processor -- start 2022-07-20T02:26:18.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 -- start start 2022-07-20T02:26:18.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc73405ff00 0x7fc734152740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc734060370 con 0x7fc734067ab0 2022-07-20T02:26:18.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73409ac80 con 0x7fc7340b31a0 2022-07-20T02:26:18.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc73a7ee700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73409af20 con 0x7fc73405ff00 2022-07-20T02:26:18.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc7337fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc734067ab0 0x7fc734061030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35740/0 (socket says 172.21.15.26:35740) 2022-07-20T02:26:18.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc7337fe700 1 -- 172.21.15.26:0/425539013 learned_addr learned my addr 172.21.15.26:0/425539013 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:18.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc738d8b700 1 --2- 172.21.15.26:0/425539013 >> v2:172.21.15.120:3300/0 conn(0x7fc73405ff00 0x7fc734152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520119137 0 0) 0x7fc72c001490 con 0x7fc734067ab0 2022-07-20T02:26:18.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc714003260 con 0x7fc734067ab0 2022-07-20T02:26:18.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066222366 0 0) 0x7fc714003260 con 0x7fc734067ab0 2022-07-20T02:26:18.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 >> v2:172.21.15.120:3300/0 conn(0x7fc73405ff00 msgr2=0x7fc734152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 --2- 172.21.15.26:0/425539013 >> v2:172.21.15.120:3300/0 conn(0x7fc73405ff00 0x7fc734152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 >> v1:172.21.15.120:6789/0 conn(0x7fc7340b31a0 legacy=0x7fc734060920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73409b1c0 con 0x7fc734067ab0 2022-07-20T02:26:18.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fc72c001b70 con 0x7fc734067ab0 2022-07-20T02:26:18.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc738d8b700 1 --2- 172.21.15.26:0/425539013 >> v2:172.21.15.120:3300/0 conn(0x7fc73405ff00 0x7fc734152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:18.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 >> v1:172.21.15.26:6789/0 conn(0x7fc734067ab0 legacy=0x7fc734061030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 0x7fc714003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc714004500 0x7fc714004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc714004e70 0x7fc71400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73409af20 con 0x7fc714003750 2022-07-20T02:26:18.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71400bd20 con 0x7fc714004500 2022-07-20T02:26:18.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71400bee0 con 0x7fc714004e70 2022-07-20T02:26:18.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc733fff700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 0x7fc714003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc738d8b700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc714004e70 0x7fc71400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.556+0000 7fc7337fe700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc714004500 0x7fc714004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 -- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc714004e70 msgr2=0x7fc71400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc714004e70 0x7fc71400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 -- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc714004500 msgr2=0x7fc714004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc714004500 0x7fc714004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc738d8b700 1 -- 172.21.15.26:0/425539013 reap_dead start 2022-07-20T02:26:18.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 -- 172.21.15.26:0/425539013 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714003260 con 0x7fc714003750 2022-07-20T02:26:18.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc7337fe700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc714004500 0x7fc714004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:18.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 0x7fc714003b70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fc734068970 tx=0x7fc71c007a00).ready entity=mon.0 client_cookie=a4b385607927a587 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c007d10 con 0x7fc714003750 2022-07-20T02:26:18.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c007030 con 0x7fc714003750 2022-07-20T02:26:18.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc732ffd700 1 -- 172.21.15.26:0/425539013 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c007390 con 0x7fc714003750 2022-07-20T02:26:18.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 msgr2=0x7fc714003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 0x7fc714003b70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fc734068970 tx=0x7fc71c007a00).stop 2022-07-20T02:26:18.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- 172.21.15.26:0/425539013 shutdown_connections 2022-07-20T02:26:18.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/425539013 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc714003750 0x7fc714003b70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- 172.21.15.26:0/425539013 >> 172.21.15.26:0/425539013 conn(0x7fc7340b6920 msgr2=0x7fc7340b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- 172.21.15.26:0/425539013 shutdown_connections 2022-07-20T02:26:18.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- 172.21.15.26:0/425539013 wait complete. 2022-07-20T02:26:18.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 Processor -- start 2022-07-20T02:26:18.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- start start 2022-07-20T02:26:18.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 0x7fc734148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc734148610 0x7fc734063e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc734066b10 con 0x7fc734064300 2022-07-20T02:26:18.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc734149d70 con 0x7fc734148610 2022-07-20T02:26:18.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc73a7ee700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340671f0 con 0x7fc734149990 2022-07-20T02:26:18.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc738d8b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc738d8b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36474/0 (socket says 172.21.15.26:36474) 2022-07-20T02:26:18.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc738d8b700 1 -- 172.21.15.26:0/2232177816 learned_addr learned my addr 172.21.15.26:0/2232177816 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc7337fe700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc734148610 0x7fc734063e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.560+0000 7fc733fff700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 0x7fc734148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 -- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 msgr2=0x7fc734148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 0x7fc734148120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 -- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc734148610 msgr2=0x7fc734063e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc734148610 0x7fc734063e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 -- 172.21.15.26:0/2232177816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734066130 con 0x7fc734064300 2022-07-20T02:26:18.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc733fff700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 0x7fc734148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:18.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc738d8b700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc72400b000 tx=0x7fc724006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc724011e80 con 0x7fc734064300 2022-07-20T02:26:18.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72400f910 con 0x7fc734064300 2022-07-20T02:26:18.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7340663d0 con 0x7fc734064300 2022-07-20T02:26:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc72400fc70 con 0x7fc734064300 2022-07-20T02:26:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7340977b0 con 0x7fc734064300 2022-07-20T02:26:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7340574e0 con 0x7fc734064300 2022-07-20T02:26:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc724014440 con 0x7fc734064300 2022-07-20T02:26:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 0x7fc718055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc72406c600 con 0x7fc734064300 2022-07-20T02:26:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.564+0000 7fc733fff700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 0x7fc718055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.568+0000 7fc733fff700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 0x7fc718055f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc71c019000 tx=0x7fc71c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.572+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc724043410 con 0x7fc734064300 2022-07-20T02:26:18.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.696+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc73409bf00 con 0x7fc734064300 2022-07-20T02:26:18.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.696+0000 7fc7317fa700 1 -- 172.21.15.26:0/2232177816 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v34) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc7240180a0 con 0x7fc734064300 2022-07-20T02:26:18.702 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:26:18.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 msgr2=0x7fc718055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 0x7fc718055f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc71c019000 tx=0x7fc71c015000).stop 2022-07-20T02:26:18.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 msgr2=0x7fc73415a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc72400b000 tx=0x7fc724006df0).stop 2022-07-20T02:26:18.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 shutdown_connections 2022-07-20T02:26:18.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc734064300 0x7fc73415a3e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc718053aa0 0x7fc718055f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc734148610 0x7fc734063e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 --2- 172.21.15.26:0/2232177816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc734149990 0x7fc734148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 >> 172.21.15.26:0/2232177816 conn(0x7fc7340b6920 msgr2=0x7fc734095350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 shutdown_connections 2022-07-20T02:26:18.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.700+0000 7fc73a7ee700 1 -- 172.21.15.26:0/2232177816 wait complete. 2022-07-20T02:26:18.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:26:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 Processor -- start 2022-07-20T02:26:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 -- start start 2022-07-20T02:26:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8b040aaf00 0x7f8b040b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0414a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0414b470 con 0x7f8b040b7d80 2022-07-20T02:26:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b04064650 con 0x7f8b040aaf00 2022-07-20T02:26:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040b41a0 con 0x7f8b04149ce0 2022-07-20T02:26:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8b040aaf00 0x7f8b040b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0259c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8b040b7d80 0x7f8b0414ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:49970/0 (socket says 172.21.15.26:49970) 2022-07-20T02:26:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b0259c700 1 -- 172.21.15.26:0/3919721284 learned_addr learned my addr 172.21.15.26:0/3919721284 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/3919721284 >> v2:172.21.15.26:3300/0 conn(0x7f8b040aaf00 0x7f8b040b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36480/0 (socket says 172.21.15.26:36480) 2022-07-20T02:26:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b02d9d700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0414a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b01d9b700 1 -- 172.21.15.26:0/3919721284 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805077115 0 0) 0x7f8b0414b470 con 0x7f8b040b7d80 2022-07-20T02:26:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8b01d9b700 1 -- 172.21.15.26:0/3919721284 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae4003260 con 0x7f8b040b7d80 2022-07-20T02:26:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 -- 172.21.15.26:0/3919721284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 msgr2=0x7f8b0414a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0414a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 -- 172.21.15.26:0/3919721284 >> v1:172.21.15.120:6789/0 conn(0x7f8b040b7d80 legacy=0x7f8b0414ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 -- 172.21.15.26:0/3919721284 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040b4440 con 0x7f8b040aaf00 2022-07-20T02:26:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.828+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b040b4b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).ready entity=mon.0 client_cookie=30d6da660fde08e1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b01d9b700 1 -- 172.21.15.26:0/3919721284 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d4a0 con 0x7f8b040aaf00 2022-07-20T02:26:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b01d9b700 1 -- 172.21.15.26:0/3919721284 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec000e90 con 0x7f8b040aaf00 2022-07-20T02:26:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b01d9b700 1 -- 172.21.15.26:0/3919721284 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d7f0 con 0x7f8b040aaf00 2022-07-20T02:26:18.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- 172.21.15.26:0/3919721284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 msgr2=0x7f8b040b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b040b4b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).stop 2022-07-20T02:26:18.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- 172.21.15.26:0/3919721284 shutdown_connections 2022-07-20T02:26:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0414a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 --2- 172.21.15.26:0/3919721284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b040b4b80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- 172.21.15.26:0/3919721284 >> 172.21.15.26:0/3919721284 conn(0x7f8b0405f840 msgr2=0x7f8b0405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- 172.21.15.26:0/3919721284 shutdown_connections 2022-07-20T02:26:18.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- 172.21.15.26:0/3919721284 wait complete. 2022-07-20T02:26:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 Processor -- start 2022-07-20T02:26:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.832+0000 7f8b0359e700 1 -- start start 2022-07-20T02:26:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 0x7f8b041567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040b16f0 con 0x7f8b040aaf00 2022-07-20T02:26:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040aefb0 con 0x7f8b040b7d80 2022-07-20T02:26:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040aed50 con 0x7f8b04149ce0 2022-07-20T02:26:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8af9d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 0x7f8b041567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36484/0 (socket says 172.21.15.26:36484) 2022-07-20T02:26:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 -- 172.21.15.26:0/1335443180 learned_addr learned my addr 172.21.15.26:0/1335443180 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b02d9d700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 -- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 msgr2=0x7f8b0415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 -- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 msgr2=0x7f8b041567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 0x7f8b041567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 -- 172.21.15.26:0/1335443180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec00b180 con 0x7f8b040aaf00 2022-07-20T02:26:18.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 0x7f8b041567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:18.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0259c700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8af4001110 tx=0x7f8af4007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af40070d0 con 0x7f8b040aaf00 2022-07-20T02:26:18.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af4007230 con 0x7f8b040aaf00 2022-07-20T02:26:18.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b04159db0 con 0x7f8b040aaf00 2022-07-20T02:26:18.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af4005610 con 0x7f8b040aaf00 2022-07-20T02:26:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b040ad5d0 con 0x7f8b040aaf00 2022-07-20T02:26:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0415a250 con 0x7f8b040aaf00 2022-07-20T02:26:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.836+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8af401a060 con 0x7f8b040aaf00 2022-07-20T02:26:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.840+0000 7f8afbfff700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 0x7f8ae8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.840+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8af406b470 con 0x7f8b040aaf00 2022-07-20T02:26:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.840+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 0x7f8ae8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.840+0000 7f8af9d9b700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 0x7f8ae8055ea0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8aec01a000 tx=0x7f8aec014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.844+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b0415a250 con 0x7f8b040aaf00 2022-07-20T02:26:18.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b0415a250 con 0x7f8b040aaf00 2022-07-20T02:26:18.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8afbfff700 1 -- 172.21.15.26:0/1335443180 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8b0415a250 con 0x7f8b040aaf00 2022-07-20T02:26:18.975 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:26:18.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 msgr2=0x7f8ae8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 0x7f8ae8055ea0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8aec01a000 tx=0x7f8aec014000).stop 2022-07-20T02:26:18.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 msgr2=0x7f8b04066a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:18.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8af4001110 tx=0x7f8af4007b80).stop 2022-07-20T02:26:18.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 shutdown_connections 2022-07-20T02:26:18.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ae80539e0 0x7f8ae8055ea0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b04149ce0 0x7f8b0415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b040b7d80 0x7f8b041567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 --2- 172.21.15.26:0/1335443180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b040aaf00 0x7f8b04066a40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:18.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 >> 172.21.15.26:0/1335443180 conn(0x7f8b0405f840 msgr2=0x7f8b040b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:18.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 shutdown_connections 2022-07-20T02:26:18.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:18.972+0000 7f8b0359e700 1 -- 172.21.15.26:0/1335443180 wait complete. 2022-07-20T02:26:18.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:26:19.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 Processor -- start 2022-07-20T02:26:19.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 -- start start 2022-07-20T02:26:19.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640618a0 0x7f0764144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f07640b4240 0x7f0764147800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764061c80 con 0x7f076414ac80 2022-07-20T02:26:19.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07640b1ab0 con 0x7f07640b4240 2022-07-20T02:26:19.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0763fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076414b190 con 0x7f07640618a0 2022-07-20T02:26:19.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f07640b4240 0x7f0764147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f07640b4240 0x7f0764147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36496/0 (socket says 172.21.15.26:36496) 2022-07-20T02:26:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 -- 172.21.15.26:0/4137157575 learned_addr learned my addr 172.21.15.26:0/4137157575 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0762ffd700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640618a0 0x7f0764144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 -- 172.21.15.26:0/4137157575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640618a0 msgr2=0x7f0764144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640618a0 0x7f0764144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 -- 172.21.15.26:0/4137157575 >> v1:172.21.15.120:6789/0 conn(0x7f076414ac80 legacy=0x7f0764144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 -- 172.21.15.26:0/4137157575 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076414b430 con 0x7f07640b4240 2022-07-20T02:26:19.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f07637fe700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07640b4240 0x7f0764147800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f075c002f70 tx=0x7f075c006eb0).ready entity=mon.0 client_cookie=41458971736fd736 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0761ffb700 1 -- 172.21.15.26:0/4137157575 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075c00d570 con 0x7f07640b4240 2022-07-20T02:26:19.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0761ffb700 1 -- 172.21.15.26:0/4137157575 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c000e90 con 0x7f07640b4240 2022-07-20T02:26:19.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.096+0000 7f0761ffb700 1 -- 172.21.15.26:0/4137157575 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075c00d900 con 0x7f07640b4240 2022-07-20T02:26:19.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- 172.21.15.26:0/4137157575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07640b4240 msgr2=0x7f0764147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07640b4240 0x7f0764147800 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f075c002f70 tx=0x7f075c006eb0).stop 2022-07-20T02:26:19.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- 172.21.15.26:0/4137157575 shutdown_connections 2022-07-20T02:26:19.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07640b4240 0x7f0764147800 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 --2- 172.21.15.26:0/4137157575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640618a0 0x7f0764144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- 172.21.15.26:0/4137157575 >> 172.21.15.26:0/4137157575 conn(0x7f0764066f70 msgr2=0x7f0764067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- 172.21.15.26:0/4137157575 shutdown_connections 2022-07-20T02:26:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- 172.21.15.26:0/4137157575 wait complete. 2022-07-20T02:26:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 Processor -- start 2022-07-20T02:26:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 -- start start 2022-07-20T02:26:19.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.100+0000 7f0763fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640b4240 0x7f076413e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f076414ac80 0x7f076413ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076413f2d0 con 0x7f076414ac80 2022-07-20T02:26:19.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07640af330 con 0x7f07640618a0 2022-07-20T02:26:19.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07640b79e0 con 0x7f07640b4240 2022-07-20T02:26:19.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f07637fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f076414ac80 0x7f076413ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:56190/0 (socket says 172.21.15.26:56190) 2022-07-20T02:26:19.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 -- 172.21.15.26:0/801607499 learned_addr learned my addr 172.21.15.26:0/801607499 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:19.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f07627fc700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640b4240 0x7f076413e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 -- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640b4240 msgr2=0x7f076413e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640b4240 0x7f076413e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 -- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f076414ac80 msgr2=0x7f076413ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f076414ac80 0x7f076413ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 -- 172.21.15.26:0/801607499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c00b1e0 con 0x7f07640618a0 2022-07-20T02:26:19.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0762ffd700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f074c00c000 tx=0x7f074c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074c010780 con 0x7f07640618a0 2022-07-20T02:26:19.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c0108e0 con 0x7f07640618a0 2022-07-20T02:26:19.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0764146f20 con 0x7f07640618a0 2022-07-20T02:26:19.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074c010c40 con 0x7f07640618a0 2022-07-20T02:26:19.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07640b7460 con 0x7f07640618a0 2022-07-20T02:26:19.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f074c01c460 con 0x7f07640618a0 2022-07-20T02:26:19.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.104+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0764147540 con 0x7f07640618a0 2022-07-20T02:26:19.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.108+0000 7f075bfff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 0x7f0748055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.108+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f074c06c620 con 0x7f07640618a0 2022-07-20T02:26:19.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.108+0000 7f07627fc700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 0x7f0748055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.108+0000 7f07627fc700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 0x7f0748055e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f075400d000 tx=0x7f0754007f10).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.116+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f074c0431f0 con 0x7f07640618a0 2022-07-20T02:26:19.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.244+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f076414b6d0 con 0x7f07640618a0 2022-07-20T02:26:19.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.244+0000 7f075bfff700 1 -- 172.21.15.26:0/801607499 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+10 (secure 0 0 0) 0x7f074c046810 con 0x7f07640618a0 2022-07-20T02:26:19.255 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:26:19.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 msgr2=0x7f0748055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 0x7f0748055e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f075400d000 tx=0x7f0754007f10).stop 2022-07-20T02:26:19.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 msgr2=0x7f07640b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f074c00c000 tx=0x7f074c006df0).stop 2022-07-20T02:26:19.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 shutdown_connections 2022-07-20T02:26:19.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f076414ac80 0x7f076413ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0748053950 0x7f0748055e10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07640b4240 0x7f076413e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 --2- 172.21.15.26:0/801607499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07640618a0 0x7f07640b4d60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 >> 172.21.15.26:0/801607499 conn(0x7f0764066f70 msgr2=0x7f076413fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 shutdown_connections 2022-07-20T02:26:19.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.248+0000 7f0763fff700 1 -- 172.21.15.26:0/801607499 wait complete. 2022-07-20T02:26:19.269 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:26:19.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:19.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 Processor -- start 2022-07-20T02:26:19.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- start start 2022-07-20T02:26:19.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7c50147be0 0x7f7c500aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 0x7f7c500af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c500b62d0 con 0x7f7c500b5ef0 2022-07-20T02:26:19.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5014aac0 con 0x7f7c50147be0 2022-07-20T02:26:19.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5014ad60 con 0x7f7c500a7fc0 2022-07-20T02:26:19.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7c50147be0 0x7f7c500aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7c50147be0 0x7f7c500aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36506/0 (socket says 172.21.15.26:36506) 2022-07-20T02:26:19.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 -- 172.21.15.26:0/3549101521 learned_addr learned my addr 172.21.15.26:0/3549101521 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:19.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 -- 172.21.15.26:0/3549101521 >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 msgr2=0x7f7c500af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:19.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5665c700 1 --2- 172.21.15.26:0/3549101521 >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 0x7f7c500af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 --2- 172.21.15.26:0/3549101521 >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 0x7f7c500af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 -- 172.21.15.26:0/3549101521 >> v1:172.21.15.120:6789/0 conn(0x7f7c500b5ef0 legacy=0x7f7c501474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:19.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 -- 172.21.15.26:0/3549101521 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5014b000 con 0x7f7c50147be0 2022-07-20T02:26:19.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5665c700 1 --2- 172.21.15.26:0/3549101521 >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 0x7f7c500af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:19.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c5565a700 1 --2- 172.21.15.26:0/3549101521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c50147be0 0x7f7c500aee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7c50060810 tx=0x7f7c4c006eb0).ready entity=mon.0 client_cookie=3913194445741925 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c54e59700 1 -- 172.21.15.26:0/3549101521 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c4c00d5c0 con 0x7f7c50147be0 2022-07-20T02:26:19.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c54e59700 1 -- 172.21.15.26:0/3549101521 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c000e90 con 0x7f7c50147be0 2022-07-20T02:26:19.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c54e59700 1 -- 172.21.15.26:0/3549101521 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c4c00d910 con 0x7f7c50147be0 2022-07-20T02:26:19.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- 172.21.15.26:0/3549101521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c50147be0 msgr2=0x7f7c500aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/3549101521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c50147be0 0x7f7c500aee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7c50060810 tx=0x7f7c4c006eb0).stop 2022-07-20T02:26:19.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- 172.21.15.26:0/3549101521 shutdown_connections 2022-07-20T02:26:19.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/3549101521 >> v2:172.21.15.120:3300/0 conn(0x7f7c500a7fc0 0x7f7c500af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/3549101521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c50147be0 0x7f7c500aee40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- 172.21.15.26:0/3549101521 >> 172.21.15.26:0/3549101521 conn(0x7f7c50060310 msgr2=0x7f7c500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:19.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- 172.21.15.26:0/3549101521 shutdown_connections 2022-07-20T02:26:19.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.380+0000 7f7c56e5d700 1 -- 172.21.15.26:0/3549101521 wait complete. 2022-07-20T02:26:19.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 Processor -- start 2022-07-20T02:26:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 -- start start 2022-07-20T02:26:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c500b5ef0 0x7f7c50147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 0x7f7c500b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c50159af0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c500aa3d0 con 0x7f7c50147be0 2022-07-20T02:26:19.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c56e5d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c500ab4c0 con 0x7f7c500b5ef0 2022-07-20T02:26:19.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36512/0 (socket says 172.21.15.26:36512) 2022-07-20T02:26:19.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 -- 172.21.15.26:0/1670867424 learned_addr learned my addr 172.21.15.26:0/1670867424 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:19.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c5665c700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 0x7f7c500b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c5565a700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c500b5ef0 0x7f7c50147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 -- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c500b5ef0 msgr2=0x7f7c50147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c500b5ef0 0x7f7c50147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 -- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 msgr2=0x7f7c500b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 0x7f7c500b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 -- 172.21.15.26:0/1670867424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c00b1e0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c5665c700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 0x7f7c500b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.384+0000 7f7c55e5b700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7c40001fa0 tx=0x7f7c40007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c40007030 con 0x7f7c500a7fc0 2022-07-20T02:26:19.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c40007190 con 0x7f7c500a7fc0 2022-07-20T02:26:19.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c500aa600 con 0x7f7c500a7fc0 2022-07-20T02:26:19.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c40006510 con 0x7f7c500a7fc0 2022-07-20T02:26:19.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c500ac240 con 0x7f7c500a7fc0 2022-07-20T02:26:19.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5014b2a0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7c40010030 con 0x7f7c500a7fc0 2022-07-20T02:26:19.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 0x7f7c30055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:19.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7c4006bcc0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.388+0000 7f7c5565a700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 0x7f7c30055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:19.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.392+0000 7f7c5565a700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 0x7f7c30055e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7c4c01c000 tx=0x7f7c4c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:19.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.396+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c40042a80 con 0x7f7c500a7fc0 2022-07-20T02:26:19.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.560+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c500574e0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.560+0000 7f7c3effd700 1 -- 172.21.15.26:0/1670867424 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7f7c400460a0 con 0x7f7c500a7fc0 2022-07-20T02:26:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:19.565 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 msgr2=0x7f7c30055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 0x7f7c30055e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7c4c01c000 tx=0x7f7c4c015000).stop 2022-07-20T02:26:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 msgr2=0x7f7c500ab7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7c40001fa0 tx=0x7f7c40007c30).stop 2022-07-20T02:26:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 shutdown_connections 2022-07-20T02:26:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c30053950 0x7f7c30055e10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c50147be0 0x7f7c500b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c500b5ef0 0x7f7c50147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 --2- 172.21.15.26:0/1670867424 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c500a7fc0 0x7f7c500ab7f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 >> 172.21.15.26:0/1670867424 conn(0x7f7c50060310 msgr2=0x7f7c500b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 shutdown_connections 2022-07-20T02:26:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:19.564+0000 7f7c56e5d700 1 -- 172.21.15.26:0/1670867424 wait complete. 2022-07-20T02:26:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:19.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:26:14.833033+0000', 'modified': '2022-07-20T02:26:15.835906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4881}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4881': {'gid': 4881, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6835/2813102162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6834', 'nonce': 2813102162}, {'type': 'v1', 'addr': '172.21.15.26:6835', 'nonce': 2813102162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:26:19.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:26:19.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:26:20.033 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":4881,"want_state":"up:active","state":"up:active","rank_uptime":5.1791799090000001,"mdsmap_epoch":13,"osdmap_epoch":34,"osdmap_epoch_barrier":33,"uptime":12.448876447} 2022-07-20T02:26:20.034 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 4881, "mdsmap_epoch": 13, "osdmap_epoch": 34, "osdmap_epoch_barrier": 33, "rank_uptime": 5.179179909, "state": "up:active", "uptime": 12.448876447, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:26:20.035 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:26:20.036 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:26:20.036 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:26:20.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.039 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:20.055 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:26:20.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:26:20.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:26:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:26:20.198 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:26:20.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.199 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:26:20.218 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:26:20.219 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:20.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:26:20.284 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:26:20.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:26:20.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:26:20.332+0000 7f2bc941a080 -1 init, newargv = 0x5579cdfa4870 newargc=9ceph-fuse[22301]: starting ceph client 2022-07-20T02:26:20.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:26:20.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22301]: starting fuse 2022-07-20T02:26:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:26:20.374 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:26:20.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.375 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:26:20.437 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:26:20.437 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:26:20.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:26:20.440 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:26:20.441 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:26:20.442 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:26:20.442 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:26:20.443 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.444 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:26:20.445 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:26:20.445 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:26:20.446 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.447 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:26:20.448 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:26:20.448 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:26:20.449 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:26:20.450 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.457 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:26:20.458 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:26:20.459 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:26:20.470 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:26:20.471 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:26:20.472 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.473 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:26:20.474 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:26:20.523 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22301.asok 2022-07-20T02:26:20.535 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22301.asok 2022-07-20T02:26:20.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.537 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22301.asok status 2022-07-20T02:26:20.639 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:26:20.640 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:26:20.640 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:26:20.641 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:26:20.641 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:26:20.642 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:26:20.664 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:26:20.664 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22301", 2022-07-20T02:26:20.665 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:26:20.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:26:20.666 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:26:20.666 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:26:20.666 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 4832, 2022-07-20T02:26:20.667 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:26:20.667 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:26:20.668 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:26:20.668 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 4832 2022-07-20T02:26:20.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:26:20.669 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:26:20.669 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:26:20.670 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:26:20.670 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2871190150 2022-07-20T02:26:20.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:26:20.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:26:20.672 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:26:20.672 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:26:20.673 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:26:20.673 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2871190150 2022-07-20T02:26:20.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:26:20.682 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.4832 172.21.15.159:0/2871190150", 2022-07-20T02:26:20.683 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/2871190150", 2022-07-20T02:26:20.683 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:26:20.684 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 13, 2022-07-20T02:26:20.684 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 34, 2022-07-20T02:26:20.685 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:26:20.685 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:26:20.686 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:26:20.687 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:26:20.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.690 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:20.691 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:20.704 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:26:20.711 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:26:20.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.713 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:20.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_min_caps_per_client 2022-07-20T02:26:20.892 INFO:teuthology.orchestra.run.smithi026.stdout:{"mds_min_caps_per_client":"100"} 2022-07-20T02:26:20.893 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_min_caps_per_client": "100" } 2022-07-20T02:26:20.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:26:20.896 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:26:20.897 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:26:20.916 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:26:20.917 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:26:20.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:26:20.918 DEBUG:teuthology.orchestra.run.smithi159:> import sys 2022-07-20T02:26:20.918 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:26:20.919 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:26:20.919 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.920 DEBUG:teuthology.orchestra.run.smithi159:> n = 1000 2022-07-20T02:26:20.920 DEBUG:teuthology.orchestra.run.smithi159:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T02:26:20.920 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.921 DEBUG:teuthology.orchestra.run.smithi159:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T02:26:20.921 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T02:26:20.922 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:26:20.922 DEBUG:teuthology.orchestra.run.smithi159:> for i in range(0, n): 2022-07-20T02:26:20.923 DEBUG:teuthology.orchestra.run.smithi159:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T02:26:20.923 DEBUG:teuthology.orchestra.run.smithi159:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T02:26:20.924 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:26:20.930 DEBUG:teuthology.orchestra.run.smithi159:> if True: 2022-07-20T02:26:20.932 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:26:20.934 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(f.fileno()) 2022-07-20T02:26:20.944 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:26:32.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:32.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 Processor -- start 2022-07-20T02:26:32.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 -- start start 2022-07-20T02:26:32.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60d01484a0 0x7f60d0066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60d0061050 0x7f60d0066f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d00b1bc0 con 0x7f60d00b2920 2022-07-20T02:26:32.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d014b040 con 0x7f60d0061050 2022-07-20T02:26:32.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.352+0000 7f60d5039700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d00a8860 con 0x7f60d01484a0 2022-07-20T02:26:32.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60d0061050 0x7f60d0066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60d0061050 0x7f60d0066f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36528/0 (socket says 172.21.15.26:36528) 2022-07-20T02:26:32.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 -- 172.21.15.26:0/121657216 learned_addr learned my addr 172.21.15.26:0/121657216 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:32.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60ce59c700 1 --2- 172.21.15.26:0/121657216 >> v2:172.21.15.120:3300/0 conn(0x7f60d01484a0 0x7f60d0066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 -- 172.21.15.26:0/121657216 >> v2:172.21.15.120:3300/0 conn(0x7f60d01484a0 msgr2=0x7f60d0066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:32.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- 172.21.15.26:0/121657216 >> v2:172.21.15.120:3300/0 conn(0x7f60d01484a0 0x7f60d0066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 -- 172.21.15.26:0/121657216 >> v1:172.21.15.120:6789/0 conn(0x7f60d00b2920 legacy=0x7f60d0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:32.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 -- 172.21.15.26:0/121657216 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d0147700 con 0x7f60d0061050 2022-07-20T02:26:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- 172.21.15.26:0/121657216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d0061050 0x7f60d0066f70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f60d0147c70 tx=0x7f60c8009290).ready entity=mon.0 client_cookie=c354790a305e73bd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cdd9b700 1 -- 172.21.15.26:0/121657216 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60c800b830 con 0x7f60d0061050 2022-07-20T02:26:32.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cdd9b700 1 -- 172.21.15.26:0/121657216 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c800b990 con 0x7f60d0061050 2022-07-20T02:26:32.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cdd9b700 1 -- 172.21.15.26:0/121657216 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60c800bb40 con 0x7f60d0061050 2022-07-20T02:26:32.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- 172.21.15.26:0/121657216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d0061050 msgr2=0x7f60d0066f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- 172.21.15.26:0/121657216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d0061050 0x7f60d0066f70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f60d0147c70 tx=0x7f60c8009290).stop 2022-07-20T02:26:32.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- 172.21.15.26:0/121657216 shutdown_connections 2022-07-20T02:26:32.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- 172.21.15.26:0/121657216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d0061050 0x7f60d0066f70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- 172.21.15.26:0/121657216 >> v2:172.21.15.120:3300/0 conn(0x7f60d01484a0 0x7f60d0066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- 172.21.15.26:0/121657216 >> 172.21.15.26:0/121657216 conn(0x7f60d00b60a0 msgr2=0x7f60d00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- 172.21.15.26:0/121657216 shutdown_connections 2022-07-20T02:26:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- 172.21.15.26:0/121657216 wait complete. 2022-07-20T02:26:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 Processor -- start 2022-07-20T02:26:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- start start 2022-07-20T02:26:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d0061050 0x7f60d0063180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d00b2920 0x7f60d0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00661f0 con 0x7f60d01484a0 2022-07-20T02:26:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00649c0 con 0x7f60d00b2920 2022-07-20T02:26:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60d5039700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d0064c60 con 0x7f60d0061050 2022-07-20T02:26:32.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36536/0 (socket says 172.21.15.26:36536) 2022-07-20T02:26:32.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60cf59e700 1 -- 172.21.15.26:0/198820171 learned_addr learned my addr 172.21.15.26:0/198820171 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:32.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60ce59c700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d00b2920 0x7f60d0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.356+0000 7f60ced9d700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d0061050 0x7f60d0063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 -- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d0061050 msgr2=0x7f60d0063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d0061050 0x7f60d0063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 -- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d00b2920 msgr2=0x7f60d0062b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d00b2920 0x7f60d0062b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 -- 172.21.15.26:0/198820171 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8009530 con 0x7f60d01484a0 2022-07-20T02:26:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60cf59e700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f60d0147c70 tx=0x7f60c8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60c8021440 con 0x7f60d01484a0 2022-07-20T02:26:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c80215a0 con 0x7f60d01484a0 2022-07-20T02:26:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60c8021900 con 0x7f60d01484a0 2022-07-20T02:26:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d0064f00 con 0x7f60d01484a0 2022-07-20T02:26:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d00a9880 con 0x7f60d01484a0 2022-07-20T02:26:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d00650c0 con 0x7f60d01484a0 2022-07-20T02:26:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60c8027020 con 0x7f60d01484a0 2022-07-20T02:26:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 0x7f60ac055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f60c806d4a0 con 0x7f60d01484a0 2022-07-20T02:26:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.360+0000 7f60ced9d700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 0x7f60ac055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.364+0000 7f60ced9d700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 0x7f60ac055ff0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f60c000b000 tx=0x7f60c0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.368+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60c80441c0 con 0x7f60d01484a0 2022-07-20T02:26:32.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.536+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d014ada0 con 0x7f60d01484a0 2022-07-20T02:26:32.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.536+0000 7f60bffff700 1 -- 172.21.15.26:0/198820171 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7f60c8015020 con 0x7f60d01484a0 2022-07-20T02:26:32.545 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:32.546 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:32.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 msgr2=0x7f60ac055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 0x7f60ac055ff0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f60c000b000 tx=0x7f60c0009040).stop 2022-07-20T02:26:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 msgr2=0x7f60d00656b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f60d0147c70 tx=0x7f60c8016000).stop 2022-07-20T02:26:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 shutdown_connections 2022-07-20T02:26:32.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60d01484a0 0x7f60d00656b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60ac053b30 0x7f60ac055ff0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60d00b2920 0x7f60d0062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 --2- 172.21.15.26:0/198820171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60d0061050 0x7f60d0063180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 >> 172.21.15.26:0/198820171 conn(0x7f60d00b60a0 msgr2=0x7f60d00b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:32.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 shutdown_connections 2022-07-20T02:26:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.540+0000 7f60d5039700 1 -- 172.21.15.26:0/198820171 wait complete. 2022-07-20T02:26:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:32.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-20T02:26:32.677 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T02:26:32.677 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T02:26:32.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:32.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.776+0000 7fb7190fd700 1 Processor -- start 2022-07-20T02:26:32.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.776+0000 7fb7190fd700 1 -- start start 2022-07-20T02:26:32.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb7140ac150 0x7fb7140aab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140afe40 0x7fb7140ab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71405d370 con 0x7fb7140b5d50 2022-07-20T02:26:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7140ab770 con 0x7fb7140ac150 2022-07-20T02:26:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140aba10 con 0x7fb7140afe40 2022-07-20T02:26:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb7140ac150 0x7fb7140aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb7140ac150 0x7fb7140aab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36540/0 (socket says 172.21.15.26:36540) 2022-07-20T02:26:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 -- 172.21.15.26:0/3174257855 learned_addr learned my addr 172.21.15.26:0/3174257855 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7188fc700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140afe40 0x7fb7140ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 -- 172.21.15.26:0/3174257855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140afe40 msgr2=0x7fb7140ab010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140afe40 0x7fb7140ab010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 -- 172.21.15.26:0/3174257855 >> v1:172.21.15.120:6789/0 conn(0x7fb7140b5d50 legacy=0x7fb714144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 -- 172.21.15.26:0/3174257855 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7140abcb0 con 0x7fb7140ac150 2022-07-20T02:26:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140ac150 0x7fb7140aab20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb6fc002f70 tx=0x7fb6fc006eb0).ready entity=mon.0 client_cookie=1ed6e840e2f8c713 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7137fe700 1 -- 172.21.15.26:0/3174257855 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6fc00d5c0 con 0x7fb7140ac150 2022-07-20T02:26:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7137fe700 1 -- 172.21.15.26:0/3174257855 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc000e90 con 0x7fb7140ac150 2022-07-20T02:26:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7137fe700 1 -- 172.21.15.26:0/3174257855 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6fc00d950 con 0x7fb7140ac150 2022-07-20T02:26:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- 172.21.15.26:0/3174257855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140ac150 msgr2=0x7fb7140aab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140ac150 0x7fb7140aab20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb6fc002f70 tx=0x7fb6fc006eb0).stop 2022-07-20T02:26:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- 172.21.15.26:0/3174257855 shutdown_connections 2022-07-20T02:26:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140afe40 0x7fb7140ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- 172.21.15.26:0/3174257855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140ac150 0x7fb7140aab20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- 172.21.15.26:0/3174257855 >> 172.21.15.26:0/3174257855 conn(0x7fb714060310 msgr2=0x7fb7140b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- 172.21.15.26:0/3174257855 shutdown_connections 2022-07-20T02:26:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- 172.21.15.26:0/3174257855 wait complete. 2022-07-20T02:26:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 Processor -- start 2022-07-20T02:26:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- start start 2022-07-20T02:26:32.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7140ac150 0x7fb714066e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 0x7fb71413bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140ae010 con 0x7fb7140afe40 2022-07-20T02:26:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb714155cc0 con 0x7fb7140ac150 2022-07-20T02:26:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb7190fd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140ae240 con 0x7fb7140b5d50 2022-07-20T02:26:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36548/0 (socket says 172.21.15.26:36548) 2022-07-20T02:26:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.780+0000 7fb70bfff700 1 -- 172.21.15.26:0/408292904 learned_addr learned my addr 172.21.15.26:0/408292904 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb713fff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7140ac150 0x7fb714066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb7188fc700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 0x7fb71413bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 -- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 msgr2=0x7fb71413bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 0x7fb71413bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 -- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7140ac150 msgr2=0x7fb714066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7140ac150 0x7fb714066e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 -- 172.21.15.26:0/408292904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc00b1e0 con 0x7fb7140afe40 2022-07-20T02:26:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb7188fc700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 0x7fb71413bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb70bfff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fb6fc002f70 tx=0x7fb6fc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6fc011950 con 0x7fb7140afe40 2022-07-20T02:26:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc011ab0 con 0x7fb7140afe40 2022-07-20T02:26:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7140ae850 con 0x7fb7140afe40 2022-07-20T02:26:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6fc011e10 con 0x7fb7140afe40 2022-07-20T02:26:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71413b8c0 con 0x7fb7140afe40 2022-07-20T02:26:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71405cff0 con 0x7fb7140afe40 2022-07-20T02:26:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.784+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb6fc025030 con 0x7fb7140afe40 2022-07-20T02:26:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.788+0000 7fb711ffb700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 0x7fb6f8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.788+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb6fc06d300 con 0x7fb7140afe40 2022-07-20T02:26:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.788+0000 7fb713fff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 0x7fb6f8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.788+0000 7fb713fff700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 0x7fb6f8055e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb70400b000 tx=0x7fb704009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.792+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6fc0440c0 con 0x7fb7140afe40 2022-07-20T02:26:32.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7140574e0 con 0x7fb7140afe40 2022-07-20T02:26:32.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb711ffb700 1 -- 172.21.15.26:0/408292904 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7fb6fc023090 con 0x7fb7140afe40 2022-07-20T02:26:32.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:32.970 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:32.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 msgr2=0x7fb6f8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 0x7fb6f8055e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb70400b000 tx=0x7fb704009040).stop 2022-07-20T02:26:32.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 msgr2=0x7fb7141563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:32.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fb6fc002f70 tx=0x7fb6fc015000).stop 2022-07-20T02:26:32.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 shutdown_connections 2022-07-20T02:26:32.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7140afe40 0x7fb7141563c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6f8053950 0x7fb6f8055e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7140b5d50 0x7fb71413bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 --2- 172.21.15.26:0/408292904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7140ac150 0x7fb714066e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:32.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 >> 172.21.15.26:0/408292904 conn(0x7fb714060310 msgr2=0x7fb7140b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:32.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 shutdown_connections 2022-07-20T02:26:32.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:32.960+0000 7fb7190fd700 1 -- 172.21.15.26:0/408292904 wait complete. 2022-07-20T02:26:32.977 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:32.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T02:26:33.097 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T02:26:33.098 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T02:26:33.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:26:33.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.200+0000 7fc6aeddb700 1 Processor -- start 2022-07-20T02:26:33.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.200+0000 7fc6aeddb700 1 -- start start 2022-07-20T02:26:33.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6a805cfd0 0x7fc6a8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 0x7fc6a80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a81474e0 con 0x7fc6a8060f00 2022-07-20T02:26:33.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a805d3b0 con 0x7fc6a805cfd0 2022-07-20T02:26:33.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a81431e0 con 0x7fc6a8147000 2022-07-20T02:26:33.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6a805cfd0 0x7fc6a8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6a805cfd0 0x7fc6a8143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36556/0 (socket says 172.21.15.26:36556) 2022-07-20T02:26:33.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 -- 172.21.15.26:0/316567942 learned_addr learned my addr 172.21.15.26:0/316567942 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6ad5d8700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 0x7fc6a80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6acdd7700 1 -- 172.21.15.26:0/316567942 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920570302 0 0) 0x7fc6a81474e0 con 0x7fc6a8060f00 2022-07-20T02:26:33.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6acdd7700 1 -- 172.21.15.26:0/316567942 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc690003260 con 0x7fc6a8060f00 2022-07-20T02:26:33.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 -- 172.21.15.26:0/316567942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 msgr2=0x7fc6a80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 0x7fc6a80b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 -- 172.21.15.26:0/316567942 >> v1:172.21.15.120:6789/0 conn(0x7fc6a8060f00 legacy=0x7fc6a813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 -- 172.21.15.26:0/316567942 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8143480 con 0x7fc6a805cfd0 2022-07-20T02:26:33.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6ad5d8700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 0x7fc6a80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:26:33.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6addd9700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a805cfd0 0x7fc6a8143bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fc6a4002f70 tx=0x7fc6a4006eb0).ready entity=mon.0 client_cookie=24214bd87b705e71 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6acdd7700 1 -- 172.21.15.26:0/316567942 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a400d5c0 con 0x7fc6a805cfd0 2022-07-20T02:26:33.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6acdd7700 1 -- 172.21.15.26:0/316567942 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a4000e90 con 0x7fc6a805cfd0 2022-07-20T02:26:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6acdd7700 1 -- 172.21.15.26:0/316567942 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a400d950 con 0x7fc6a805cfd0 2022-07-20T02:26:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- 172.21.15.26:0/316567942 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a805cfd0 msgr2=0x7fc6a8143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a805cfd0 0x7fc6a8143bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fc6a4002f70 tx=0x7fc6a4006eb0).stop 2022-07-20T02:26:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- 172.21.15.26:0/316567942 shutdown_connections 2022-07-20T02:26:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8147000 0x7fc6a80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/316567942 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a805cfd0 0x7fc6a8143bc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- 172.21.15.26:0/316567942 >> 172.21.15.26:0/316567942 conn(0x7fc6a80620c0 msgr2=0x7fc6a805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- 172.21.15.26:0/316567942 shutdown_connections 2022-07-20T02:26:33.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.204+0000 7fc6aeddb700 1 -- 172.21.15.26:0/316567942 wait complete. 2022-07-20T02:26:33.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 Processor -- start 2022-07-20T02:26:33.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- start start 2022-07-20T02:26:33.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8060f00 0x7fc6a80b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a8147000 0x7fc6a81444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80b9170 con 0x7fc6a8145520 2022-07-20T02:26:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80b3d40 con 0x7fc6a8147000 2022-07-20T02:26:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8152fe0 con 0x7fc6a8060f00 2022-07-20T02:26:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36564/0 (socket says 172.21.15.26:36564) 2022-07-20T02:26:33.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 -- 172.21.15.26:0/4270613555 learned_addr learned my addr 172.21.15.26:0/4270613555 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6addd9700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8060f00 0x7fc6a80b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ad5d8700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a8147000 0x7fc6a81444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 -- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8060f00 msgr2=0x7fc6a80b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8060f00 0x7fc6a80b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 -- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a8147000 msgr2=0x7fc6a81444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a8147000 0x7fc6a81444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 -- 172.21.15.26:0/4270613555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69800b180 con 0x7fc6a8145520 2022-07-20T02:26:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6ae5da700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fc6a80b3e80 tx=0x7fc6a0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a0002310 con 0x7fc6a8145520 2022-07-20T02:26:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a0002470 con 0x7fc6a8145520 2022-07-20T02:26:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a0001840 con 0x7fc6a8145520 2022-07-20T02:26:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a400b1e0 con 0x7fc6a8145520 2022-07-20T02:26:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.208+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a80b23f0 con 0x7fc6a8145520 2022-07-20T02:26:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8143720 con 0x7fc6a8145520 2022-07-20T02:26:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc6a0019020 con 0x7fc6a8145520 2022-07-20T02:26:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc696ffd700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 0x7fc688055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc6a006beb0 con 0x7fc6a8145520 2022-07-20T02:26:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc6addd9700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 0x7fc688055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.212+0000 7fc6addd9700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 0x7fc688055ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc6a401c000 tx=0x7fc6a4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.216+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6a0042b70 con 0x7fc6a8145520 2022-07-20T02:26:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.376+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6a80574e0 con 0x7fc6a8145520 2022-07-20T02:26:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc696ffd700 1 -- 172.21.15.26:0/4270613555 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7fc6a0046190 con 0x7fc6a8145520 2022-07-20T02:26:33.387 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:26:33.387 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:26:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 msgr2=0x7fc688055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 0x7fc688055ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc6a401c000 tx=0x7fc6a4015000).stop 2022-07-20T02:26:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 msgr2=0x7fc6a80afc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fc6a80b3e80 tx=0x7fc6a0007e10).stop 2022-07-20T02:26:33.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 shutdown_connections 2022-07-20T02:26:33.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6a8145520 0x7fc6a80afc80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc688053a10 0x7fc688055ed0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6a8147000 0x7fc6a81444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 --2- 172.21.15.26:0/4270613555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6a8060f00 0x7fc6a80b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 >> 172.21.15.26:0/4270613555 conn(0x7fc6a805cfd0 msgr2=0x7fc6a80b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 shutdown_connections 2022-07-20T02:26:33.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.380+0000 7fc6aeddb700 1 -- 172.21.15.26:0/4270613555 wait complete. 2022-07-20T02:26:33.399 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:26:33.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-07-20T02:26:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 Processor -- start 2022-07-20T02:26:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 -- start start 2022-07-20T02:26:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 0x7f8cc8067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cc8143070 0x7f8cc81434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc8146750 con 0x7f8cc8140320 2022-07-20T02:26:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc81469f0 con 0x7f8cc8143070 2022-07-20T02:26:33.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.528+0000 7f8cc7fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc8146c90 con 0x7f8cc80b5d50 2022-07-20T02:26:33.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cc8143070 0x7f8cc81434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc77fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8cc8140320 0x7f8cc81407d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35852/0 (socket says 172.21.15.26:35852) 2022-07-20T02:26:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc77fe700 1 -- 172.21.15.26:0/3632887344 learned_addr learned my addr 172.21.15.26:0/3632887344 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:26:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/3632887344 >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 0x7f8cc8067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3632887344 >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 msgr2=0x7f8cc8067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/3632887344 >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 0x7f8cc8067490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3632887344 >> v1:172.21.15.26:6789/0 conn(0x7f8cc8140320 legacy=0x7f8cc81407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3632887344 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8146f30 con 0x7f8cc8143070 2022-07-20T02:26:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/3632887344 >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 0x7f8cc8067490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/3632887344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8143070 0x7f8cc81434c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f8cc8066d30 tx=0x7f8cc00091d0).ready entity=mon.0 client_cookie=c7cccbd88d861384 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc5ffb700 1 -- 172.21.15.26:0/3632887344 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0009ea0 con 0x7f8cc8143070 2022-07-20T02:26:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc5ffb700 1 -- 172.21.15.26:0/3632887344 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc000b860 con 0x7f8cc8143070 2022-07-20T02:26:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc5ffb700 1 -- 172.21.15.26:0/3632887344 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc000b9d0 con 0x7f8cc8143070 2022-07-20T02:26:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- 172.21.15.26:0/3632887344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8143070 msgr2=0x7f8cc81434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/3632887344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8143070 0x7f8cc81434c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f8cc8066d30 tx=0x7f8cc00091d0).stop 2022-07-20T02:26:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- 172.21.15.26:0/3632887344 shutdown_connections 2022-07-20T02:26:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/3632887344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8143070 0x7f8cc81434c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/3632887344 >> v2:172.21.15.120:3300/0 conn(0x7f8cc80b5d50 0x7f8cc8067490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- 172.21.15.26:0/3632887344 >> 172.21.15.26:0/3632887344 conn(0x7f8cc8060310 msgr2=0x7f8cc80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- 172.21.15.26:0/3632887344 shutdown_connections 2022-07-20T02:26:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- 172.21.15.26:0/3632887344 wait complete. 2022-07-20T02:26:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 Processor -- start 2022-07-20T02:26:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.532+0000 7f8cc7fff700 1 -- start start 2022-07-20T02:26:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 0x7f8cc80abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc8143070 0x7f8cc8141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80aaeb0 con 0x7f8cc8140320 2022-07-20T02:26:33.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80aa230 con 0x7f8cc8143070 2022-07-20T02:26:33.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80aa4d0 con 0x7f8cc80b5d50 2022-07-20T02:26:33.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36576/0 (socket says 172.21.15.26:36576) 2022-07-20T02:26:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 -- 172.21.15.26:0/216905110 learned_addr learned my addr 172.21.15.26:0/216905110 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 0x7f8cc80abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc8143070 0x7f8cc8141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 -- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 msgr2=0x7f8cc80abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 0x7f8cc80abcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 -- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc8143070 msgr2=0x7f8cc8141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc8143070 0x7f8cc8141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 -- 172.21.15.26:0/216905110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0009470 con 0x7f8cc8140320 2022-07-20T02:26:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 0x7f8cc80abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8cc8066d30 tx=0x7f8cc0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cbffff700 1 -- 172.21.15.26:0/216905110 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0021630 con 0x7f8cc8140320 2022-07-20T02:26:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cbffff700 1 -- 172.21.15.26:0/216905110 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0021790 con 0x7f8cc8140320 2022-07-20T02:26:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8145060 con 0x7f8cc8140320 2022-07-20T02:26:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cbffff700 1 -- 172.21.15.26:0/216905110 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0021af0 con 0x7f8cc8140320 2022-07-20T02:26:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.536+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc8145500 con 0x7f8cc8140320 2022-07-20T02:26:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cbffff700 1 -- 172.21.15.26:0/216905110 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8cc000f070 con 0x7f8cc8140320 2022-07-20T02:26:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cbffff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 0x7f8cac055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cbffff700 1 -- 172.21.15.26:0/216905110 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8cc006d760 con 0x7f8cc8140320 2022-07-20T02:26:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 0x7f8cac055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 0x7f8cac055e40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8cb000c000 tx=0x7f8cb0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 Processor -- start 2022-07-20T02:26:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 -- start start 2022-07-20T02:26:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x140df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x140c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x134b010 con 0x140e410 2022-07-20T02:26:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x132b250 con 0x1413bc0 2022-07-20T02:26:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x13d9170 con 0x140c330 2022-07-20T02:26:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36582/0 (socket says 172.21.15.26:36582) 2022-07-20T02:26:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc67fc700 1 -- 172.21.15.26:0/423346255 learned_addr learned my addr 172.21.15.26:0/423346255 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x140df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc67fc700 1 -- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 msgr2=0x140c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.540+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x140c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x140c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 -- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 msgr2=0x140df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x140df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 -- 172.21.15.26:0/423346255 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1383030 con 0x140e410 2022-07-20T02:26:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x140df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x140c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:26:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x13db240 tx=0x7f8cc0012000).ready entity=mon.0 client_cookie=71cd3aef6a7d54b5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc7fff700 1 -- 172.21.15.26:0/423346255 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc00476d0 con 0x140e410 2022-07-20T02:26:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc7fff700 1 -- 172.21.15.26:0/423346255 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0047830 con 0x140e410 2022-07-20T02:26:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc7fff700 1 -- 172.21.15.26:0/423346255 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc00479a0 con 0x140e410 2022-07-20T02:26:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- 172.21.15.26:0/423346255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 msgr2=0x140e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x13db240 tx=0x7f8cc0012000).stop 2022-07-20T02:26:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- 172.21.15.26:0/423346255 shutdown_connections 2022-07-20T02:26:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x140e870 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x140c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- 172.21.15.26:0/423346255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x140df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- 172.21.15.26:0/423346255 >> 172.21.15.26:0/423346255 conn(0x13d8540 msgr2=0x1413150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- 172.21.15.26:0/423346255 shutdown_connections 2022-07-20T02:26:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- 172.21.15.26:0/423346255 wait complete. 2022-07-20T02:26:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- start start 2022-07-20T02:26:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 Processor -- start 2022-07-20T02:26:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140c330 0x13d6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x1413bc0 0x14144c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x13d6600 con 0x140e410 2022-07-20T02:26:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x13d97f0 con 0x140c330 2022-07-20T02:26:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x13d9a10 con 0x1413bc0 2022-07-20T02:26:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cd750c740 10 client.0 ll_register_callbacks cb 0x1392610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:26:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36590/0 (socket says 172.21.15.26:36590) 2022-07-20T02:26:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3503209600 learned_addr learned my addr 172.21.15.26:0/3503209600 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.544+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140c330 0x13d6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x1413bc0 0x14144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x1413bc0 msgr2=0x14144c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x1413bc0 0x14144c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140c330 msgr2=0x13d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140c330 0x13d6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 -- 172.21.15.26:0/3503209600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc8145500 con 0x140e410 2022-07-20T02:26:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x13d9930 tx=0x7f8cc0047e50).ready entity=mon.0 client_cookie=1f1f1e166b649794 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:26:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0005a70 con 0x140e410 2022-07-20T02:26:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8cc0005bd0 con 0x140e410 2022-07-20T02:26:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1340fc0 con 0x140e410 2022-07-20T02:26:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cc0005d90 con 0x140e410 2022-07-20T02:26:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 10 client.5118 fetch_fsmap learned FSMap version 13 2022-07-20T02:26:33.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x131d9b0 con 0x140e410 2022-07-20T02:26:33.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1340fc0 con 0x140e410 2022-07-20T02:26:33.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 13) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f8cc0046840 con 0x140e410 2022-07-20T02:26:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 10 client.5118 fetch_fsmap finished waiting for FSMap version 13 2022-07-20T02:26:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 10 client.5118 resolve_mds: resolved ID 'a' to GID 4881 2022-07-20T02:26:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 20 client.5118 populate_metadata read hostname 'smithi026' 2022-07-20T02:26:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1407e00 0x1409f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 4 client.5118 mds_command: new command op to 4881 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T02:26:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 --> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1339040 con 0x1407e00 2022-07-20T02:26:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1407e00 0x1409f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 10 client.5118 ms_handle_connect on v2:172.21.15.26:6834/2813102162 2022-07-20T02:26:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1407e00 0x1409f00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7031e961f8de234 server_cookie=249dec0ee572508c in_seq=0 out_seq=0 2022-07-20T02:26:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 1 -- 172.21.15.26:0/3503209600 <== mds.0 v2:172.21.15.26:6834/2813102162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x1339040 con 0x1407e00 2022-07-20T02:26:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 10 client.5118 handle_command_reply: tid=0 2022-07-20T02:26:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 client.5118 shutdown 2022-07-20T02:26:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 msgr2=0x13d9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x13d9930 tx=0x7f8cc0047e50).stop 2022-07-20T02:26:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 shutdown_connections 2022-07-20T02:26:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1407e00 0x1409f00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140e410 0x13d9300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x1413bc0 0x14144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 --2- 172.21.15.26:0/3503209600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140c330 0x13d6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 >> 172.21.15.26:0/3503209600 conn(0x13d8540 msgr2=0x140e830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8ca5ffb700 0 client.5118 ms_handle_reset on v2:172.21.15.26:6834/2813102162 2022-07-20T02:26:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 shutdown_connections 2022-07-20T02:26:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 1 -- 172.21.15.26:0/3503209600 wait complete. 2022-07-20T02:26:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.548+0000 7f8cd750c740 20 client.5118 trim_cache size 0 max 16384 2022-07-20T02:26:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 Processor -- start 2022-07-20T02:26:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- start start 2022-07-20T02:26:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x1412cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 0x14131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 0x140f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x140b670 con 0x1413bc0 2022-07-20T02:26:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x134b010 con 0x140e410 2022-07-20T02:26:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x13a9510 con 0x140c330 2022-07-20T02:26:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 0x140f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 0x14131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 0x140f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36600/0 (socket says 172.21.15.26:36600) 2022-07-20T02:26:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc77fe700 1 -- 172.21.15.26:0/934259156 learned_addr learned my addr 172.21.15.26:0/934259156 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x1412cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 -- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 msgr2=0x1412cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x1412cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 -- 172.21.15.26:0/934259156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 msgr2=0x140f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 0x140f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 -- 172.21.15.26:0/934259156 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x133a5b0 con 0x140e410 2022-07-20T02:26:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 0x14131b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x13d97e0 tx=0x7f8cc0037000).ready entity=mon.1 client_cookie=f2a2421c2abf3b1c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8ca7fff700 1 -- 172.21.15.26:0/934259156 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc00476d0 con 0x140e410 2022-07-20T02:26:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8ca7fff700 1 -- 172.21.15.26:0/934259156 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0046e10 con 0x140e410 2022-07-20T02:26:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8ca7fff700 1 -- 172.21.15.26:0/934259156 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0048180 con 0x140e410 2022-07-20T02:26:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 msgr2=0x14131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:26:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 0x14131b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x13d97e0 tx=0x7f8cc0037000).stop 2022-07-20T02:26:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- 172.21.15.26:0/934259156 shutdown_connections 2022-07-20T02:26:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x1413bc0 0x140f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x140e410 0x14131b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 --2- 172.21.15.26:0/934259156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140c330 0x1412cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- 172.21.15.26:0/934259156 >> 172.21.15.26:0/934259156 conn(0x13d8540 msgr2=0x13d4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:26:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- 172.21.15.26:0/934259156 shutdown_connections 2022-07-20T02:26:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- 172.21.15.26:0/934259156 wait complete. 2022-07-20T02:26:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 -- start start 2022-07-20T02:26:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.564+0000 7f8cd750c740 1 Processor -- start 2022-07-20T02:26:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140e410 0x146e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x146e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x132b250 con 0x140c330 2022-07-20T02:26:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x1475310 con 0x1413bc0 2022-07-20T02:26:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x1416150 con 0x140e410 2022-07-20T02:26:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 10 client.0 ll_register_callbacks cb 0x13d9970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:26:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36602/0 (socket says 172.21.15.26:36602) 2022-07-20T02:26:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 -- 172.21.15.26:0/106102617 learned_addr learned my addr 172.21.15.26:0/106102617 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:26:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x146e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140e410 0x146e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 -- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140e410 msgr2=0x146e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140e410 0x146e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 -- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 msgr2=0x146e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:26:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x146e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:26:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 -- 172.21.15.26:0/106102617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x131d9b0 con 0x140c330 2022-07-20T02:26:33.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc77fe700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x146e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:26:33.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:26:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc6ffd700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x1418090 tx=0x7f8cb0014000).ready entity=mon.0 client_cookie=918c1d61c58c052b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:26:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb0029070 con 0x140c330 2022-07-20T02:26:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8cb002fdb0 con 0x140c330 2022-07-20T02:26:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cc0047830 con 0x140c330 2022-07-20T02:26:33.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb00249b0 con 0x140c330 2022-07-20T02:26:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 10 client.5124 fetch_fsmap learned FSMap version 13 2022-07-20T02:26:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x133fb90 con 0x140c330 2022-07-20T02:26:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0047830 con 0x140c330 2022-07-20T02:26:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 13) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f8cb002c070 con 0x140c330 2022-07-20T02:26:33.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 10 client.5124 fetch_fsmap finished waiting for FSMap version 13 2022-07-20T02:26:33.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 10 client.5124 resolve_mds: resolved ID 'a' to GID 4881 2022-07-20T02:26:33.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 20 client.5124 populate_metadata read hostname 'smithi026' 2022-07-20T02:26:33.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1470f20 0x1471300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:26:33.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 4 client.5124 mds_command: new command op to 4881 tid=0[{"prefix": "cache drop"}] 2022-07-20T02:26:33.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 --> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x122e590 con 0x1470f20 2022-07-20T02:26:33.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1470f20 0x1471300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:26:33.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8cc67fc700 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1470f20 0x1471300 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=fd40ed986e5d8d7e server_cookie=4f36ff668a989efb in_seq=0 out_seq=0 2022-07-20T02:26:33.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:26:33.568+0000 7f8ca6ffd700 10 client.5124 ms_handle_connect on v2:172.21.15.26:6834/2813102162 2022-07-20T02:26:46.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:46.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:46.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:26:46.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:16.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:16.282 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:16.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:16.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:17.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8ca6ffd700 1 -- 172.21.15.26:0/106102617 <== mds.0 v2:172.21.15.26:6834/2813102162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0x122e590 con 0x1470f20 2022-07-20T02:27:17.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8ca6ffd700 10 client.5124 handle_command_reply: tid=0 2022-07-20T02:27:17.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 client.5124 shutdown 2022-07-20T02:27:17.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 msgr2=0x1474e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:17.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x1418090 tx=0x7f8cb0014000).stop 2022-07-20T02:27:17.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 shutdown_connections 2022-07-20T02:27:17.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:6834/2813102162,v1:172.21.15.26:6835/2813102162] conn(0x1470f20 0x1471300 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x1413bc0 0x146e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x140e410 0x146e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8ca6ffd700 0 client.5124 ms_handle_reset on v2:172.21.15.26:6834/2813102162 2022-07-20T02:27:17.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 --2- 172.21.15.26:0/106102617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x140c330 0x1474e20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 >> 172.21.15.26:0/106102617 conn(0x13d8540 msgr2=0x13dc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:17.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 shutdown_connections 2022-07-20T02:27:17.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 1 -- 172.21.15.26:0/106102617 wait complete. 2022-07-20T02:27:17.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cd750c740 20 client.5124 trim_cache size 0 max 16384 2022-07-20T02:27:17.932 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:27:17.932 INFO:teuthology.orchestra.run.smithi026.stdout: "client_recall": { 2022-07-20T02:27:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:27:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "(0) Success", 2022-07-20T02:27:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "recalled": 920 2022-07-20T02:27:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:17.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flush_journal": { 2022-07-20T02:27:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:27:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "" 2022-07-20T02:27:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:17.934 INFO:teuthology.orchestra.run.smithi026.stdout: "trim_cache": { 2022-07-20T02:27:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "trimmed": 900 2022-07-20T02:27:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2022-07-20T02:27:17.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": { 2022-07-20T02:27:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "items": 2489, 2022-07-20T02:27:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes": 431632 2022-07-20T02:27:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:17.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:17.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 44.354053176000001 2022-07-20T02:27:17.937 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:27:17.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 msgr2=0x7f8cac055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:17.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 0x7f8cac055e40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8cb000c000 tx=0x7f8cb0008040).stop 2022-07-20T02:27:17.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 msgr2=0x7f8cc80b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:17.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8cc8066d30 tx=0x7f8cc0016000).stop 2022-07-20T02:27:17.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 shutdown_connections 2022-07-20T02:27:17.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc8140320 0x7f8cc80b47c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8cac053980 0x7f8cac055e40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc8143070 0x7f8cc8141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 --2- 172.21.15.26:0/216905110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc80b5d50 0x7f8cc80abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:17.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 >> 172.21.15.26:0/216905110 conn(0x7f8cc8060310 msgr2=0x7f8cc80b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:17.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 shutdown_connections 2022-07-20T02:27:17.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:17.924+0000 7f8cc7fff700 1 -- 172.21.15.26:0/216905110 wait complete. 2022-07-20T02:27:17.942 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:27:17.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:17.960 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:17.983 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:18.005 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:18.027 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:27:18.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:18.028 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:18.029 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:18.038 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:27:18.039 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:27:18.040 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:27:18.041 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:27:18.041 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:18.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22301]: fuse finished with error 0 and tester_r 0 2022-07-20T02:27:18.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:18.103 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:18.103 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:18.111 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:27:18.114 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:27:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:27:18.214 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:18.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:18.226 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:18.227 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:27:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:27:18.242 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:27:18.243 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:27:18.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:27:18.244 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:27:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:27:18.263 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:27:18.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:27:18.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 Processor -- start 2022-07-20T02:27:18.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 -- start start 2022-07-20T02:27:18.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70805cfd0 0x7fb708143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb708060f00 0x7fb70813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7081474e0 con 0x7fb708147000 2022-07-20T02:27:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080b4e40 con 0x7fb708060f00 2022-07-20T02:27:18.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb707fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70805d3b0 con 0x7fb70805cfd0 2022-07-20T02:27:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.372+0000 7fb7077fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb708060f00 0x7fb70813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb708060f00 0x7fb70813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36616/0 (socket says 172.21.15.26:36616) 2022-07-20T02:27:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7067fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb708147000 0x7fb7080b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35892/0 (socket says 172.21.15.26:35892) 2022-07-20T02:27:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 -- 172.21.15.26:0/889964371 learned_addr learned my addr 172.21.15.26:0/889964371 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb706ffd700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70805cfd0 0x7fb708143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 -- 172.21.15.26:0/889964371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70805cfd0 msgr2=0x7fb708143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70805cfd0 0x7fb708143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 -- 172.21.15.26:0/889964371 >> v1:172.21.15.26:6789/0 conn(0x7fb708147000 legacy=0x7fb7080b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:18.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb705ffb700 1 -- 172.21.15.26:0/889964371 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195977792 0 0) 0x7fb7081474e0 con 0x7fb708147000 2022-07-20T02:27:18.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 -- 172.21.15.26:0/889964371 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7081431e0 con 0x7fb708060f00 2022-07-20T02:27:18.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb7077fe700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb708060f00 0x7fb70813c9b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fb7000032f0 tx=0x7fb700009290).ready entity=mon.0 client_cookie=eb3368dacb87e20d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:18.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb705ffb700 1 -- 172.21.15.26:0/889964371 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb70000b8a0 con 0x7fb708060f00 2022-07-20T02:27:18.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb705ffb700 1 -- 172.21.15.26:0/889964371 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70000ba00 con 0x7fb708060f00 2022-07-20T02:27:18.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb705ffb700 1 -- 172.21.15.26:0/889964371 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb70000bb70 con 0x7fb708060f00 2022-07-20T02:27:18.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 -- 172.21.15.26:0/889964371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb708060f00 msgr2=0x7fb70813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:18.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb708060f00 0x7fb70813c9b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fb7000032f0 tx=0x7fb700009290).stop 2022-07-20T02:27:18.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 -- 172.21.15.26:0/889964371 shutdown_connections 2022-07-20T02:27:18.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb708060f00 0x7fb70813c9b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:18.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 --2- 172.21.15.26:0/889964371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb70805cfd0 0x7fb708143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:18.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 -- 172.21.15.26:0/889964371 >> 172.21.15.26:0/889964371 conn(0x7fb7080620c0 msgr2=0x7fb70805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:18.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 -- 172.21.15.26:0/889964371 shutdown_connections 2022-07-20T02:27:18.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.376+0000 7fb707fff700 1 -- 172.21.15.26:0/889964371 wait complete. 2022-07-20T02:27:18.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 Processor -- start 2022-07-20T02:27:18.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- start start 2022-07-20T02:27:18.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 0x7fb708144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb708147000 0x7fb708141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb708142040 con 0x7fb7080ada10 2022-07-20T02:27:18.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb708146570 con 0x7fb708147000 2022-07-20T02:27:18.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb708152d50 con 0x7fb708060f00 2022-07-20T02:27:18.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36622/0 (socket says 172.21.15.26:36622) 2022-07-20T02:27:18.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 -- 172.21.15.26:0/3678812522 learned_addr learned my addr 172.21.15.26:0/3678812522 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:18.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb706ffd700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 0x7fb708144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7067fc700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb708147000 0x7fb708141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 -- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 msgr2=0x7fb708144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 0x7fb708144720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 -- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb708147000 msgr2=0x7fb708141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb708147000 0x7fb708141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 -- 172.21.15.26:0/3678812522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700009530 con 0x7fb7080ada10 2022-07-20T02:27:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb706ffd700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 0x7fb708144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb7077fe700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb7000032f0 tx=0x7fb700016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb70000bb70 con 0x7fb7080ada10 2022-07-20T02:27:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb700021630 con 0x7fb7080ada10 2022-07-20T02:27:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb708152ff0 con 0x7fb7080ada10 2022-07-20T02:27:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb70000b8a0 con 0x7fb7080ada10 2022-07-20T02:27:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb708146260 con 0x7fb7080ada10 2022-07-20T02:27:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.380+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708143480 con 0x7fb7080ada10 2022-07-20T02:27:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.384+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb70000f070 con 0x7fb7080ada10 2022-07-20T02:27:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.384+0000 7fb6f7fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 0x7fb6e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.384+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb70006d7b0 con 0x7fb7080ada10 2022-07-20T02:27:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.384+0000 7fb706ffd700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 0x7fb6e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.384+0000 7fb706ffd700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 0x7fb6e4055f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb6f800b000 tx=0x7fb6f8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.392+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb700044570 con 0x7fb7080ada10 2022-07-20T02:27:18.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.524+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb7080574e0 con 0x7fb7080ada10 2022-07-20T02:27:18.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.524+0000 7fb6f7fff700 1 -- 172.21.15.26:0/3678812522 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v34) v1 ==== 81+0+34085 (secure 0 0 0) 0x7fb700015020 con 0x7fb7080ada10 2022-07-20T02:27:18.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:18.532 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:27:18.532 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 34, 2022-07-20T02:27:18.532 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:27:18.532 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:27:18.533 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:26:15.835998+0000", 2022-07-20T02:27:18.533 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:27:18.533 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:27:18.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:27:18.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:27:18.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:27:18.534 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:27:18.534 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:27:18.535 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:27:18.535 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:27:18.535 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:18.535 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:27:18.535 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:27:18.536 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:27:18.536 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:27:18.536 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:27:18.537 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 6, 2022-07-20T02:27:18.537 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:27:18.537 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:27:18.537 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:27:18.538 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:27:18.538 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:27:18.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.538 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:27:18.539 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:27:18.539 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:27:18.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:18.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:18.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:18.540 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:18.540 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:18.540 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:18.540 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:18.541 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:18.542 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:27:18.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:27:18.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:27:18.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:27:18.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:27:18.544 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:18.544 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:18.544 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:18.544 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:18.544 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.545 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:27:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:18.546 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:18.547 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:18.555 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:18.555 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:18.556 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:18.557 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:18.558 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:18.559 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:18.560 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:18.561 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.562 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:18.563 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:18.564 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:18.565 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:18.566 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:27:18.567 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:27:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.568 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:27:18.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:27:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:27:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:27:18.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:18.572 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:18.573 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:27:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:27:18.574 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:27:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:27:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:27:18.575 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:18.576 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:18.577 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:18.578 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:18.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:27:18.580 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:18.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:18.582 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:18.583 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:18.584 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:18.585 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:18.586 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:18.587 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.588 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:18.589 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:18.590 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:18.591 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:18.592 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:18.593 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:27:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:27:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 5, 2022-07-20T02:27:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:27:18.596 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:26:12.967166+0000", 2022-07-20T02:27:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:18.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:18.598 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:18.599 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:18.600 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:27:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:27:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:27:18.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:27:18.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:27:18.603 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:18.604 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:18.605 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:18.606 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:18.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "33", 2022-07-20T02:27:18.608 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:18.609 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:18.611 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:18.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:18.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:18.614 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:18.615 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:18.618 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:18.619 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:27:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:27:18.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:27:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:27:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:18.621 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:27:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:27:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 6, 2022-07-20T02:27:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:27:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:26:13.835040+0000", 2022-07-20T02:27:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:27:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:27:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:27:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:27:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:27:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "34", 2022-07-20T02:27:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:18.635 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:18.636 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:18.637 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:18.638 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:27:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:27:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:27:18.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:27:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:27:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:27:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.642 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.643 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:27:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:27:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:27:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:27:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:27:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:27:18.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.653 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:27:18.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:27:18.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:18.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:27:18.656 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:27:18.657 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:27:18.657 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:27:18.657 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.657 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:27:18.659 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:27:18.659 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.659 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.659 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.660 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.660 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.661 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.661 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.661 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:27:18.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:27:18.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.673 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:27:18.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:27:18.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.680 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:27:18.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:27:18.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:27:18.698 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:27:18.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:18.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:27:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:27:18.700 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:27:18.701 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:27:18.701 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.701 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.701 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.702 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:27:18.703 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:27:18.703 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.703 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.703 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:27:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:27:18.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:27:18.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.708 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:27:18.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:27:18.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.714 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:27:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:27:18.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.718 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:27:18.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:27:18.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:18.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:27:18.724 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:27:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:27:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:27:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.725 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.726 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:27:18.727 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:27:18.727 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.727 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.727 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.727 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.728 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.728 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.728 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.729 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.729 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.729 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:27:18.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:27:18.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.732 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:27:18.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:27:18.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.736 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:27:18.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:27:18.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.740 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:27:18.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:27:18.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:18.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.743 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:27:18.743 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:27:18.743 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:27:18.744 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:27:18.744 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.744 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.744 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.745 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:27:18.745 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:27:18.746 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.746 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.746 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.746 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.747 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.747 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.747 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.747 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.754 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.754 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:27:18.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.758 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:27:18.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.761 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:27:18.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:27:18.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.767 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:27:18.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:27:18.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.772 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:27:18.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:27:18.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:18.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.780 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:27:18.780 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:27:18.781 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:27:18.781 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:27:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:27:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:27:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:27:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:27:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:27:18.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:27:18.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:27:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:27:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:27:18.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:27:18.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:18.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:27:18.820 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:27:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:27:18.821 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:27:18.822 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.823 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:27:18.826 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:27:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.827 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.828 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.829 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:27:18.830 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:27:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:27:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:27:18.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:27:18.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.840 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:27:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:27:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:27:18.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:27:18.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:27:18.854 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:27:18.855 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:27:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:27:18.856 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:27:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:27:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:18.860 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:18.861 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:18.862 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:18.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:18.864 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:18.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:27:18.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:27:18.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.877 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:27:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:27:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:27:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:27:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:27:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:27:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:27:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:27:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:27:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:27:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:27:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:27:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:27:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:27:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:27:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:27:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:27:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:27:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:27:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:27:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:27:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:27:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:27:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:27:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:27:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:27:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:27:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:27:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:27:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:27:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:27:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:27:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:27:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:27:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:27:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:27:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:27:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:27:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:27:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:27:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:27:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:27:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:27:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:27:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:27:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:27:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:27:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:27:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:27:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:27:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:27:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:27:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:27:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:27:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:27:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:27:19.022 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:27:19.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 msgr2=0x7fb6e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 0x7fb6e4055f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb6f800b000 tx=0x7fb6f8008040).stop 2022-07-20T02:27:19.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 msgr2=0x7fb7081424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb7000032f0 tx=0x7fb700016000).stop 2022-07-20T02:27:19.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 shutdown_connections 2022-07-20T02:27:19.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7080ada10 0x7fb7081424d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb6e4053a70 0x7fb6e4055f30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb708147000 0x7fb708141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 --2- 172.21.15.26:0/3678812522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb708060f00 0x7fb708144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 >> 172.21.15.26:0/3678812522 conn(0x7fb70805cfd0 msgr2=0x7fb70813a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:19.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 shutdown_connections 2022-07-20T02:27:19.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:18.528+0000 7fb707fff700 1 -- 172.21.15.26:0/3678812522 wait complete. 2022-07-20T02:27:19.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:19.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 Processor -- start 2022-07-20T02:27:19.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- start start 2022-07-20T02:27:19.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 0x7fd714098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd71405c580 0x7fd7140950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7140b0f20 con 0x7fd714143090 2022-07-20T02:27:19.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd714143470 con 0x7fd71405c580 2022-07-20T02:27:19.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71405fd90 con 0x7fd7140613b0 2022-07-20T02:27:19.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7137fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd714143090 0x7fd7140b0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35906/0 (socket says 172.21.15.26:35906) 2022-07-20T02:27:19.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7137fe700 1 -- 172.21.15.26:0/2416483155 learned_addr learned my addr 172.21.15.26:0/2416483155 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:19.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 --2- 172.21.15.26:0/2416483155 >> v2:172.21.15.26:3300/0 conn(0x7fd71405c580 0x7fd7140950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd712ffd700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 0x7fd714098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7127fc700 1 -- 172.21.15.26:0/2416483155 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719106880 0 0) 0x7fd7140b0f20 con 0x7fd714143090 2022-07-20T02:27:19.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7127fc700 1 -- 172.21.15.26:0/2416483155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd700003260 con 0x7fd714143090 2022-07-20T02:27:19.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 -- 172.21.15.26:0/2416483155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 msgr2=0x7fd714098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:19.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 0x7fd714098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 -- 172.21.15.26:0/2416483155 >> v1:172.21.15.26:6789/0 conn(0x7fd714143090 legacy=0x7fd7140b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 -- 172.21.15.26:0/2416483155 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714097d50 con 0x7fd71405c580 2022-07-20T02:27:19.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd712ffd700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 0x7fd714098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:19.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd713fff700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd71405c580 0x7fd7140950e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd708001110 tx=0x7fd704006df0).ready entity=mon.0 client_cookie=57aafb78d61cfc80 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7127fc700 1 -- 172.21.15.26:0/2416483155 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd70400d4a0 con 0x7fd71405c580 2022-07-20T02:27:19.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7127fc700 1 -- 172.21.15.26:0/2416483155 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704000e90 con 0x7fd71405c580 2022-07-20T02:27:19.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd7127fc700 1 -- 172.21.15.26:0/2416483155 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd70400d7f0 con 0x7fd71405c580 2022-07-20T02:27:19.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- 172.21.15.26:0/2416483155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd71405c580 msgr2=0x7fd7140950e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd71405c580 0x7fd7140950e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd708001110 tx=0x7fd704006df0).stop 2022-07-20T02:27:19.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- 172.21.15.26:0/2416483155 shutdown_connections 2022-07-20T02:27:19.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd71405c580 0x7fd7140950e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 --2- 172.21.15.26:0/2416483155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7140613b0 0x7fd714098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.156+0000 7fd719d9c700 1 -- 172.21.15.26:0/2416483155 >> 172.21.15.26:0/2416483155 conn(0x7fd7140b5e70 msgr2=0x7fd7140b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:19.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- 172.21.15.26:0/2416483155 shutdown_connections 2022-07-20T02:27:19.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- 172.21.15.26:0/2416483155 wait complete. 2022-07-20T02:27:19.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 Processor -- start 2022-07-20T02:27:19.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- start start 2022-07-20T02:27:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71405c580 0x7fd714153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7140613b0 0x7fd7141522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7141543d0 con 0x7fd714143090 2022-07-20T02:27:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71409b0d0 con 0x7fd7140613b0 2022-07-20T02:27:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.160+0000 7fd719d9c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71409b370 con 0x7fd71405c580 2022-07-20T02:27:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd7137fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71405c580 0x7fd714153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36636/0 (socket says 172.21.15.26:36636) 2022-07-20T02:27:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 -- 172.21.15.26:0/890605161 learned_addr learned my addr 172.21.15.26:0/890605161 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd712ffd700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7140613b0 0x7fd7141522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 -- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71405c580 msgr2=0x7fd714153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71405c580 0x7fd714153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 -- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7140613b0 msgr2=0x7fd7141522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7140613b0 0x7fd7141522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 -- 172.21.15.26:0/890605161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c009470 con 0x7fd714143090 2022-07-20T02:27:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd713fff700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fd708001110 tx=0x7fd704014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd704012930 con 0x7fd714143090 2022-07-20T02:27:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704012a90 con 0x7fd714143090 2022-07-20T02:27:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70400b180 con 0x7fd714143090 2022-07-20T02:27:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd704012df0 con 0x7fd714143090 2022-07-20T02:27:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71409bc60 con 0x7fd714143090 2022-07-20T02:27:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.164+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71409b530 con 0x7fd714143090 2022-07-20T02:27:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.168+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd70401c070 con 0x7fd714143090 2022-07-20T02:27:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.168+0000 7fd710ff9700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 0x7fd6f4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.168+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd70406cd10 con 0x7fd714143090 2022-07-20T02:27:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.168+0000 7fd7137fe700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 0x7fd6f4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.168+0000 7fd7137fe700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 0x7fd6f4055e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd70800d000 tx=0x7fd708009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.176+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd704072020 con 0x7fd714143090 2022-07-20T02:27:19.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.340+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd714097ff0 con 0x7fd714143090 2022-07-20T02:27:19.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.340+0000 7fd710ff9700 1 -- 172.21.15.26:0/890605161 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2491 (secure 0 0 0) 0x7fd704019090 con 0x7fd714143090 2022-07-20T02:27:19.352 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:19.352 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:26:15.835906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4881":{"gid":4881,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6835/2813102162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2813102162},{"type":"v1","addr":"172.21.15.26:6835","nonce":2813102162}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:27:19.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 msgr2=0x7fd6f4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 0x7fd6f4055e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd70800d000 tx=0x7fd708009040).stop 2022-07-20T02:27:19.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 msgr2=0x7fd7141527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fd708001110 tx=0x7fd704014000).stop 2022-07-20T02:27:19.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 shutdown_connections 2022-07-20T02:27:19.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd714143090 0x7fd7141527a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6f4053950 0x7fd6f4055e10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7140613b0 0x7fd7141522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 --2- 172.21.15.26:0/890605161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71405c580 0x7fd714153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 >> 172.21.15.26:0/890605161 conn(0x7fd7140b5e70 msgr2=0x7fd7140b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:19.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 shutdown_connections 2022-07-20T02:27:19.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.344+0000 7fd719d9c700 1 -- 172.21.15.26:0/890605161 wait complete. 2022-07-20T02:27:19.376 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2022-07-20T02:27:19.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:27:19.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 Processor -- start 2022-07-20T02:27:19.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 -- start start 2022-07-20T02:27:19.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 0x7f80f0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f01474e0 con 0x7f80f0060f00 2022-07-20T02:27:19.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f005d3b0 con 0x7f80f005cfd0 2022-07-20T02:27:19.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f5472700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01431e0 con 0x7f80f0147000 2022-07-20T02:27:19.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:58158/0 (socket says 172.21.15.26:58158) 2022-07-20T02:27:19.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80effff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 0x7f80f0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80f4c71700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f80f0060f00 0x7f80f013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:50134/0 (socket says 172.21.15.26:50134) 2022-07-20T02:27:19.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 -- 172.21.15.26:0/4123593211 learned_addr learned my addr 172.21.15.26:0/4123593211 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:19.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 -- 172.21.15.26:0/4123593211 >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 msgr2=0x7f80f0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:19.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 --2- 172.21.15.26:0/4123593211 >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 0x7f80f0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 -- 172.21.15.26:0/4123593211 >> v1:172.21.15.120:6789/0 conn(0x7f80f0060f00 legacy=0x7f80f013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 -- 172.21.15.26:0/4123593211 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0143480 con 0x7f80f0147000 2022-07-20T02:27:19.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80effff700 1 --2- 172.21.15.26:0/4123593211 >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 0x7f80f0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:19.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80ef7fe700 1 --2- 172.21.15.26:0/4123593211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f80e4002f70 tx=0x7f80e4006eb0).ready entity=mon.2 client_cookie=5a7c783859810d3f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80eeffd700 1 -- 172.21.15.26:0/4123593211 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e400d570 con 0x7f80f0147000 2022-07-20T02:27:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80eeffd700 1 -- 172.21.15.26:0/4123593211 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4000e90 con 0x7f80f0147000 2022-07-20T02:27:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.512+0000 7f80eeffd700 1 -- 172.21.15.26:0/4123593211 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e400d900 con 0x7f80f0147000 2022-07-20T02:27:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- 172.21.15.26:0/4123593211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 msgr2=0x7f80f00b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- 172.21.15.26:0/4123593211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f80e4002f70 tx=0x7f80e4006eb0).stop 2022-07-20T02:27:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- 172.21.15.26:0/4123593211 shutdown_connections 2022-07-20T02:27:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- 172.21.15.26:0/4123593211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0147000 0x7f80f00b4730 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- 172.21.15.26:0/4123593211 >> v2:172.21.15.120:3300/0 conn(0x7f80f005cfd0 0x7f80f0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- 172.21.15.26:0/4123593211 >> 172.21.15.26:0/4123593211 conn(0x7f80f00620c0 msgr2=0x7f80f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- 172.21.15.26:0/4123593211 shutdown_connections 2022-07-20T02:27:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- 172.21.15.26:0/4123593211 wait complete. 2022-07-20T02:27:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 Processor -- start 2022-07-20T02:27:19.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- start start 2022-07-20T02:27:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0060f00 0x7f80f01416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80f0147000 0x7f80f0152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f00adeb0 con 0x7f80f00b86c0 2022-07-20T02:27:19.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f00ad710 con 0x7f80f0147000 2022-07-20T02:27:19.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f5472700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f00ad970 con 0x7f80f0060f00 2022-07-20T02:27:19.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36650/0 (socket says 172.21.15.26:36650) 2022-07-20T02:27:19.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 -- 172.21.15.26:0/1094506662 learned_addr learned my addr 172.21.15.26:0/1094506662 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:19.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80ef7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80f0147000 0x7f80f0152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80effff700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0060f00 0x7f80f01416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 -- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0060f00 msgr2=0x7f80f01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0060f00 0x7f80f01416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 -- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80f0147000 msgr2=0x7f80f0152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:19.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.516+0000 7f80f4c71700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80f0147000 0x7f80f0152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80f4c71700 1 -- 172.21.15.26:0/1094506662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e000b180 con 0x7f80f00b86c0 2022-07-20T02:27:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80f4c71700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f80f0068660 tx=0x7f80e8007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e8005ae0 con 0x7f80f00b86c0 2022-07-20T02:27:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e8005c40 con 0x7f80f00b86c0 2022-07-20T02:27:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e400b1e0 con 0x7f80f00b86c0 2022-07-20T02:27:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e80070d0 con 0x7f80f00b86c0 2022-07-20T02:27:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f0146d30 con 0x7f80f00b86c0 2022-07-20T02:27:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f0143720 con 0x7f80f00b86c0 2022-07-20T02:27:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f80e801a060 con 0x7f80f00b86c0 2022-07-20T02:27:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 0x7f80d0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f80e806b9a0 con 0x7f80f00b86c0 2022-07-20T02:27:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.520+0000 7f80effff700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 0x7f80d0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:19.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.524+0000 7f80effff700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 0x7f80d0055f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f80e000b000 tx=0x7f80e0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:19.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.528+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80e8042760 con 0x7f80f00b86c0 2022-07-20T02:27:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:19.692+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f80f00574e0 con 0x7f80f00b86c0 2022-07-20T02:27:20.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:27:20.484+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:27:20.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.492+0000 7f80ed7fa700 1 -- 172.21.15.26:0/1094506662 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7f80e8018070 con 0x7f80f00b86c0 2022-07-20T02:27:20.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 msgr2=0x7f80d0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 0x7f80d0055f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f80e000b000 tx=0x7f80e0006df0).stop 2022-07-20T02:27:20.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 msgr2=0x7f80f00b8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f80f0068660 tx=0x7f80e8007f40).stop 2022-07-20T02:27:20.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 shutdown_connections 2022-07-20T02:27:20.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80f00b86c0 0x7f80f00b8b70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80d0053a70 0x7f80d0055f30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80f0147000 0x7f80f0152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 --2- 172.21.15.26:0/1094506662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80f0060f00 0x7f80f01416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 >> 172.21.15.26:0/1094506662 conn(0x7f80f005cfd0 msgr2=0x7f80f00b5990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:20.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 shutdown_connections 2022-07-20T02:27:20.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.496+0000 7f80f5472700 1 -- 172.21.15.26:0/1094506662 wait complete. 2022-07-20T02:27:20.511 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:27:20.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:20.565 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:27:20.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 Processor -- start 2022-07-20T02:27:20.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 -- start start 2022-07-20T02:27:20.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 0x7ff4f0098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff4f00b0810 0x7ff4f00950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f0143470 con 0x7ff4f00b0c50 2022-07-20T02:27:20.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f005fd90 con 0x7ff4f0143090 2022-07-20T02:27:20.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4f5777700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f0097d50 con 0x7ff4f00b0810 2022-07-20T02:27:20.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4eeffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 0x7ff4f0098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ef7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff4f00b0c50 0x7ff4f00955d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35950/0 (socket says 172.21.15.26:35950) 2022-07-20T02:27:20.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ef7fe700 1 -- 172.21.15.26:0/3467939229 learned_addr learned my addr 172.21.15.26:0/3467939229 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:20.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/3467939229 >> v2:172.21.15.120:3300/0 conn(0x7ff4f00b0810 0x7ff4f00950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/3467939229 >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 msgr2=0x7ff4f0098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:20.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/3467939229 >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 0x7ff4f0098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/3467939229 >> v1:172.21.15.26:6789/0 conn(0x7ff4f00b0c50 legacy=0x7ff4f00955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/3467939229 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0097ff0 con 0x7ff4f00b0810 2022-07-20T02:27:20.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.624+0000 7ff4edffb700 1 -- 172.21.15.26:0/3467939229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209978301 0 0) 0x7ff4f0143470 con 0x7ff4f00b0c50 2022-07-20T02:27:20.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4eeffd700 1 --2- 172.21.15.26:0/3467939229 >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 0x7ff4f0098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:20.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/3467939229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f00b0810 0x7ff4f00950e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff4e4002f70 tx=0x7ff4e4006df0).ready entity=mon.1 client_cookie=1b89461134899375 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:20.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4edffb700 1 -- 172.21.15.26:0/3467939229 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4e400d570 con 0x7ff4f00b0810 2022-07-20T02:27:20.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4edffb700 1 -- 172.21.15.26:0/3467939229 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e4000e90 con 0x7ff4f00b0810 2022-07-20T02:27:20.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4edffb700 1 -- 172.21.15.26:0/3467939229 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4e400d8c0 con 0x7ff4f00b0810 2022-07-20T02:27:20.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 -- 172.21.15.26:0/3467939229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f00b0810 msgr2=0x7ff4f00950e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 --2- 172.21.15.26:0/3467939229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f00b0810 0x7ff4f00950e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff4e4002f70 tx=0x7ff4e4006df0).stop 2022-07-20T02:27:20.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 -- 172.21.15.26:0/3467939229 shutdown_connections 2022-07-20T02:27:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 --2- 172.21.15.26:0/3467939229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f00b0810 0x7ff4f00950e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 --2- 172.21.15.26:0/3467939229 >> v2:172.21.15.26:3300/0 conn(0x7ff4f0143090 0x7ff4f0098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 -- 172.21.15.26:0/3467939229 >> 172.21.15.26:0/3467939229 conn(0x7ff4f00b5e70 msgr2=0x7ff4f00b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 -- 172.21.15.26:0/3467939229 shutdown_connections 2022-07-20T02:27:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 -- 172.21.15.26:0/3467939229 wait complete. 2022-07-20T02:27:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.628+0000 7ff4f5777700 1 Processor -- start 2022-07-20T02:27:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- start start 2022-07-20T02:27:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4f00b0810 0x7ff4f00610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f0143090 0x7ff4f0063e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0064c80 con 0x7ff4f00b0c50 2022-07-20T02:27:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f00ae980 con 0x7ff4f0143090 2022-07-20T02:27:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f00afa30 con 0x7ff4f00b0810 2022-07-20T02:27:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36674/0 (socket says 172.21.15.26:36674) 2022-07-20T02:27:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/412537757 learned_addr learned my addr 172.21.15.26:0/412537757 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ef7fe700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f0143090 0x7ff4f0063e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4eeffd700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4f00b0810 0x7ff4f00610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4f00b0810 msgr2=0x7ff4f00610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4f00b0810 0x7ff4f00610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f0143090 msgr2=0x7ff4f0063e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f0143090 0x7ff4f0063e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 -- 172.21.15.26:0/412537757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e000b180 con 0x7ff4f00b0c50 2022-07-20T02:27:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4ee7fc700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff4f0060dd0 tx=0x7ff4e4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4e400dea0 con 0x7ff4f00b0c50 2022-07-20T02:27:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e4013da0 con 0x7ff4f00b0c50 2022-07-20T02:27:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e400b180 con 0x7ff4f00b0c50 2022-07-20T02:27:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4e401e5f0 con 0x7ff4f00b0c50 2022-07-20T02:27:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.632+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f00aeda0 con 0x7ff4f00b0c50 2022-07-20T02:27:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f0066220 con 0x7ff4f00b0c50 2022-07-20T02:27:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff4e4025030 con 0x7ff4f00b0c50 2022-07-20T02:27:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4dffff700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 0x7ff4cc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6506+0+0 (secure 0 0 0) 0x7ff4e406cc60 con 0x7ff4f00b0c50 2022-07-20T02:27:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4eeffd700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 0x7ff4cc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.636+0000 7ff4eeffd700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 0x7ff4cc055e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff4e000b000 tx=0x7ff4e0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.640+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4f0066220 con 0x7ff4f00b0c50 2022-07-20T02:27:20.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4f0066220 con 0x7ff4f00b0c50 2022-07-20T02:27:20.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4dffff700 1 -- 172.21.15.26:0/412537757 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2130 (secure 0 0 0) 0x7ff4f0066220 con 0x7ff4f00b0c50 2022-07-20T02:27:20.808 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:20.808 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:26:14.833033+0000","modified":"2022-07-20T02:27:20.487584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T02:27:20.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 msgr2=0x7ff4cc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 0x7ff4cc055e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff4e000b000 tx=0x7ff4e0006df0).stop 2022-07-20T02:27:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 msgr2=0x7ff4f01501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff4f0060dd0 tx=0x7ff4e4014000).stop 2022-07-20T02:27:20.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 shutdown_connections 2022-07-20T02:27:20.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4f00b0c50 0x7ff4f01501f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff4cc053980 0x7ff4cc055e40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4f0143090 0x7ff4f0063e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 --2- 172.21.15.26:0/412537757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4f00b0810 0x7ff4f00610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:20.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 >> 172.21.15.26:0/412537757 conn(0x7ff4f00b5e70 msgr2=0x7ff4f0095520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:20.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 shutdown_connections 2022-07-20T02:27:20.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.804+0000 7ff4f5777700 1 -- 172.21.15.26:0/412537757 wait complete. 2022-07-20T02:27:20.818 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 14 2022-07-20T02:27:20.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:27:20.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 Processor -- start 2022-07-20T02:27:20.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 -- start start 2022-07-20T02:27:20.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f38a00b5d50 0x7f38a013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f38a00ac8e0 0x7f38a013d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:20.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a0140190 con 0x7f38a00b1900 2022-07-20T02:27:20.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a00af740 con 0x7f38a00b5d50 2022-07-20T02:27:20.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.972+0000 7f38a4b3a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a0140430 con 0x7f38a00ac8e0 2022-07-20T02:27:21.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f38a00b5d50 0x7f38a013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389effd700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f38a00b1900 0x7f38a00afe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:50172/0 (socket says 172.21.15.26:50172) 2022-07-20T02:27:21.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389effd700 1 -- 172.21.15.26:0/500196402 learned_addr learned my addr 172.21.15.26:0/500196402 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:21.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 --2- 172.21.15.26:0/500196402 >> v2:172.21.15.120:3300/0 conn(0x7f38a00ac8e0 0x7f38a013d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 -- 172.21.15.26:0/500196402 >> v2:172.21.15.26:3300/0 conn(0x7f38a00b5d50 msgr2=0x7f38a013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:21.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 --2- 172.21.15.26:0/500196402 >> v2:172.21.15.26:3300/0 conn(0x7f38a00b5d50 0x7f38a013d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 -- 172.21.15.26:0/500196402 >> v1:172.21.15.120:6789/0 conn(0x7f38a00b1900 legacy=0x7f38a00afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 -- 172.21.15.26:0/500196402 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a01406d0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389e7fc700 1 -- 172.21.15.26:0/500196402 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298969031 0 0) 0x7f38a0140190 con 0x7f38a00b1900 2022-07-20T02:27:21.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389ffff700 1 --2- 172.21.15.26:0/500196402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d7e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f38980032f0 tx=0x7f38980091d0).ready entity=mon.1 client_cookie=eb174556dd39536e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389e7fc700 1 -- 172.21.15.26:0/500196402 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f389800b680 con 0x7f38a00ac8e0 2022-07-20T02:27:21.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389e7fc700 1 -- 172.21.15.26:0/500196402 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389800b7e0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f389e7fc700 1 -- 172.21.15.26:0/500196402 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f389800b950 con 0x7f38a00ac8e0 2022-07-20T02:27:21.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- 172.21.15.26:0/500196402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00ac8e0 msgr2=0x7f38a013d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/500196402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d7e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f38980032f0 tx=0x7f38980091d0).stop 2022-07-20T02:27:21.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- 172.21.15.26:0/500196402 shutdown_connections 2022-07-20T02:27:21.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/500196402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d7e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/500196402 >> v2:172.21.15.26:3300/0 conn(0x7f38a00b5d50 0x7f38a013d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- 172.21.15.26:0/500196402 >> 172.21.15.26:0/500196402 conn(0x7f38a0060310 msgr2=0x7f38a00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:21.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- 172.21.15.26:0/500196402 shutdown_connections 2022-07-20T02:27:21.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- 172.21.15.26:0/500196402 wait complete. 2022-07-20T02:27:21.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 Processor -- start 2022-07-20T02:27:21.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 -- start start 2022-07-20T02:27:21.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 0x7f38a0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.976+0000 7f38a4b3a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00b5d50 0x7f38a0155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a00b4ba0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a0144070 con 0x7f38a00b5d50 2022-07-20T02:27:21.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a00b40a0 con 0x7f38a00b1900 2022-07-20T02:27:21.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36688/0 (socket says 172.21.15.26:36688) 2022-07-20T02:27:21.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00b5d50 0x7f38a0155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 -- 172.21.15.26:0/3685038462 learned_addr learned my addr 172.21.15.26:0/3685038462 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:21.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389effd700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 0x7f38a0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 -- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 msgr2=0x7f38a0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 0x7f38a0155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 -- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00b5d50 msgr2=0x7f38a0155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00b5d50 0x7f38a0155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 -- 172.21.15.26:0/3685038462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898009470 con 0x7f38a00ac8e0 2022-07-20T02:27:21.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389effd700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 0x7f38a0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:21.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389f7fe700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f389000c000 tx=0x7f3890006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3890011750 con 0x7f38a00ac8e0 2022-07-20T02:27:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38900118b0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a013e6f0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3890011c10 con 0x7f38a00ac8e0 2022-07-20T02:27:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a0134b20 con 0x7f38a00ac8e0 2022-07-20T02:27:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.980+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a01441e0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.984+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3890011d70 con 0x7f38a00ac8e0 2022-07-20T02:27:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.984+0000 7f389cff9700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 0x7f3884055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.984+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f389006c5d0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.984+0000 7f389effd700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 0x7f3884055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.984+0000 7f389effd700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 0x7f3884055e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f388800d000 tx=0x7f3888009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:20.988+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38a01441e0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.152+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f38a01441e0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.504+0000 7f389cff9700 1 -- 172.21.15.26:0/3685038462 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f38a01441e0 con 0x7f38a00ac8e0 2022-07-20T02:27:21.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 msgr2=0x7f3884055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 0x7f3884055e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f388800d000 tx=0x7f3888009040).stop 2022-07-20T02:27:21.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 msgr2=0x7f38a013d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f389000c000 tx=0x7f3890006df0).stop 2022-07-20T02:27:21.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 shutdown_connections 2022-07-20T02:27:21.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3884053950 0x7f3884055e10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38a00b5d50 0x7f38a0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38a00b1900 0x7f38a0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 --2- 172.21.15.26:0/3685038462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38a00ac8e0 0x7f38a013d920 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 >> 172.21.15.26:0/3685038462 conn(0x7f38a0060310 msgr2=0x7f38a013f530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:21.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 shutdown_connections 2022-07-20T02:27:21.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.508+0000 7f38a4b3a700 1 -- 172.21.15.26:0/3685038462 wait complete. 2022-07-20T02:27:21.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:27:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 Processor -- start 2022-07-20T02:27:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 -- start start 2022-07-20T02:27:21.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc061a20 0x7f03bc1432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03bc12c750 0x7f03bc0b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc061e00 con 0x7f03bc1468c0 2022-07-20T02:27:21.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc147040 con 0x7f03bc12c750 2022-07-20T02:27:21.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.656+0000 7f03c272d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc1472e0 con 0x7f03bc061a20 2022-07-20T02:27:21.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03bc12c750 0x7f03bc0b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03bc12c750 0x7f03bc0b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36700/0 (socket says 172.21.15.26:36700) 2022-07-20T02:27:21.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 -- 172.21.15.26:0/325967617 learned_addr learned my addr 172.21.15.26:0/325967617 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:21.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03bbfff700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc061a20 0x7f03bc1432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 -- 172.21.15.26:0/325967617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc061a20 msgr2=0x7f03bc1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:21.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc061a20 0x7f03bc1432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 -- 172.21.15.26:0/325967617 >> v1:172.21.15.120:6789/0 conn(0x7f03bc1468c0 legacy=0x7f03bc12c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 -- 172.21.15.26:0/325967617 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc147580 con 0x7f03bc12c750 2022-07-20T02:27:21.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c0cca700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc12c750 0x7f03bc0b2720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f03b40032f0 tx=0x7f03b4009290).ready entity=mon.0 client_cookie=36427a7333e13ad8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03baffd700 1 -- 172.21.15.26:0/325967617 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b400b8a0 con 0x7f03bc12c750 2022-07-20T02:27:21.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03baffd700 1 -- 172.21.15.26:0/325967617 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b400ba00 con 0x7f03bc12c750 2022-07-20T02:27:21.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03baffd700 1 -- 172.21.15.26:0/325967617 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b400bb70 con 0x7f03bc12c750 2022-07-20T02:27:21.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 -- 172.21.15.26:0/325967617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc12c750 msgr2=0x7f03bc0b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc12c750 0x7f03bc0b2720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f03b40032f0 tx=0x7f03b4009290).stop 2022-07-20T02:27:21.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 -- 172.21.15.26:0/325967617 shutdown_connections 2022-07-20T02:27:21.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc12c750 0x7f03bc0b2720 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 --2- 172.21.15.26:0/325967617 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc061a20 0x7f03bc1432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 -- 172.21.15.26:0/325967617 >> 172.21.15.26:0/325967617 conn(0x7f03bc05faf0 msgr2=0x7f03bc066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 -- 172.21.15.26:0/325967617 shutdown_connections 2022-07-20T02:27:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.660+0000 7f03c272d700 1 -- 172.21.15.26:0/325967617 wait complete. 2022-07-20T02:27:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 Processor -- start 2022-07-20T02:27:21.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 -- start start 2022-07-20T02:27:21.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03bc12c750 0x7f03bc0b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc1468c0 0x7f03bc0b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc146000 con 0x7f03bc061a20 2022-07-20T02:27:21.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0ae630 con 0x7f03bc12c750 2022-07-20T02:27:21.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c272d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0ae8d0 con 0x7f03bc1468c0 2022-07-20T02:27:21.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36702/0 (socket says 172.21.15.26:36702) 2022-07-20T02:27:21.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 -- 172.21.15.26:0/4250288764 learned_addr learned my addr 172.21.15.26:0/4250288764 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03c0cca700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc1468c0 0x7f03bc0b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bb7fe700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03bc12c750 0x7f03bc0b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 -- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc1468c0 msgr2=0x7f03bc0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc1468c0 0x7f03bc0b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 -- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03bc12c750 msgr2=0x7f03bc0b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03bc12c750 0x7f03bc0b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.664+0000 7f03bbfff700 1 -- 172.21.15.26:0/4250288764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4009530 con 0x7f03bc061a20 2022-07-20T02:27:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03bbfff700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f03a40030b0 tx=0x7f03a4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a4011780 con 0x7f03bc061a20 2022-07-20T02:27:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40118e0 con 0x7f03bc061a20 2022-07-20T02:27:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc0aeb70 con 0x7f03bc061a20 2022-07-20T02:27:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a4011c40 con 0x7f03bc061a20 2022-07-20T02:27:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc0ba140 con 0x7f03bc061a20 2022-07-20T02:27:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc0af010 con 0x7f03bc061a20 2022-07-20T02:27:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f03a401c460 con 0x7f03bc061a20 2022-07-20T02:27:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 0x7f03a0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f03a406c460 con 0x7f03bc061a20 2022-07-20T02:27:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03bb7fe700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 0x7f03a0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.668+0000 7f03bb7fe700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 0x7f03a0055f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f03ac00d000 tx=0x7f03ac000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.676+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03bc0af010 con 0x7f03bc061a20 2022-07-20T02:27:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:21.804+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f03bc0af010 con 0x7f03bc061a20 2022-07-20T02:27:22.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.524+0000 7f03b97fa700 1 -- 172.21.15.26:0/4250288764 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 185+0+0 (secure 0 0 0) 0x7f03bc0af010 con 0x7f03bc061a20 2022-07-20T02:27:22.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 msgr2=0x7f03a0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 0x7f03a0055f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f03ac00d000 tx=0x7f03ac000bc0).stop 2022-07-20T02:27:22.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 msgr2=0x7f03bc0b94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f03a40030b0 tx=0x7f03a4008040).stop 2022-07-20T02:27:22.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 shutdown_connections 2022-07-20T02:27:22.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053a70 0x7f03a0055f30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03bc1468c0 0x7f03bc0b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03bc12c750 0x7f03bc0b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 --2- 172.21.15.26:0/4250288764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03bc061a20 0x7f03bc0b94c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 >> 172.21.15.26:0/4250288764 conn(0x7f03bc05faf0 msgr2=0x7f03bc0b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:22.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 shutdown_connections 2022-07-20T02:27:22.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.528+0000 7f03c272d700 1 -- 172.21.15.26:0/4250288764 wait complete. 2022-07-20T02:27:22.546 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:27:22.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:27:22.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.656+0000 7fb91e2db700 1 Processor -- start 2022-07-20T02:27:22.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.656+0000 7fb91e2db700 1 -- start start 2022-07-20T02:27:22.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb918147be0 0x7fb9180aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9180b62d0 con 0x7fb9180b5ef0 2022-07-20T02:27:22.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91814aac0 con 0x7fb918147be0 2022-07-20T02:27:22.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91814ad60 con 0x7fb9180a7fc0 2022-07-20T02:27:22.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb918147be0 0x7fb9180aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb918147be0 0x7fb9180aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36712/0 (socket says 172.21.15.26:36712) 2022-07-20T02:27:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 -- 172.21.15.26:0/3390985188 learned_addr learned my addr 172.21.15.26:0/3390985188 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91dada700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 -- 172.21.15.26:0/3390985188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 msgr2=0x7fb9180af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 -- 172.21.15.26:0/3390985188 >> v1:172.21.15.120:6789/0 conn(0x7fb9180b5ef0 legacy=0x7fb9181474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 -- 172.21.15.26:0/3390985188 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91814b000 con 0x7fb918147be0 2022-07-20T02:27:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91cad8700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb9180aee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb9140032f0 tx=0x7fb914009290).ready entity=mon.0 client_cookie=7fa881d1a125c7e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb90ffff700 1 -- 172.21.15.26:0/3390985188 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb91400b8a0 con 0x7fb918147be0 2022-07-20T02:27:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb90ffff700 1 -- 172.21.15.26:0/3390985188 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91400ba00 con 0x7fb918147be0 2022-07-20T02:27:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb90ffff700 1 -- 172.21.15.26:0/3390985188 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb91400bb70 con 0x7fb918147be0 2022-07-20T02:27:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- 172.21.15.26:0/3390985188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 msgr2=0x7fb9180aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb9180aee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb9140032f0 tx=0x7fb914009290).stop 2022-07-20T02:27:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- 172.21.15.26:0/3390985188 shutdown_connections 2022-07-20T02:27:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 --2- 172.21.15.26:0/3390985188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb9180aee40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- 172.21.15.26:0/3390985188 >> 172.21.15.26:0/3390985188 conn(0x7fb918060310 msgr2=0x7fb9180b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- 172.21.15.26:0/3390985188 shutdown_connections 2022-07-20T02:27:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- 172.21.15.26:0/3390985188 wait complete. 2022-07-20T02:27:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 Processor -- start 2022-07-20T02:27:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.660+0000 7fb91e2db700 1 -- start start 2022-07-20T02:27:22.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9180b5ef0 0x7fb918156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb918147a50 con 0x7fb918147be0 2022-07-20T02:27:22.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb918064a00 con 0x7fb9180b5ef0 2022-07-20T02:27:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb918064ca0 con 0x7fb9180a7fc0 2022-07-20T02:27:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36720/0 (socket says 172.21.15.26:36720) 2022-07-20T02:27:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91d2d9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 -- 172.21.15.26:0/1105747989 learned_addr learned my addr 172.21.15.26:0/1105747989 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91cad8700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9180b5ef0 0x7fb918156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 -- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 msgr2=0x7fb9180aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 -- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9180b5ef0 msgr2=0x7fb918156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:22.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9180b5ef0 0x7fb918156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:22.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 -- 172.21.15.26:0/1105747989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914009530 con 0x7fb918147be0 2022-07-20T02:27:22.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91dada700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fb91000c000 tx=0x7fb910006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:22.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb910011750 con 0x7fb918147be0 2022-07-20T02:27:22.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9100118b0 con 0x7fb918147be0 2022-07-20T02:27:22.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb918064f40 con 0x7fb918147be0 2022-07-20T02:27:22.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb910011c10 con 0x7fb918147be0 2022-07-20T02:27:22.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91811dd70 con 0x7fb918147be0 2022-07-20T02:27:22.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.664+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91814b2a0 con 0x7fb918147be0 2022-07-20T02:27:22.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.668+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb910011d70 con 0x7fb918147be0 2022-07-20T02:27:22.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.668+0000 7fb90e7fc700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 0x7fb900055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:22.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.668+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fb91006c350 con 0x7fb918147be0 2022-07-20T02:27:22.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.668+0000 7fb91d2d9700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 0x7fb900055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:22.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.668+0000 7fb91d2d9700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 0x7fb900055f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb90800d000 tx=0x7fb908000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:22.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.676+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb910043160 con 0x7fb918147be0 2022-07-20T02:27:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:22.804+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb9180574e0 con 0x7fb918147be0 2022-07-20T02:27:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.532+0000 7fb90e7fc700 1 -- 172.21.15.26:0/1105747989 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb910046780 con 0x7fb918147be0 2022-07-20T02:27:23.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 msgr2=0x7fb900055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 0x7fb900055f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb90800d000 tx=0x7fb908000bc0).stop 2022-07-20T02:27:23.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 msgr2=0x7fb91815a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fb91000c000 tx=0x7fb910006df0).stop 2022-07-20T02:27:23.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 shutdown_connections 2022-07-20T02:27:23.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb918147be0 0x7fb91815a190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb900053a70 0x7fb900055f30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9180b5ef0 0x7fb918156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 --2- 172.21.15.26:0/1105747989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9180a7fc0 0x7fb9180aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 >> 172.21.15.26:0/1105747989 conn(0x7fb918060310 msgr2=0x7fb9180b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 shutdown_connections 2022-07-20T02:27:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.536+0000 7fb91e2db700 1 -- 172.21.15.26:0/1105747989 wait complete. 2022-07-20T02:27:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:27:23.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:27:23.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 Processor -- start 2022-07-20T02:27:23.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 -- start start 2022-07-20T02:27:23.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f95ec0b2b30 0x7f95ec0a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b2f10 con 0x7f95ec147d70 2022-07-20T02:27:23.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0a7500 con 0x7f95ec060f10 2022-07-20T02:27:23.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f4011700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0a8e30 con 0x7f95ec0b2b30 2022-07-20T02:27:23.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f25ae700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f95ec147d70 0x7f95ec066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36008/0 (socket says 172.21.15.26:36008) 2022-07-20T02:27:23.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f25ae700 1 -- 172.21.15.26:0/2833576816 learned_addr learned my addr 172.21.15.26:0/2833576816 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:23.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f1dad700 1 --2- 172.21.15.26:0/2833576816 >> v2:172.21.15.26:3300/0 conn(0x7f95ec0b2b30 0x7f95ec0a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600872201 0 0) 0x7f95ec0b2f10 con 0x7f95ec147d70 2022-07-20T02:27:23.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d4003260 con 0x7f95ec147d70 2022-07-20T02:27:23.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f1dad700 1 -- 172.21.15.26:0/2833576816 >> v1:172.21.15.120:6789/0 conn(0x7f95ec060f10 legacy=0x7f95ec147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912167917 0 0) 0x7f95ec0a7500 con 0x7f95ec060f10 2022-07-20T02:27:23.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f1dad700 1 -- 172.21.15.26:0/2833576816 >> v1:172.21.15.26:6789/0 conn(0x7f95ec147d70 legacy=0x7f95ec066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f1dad700 1 -- 172.21.15.26:0/2833576816 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0a77a0 con 0x7f95ec0b2b30 2022-07-20T02:27:23.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f1dad700 1 --2- 172.21.15.26:0/2833576816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec0b2b30 0x7f95ec0a7ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f95dc002f70 tx=0x7f95dc006df0).ready entity=mon.0 client_cookie=b04bd0a041af31ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95dc00d610 con 0x7f95ec0b2b30 2022-07-20T02:27:23.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc000e90 con 0x7f95ec0b2b30 2022-07-20T02:27:23.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.668+0000 7f95f0dab700 1 -- 172.21.15.26:0/2833576816 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95dc00d960 con 0x7f95ec0b2b30 2022-07-20T02:27:23.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- 172.21.15.26:0/2833576816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec0b2b30 msgr2=0x7f95ec0a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 --2- 172.21.15.26:0/2833576816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec0b2b30 0x7f95ec0a7ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f95dc002f70 tx=0x7f95dc006df0).stop 2022-07-20T02:27:23.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- 172.21.15.26:0/2833576816 shutdown_connections 2022-07-20T02:27:23.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 --2- 172.21.15.26:0/2833576816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec0b2b30 0x7f95ec0a7ee0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- 172.21.15.26:0/2833576816 >> 172.21.15.26:0/2833576816 conn(0x7f95ec0b6110 msgr2=0x7f95ec0b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:23.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- 172.21.15.26:0/2833576816 shutdown_connections 2022-07-20T02:27:23.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- 172.21.15.26:0/2833576816 wait complete. 2022-07-20T02:27:23.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 Processor -- start 2022-07-20T02:27:23.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- start start 2022-07-20T02:27:23.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 0x7f95ec0663e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 0x7f95ec0668d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec148cc0 con 0x7f95ec060f10 2022-07-20T02:27:23.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec065040 con 0x7f95ec147d70 2022-07-20T02:27:23.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f4011700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec065510 con 0x7f95ec0b2b30 2022-07-20T02:27:23.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36730/0 (socket says 172.21.15.26:36730) 2022-07-20T02:27:23.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 -- 172.21.15.26:0/3664461458 learned_addr learned my addr 172.21.15.26:0/3664461458 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:23.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f25ae700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 0x7f95ec0668d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f15ac700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 0x7f95ec0663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 -- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 msgr2=0x7f95ec0663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 0x7f95ec0663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 -- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 msgr2=0x7f95ec0668d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 0x7f95ec0668d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f1dad700 1 -- 172.21.15.26:0/3664461458 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc00b180 con 0x7f95ec060f10 2022-07-20T02:27:23.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f25ae700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 0x7f95ec0668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:23.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.672+0000 7f95f15ac700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 0x7f95ec0663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:23.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f1dad700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f95dc002f70 tx=0x7f95dc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95dc00d290 con 0x7f95ec060f10 2022-07-20T02:27:23.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc01e400 con 0x7f95ec060f10 2022-07-20T02:27:23.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec0657b0 con 0x7f95ec060f10 2022-07-20T02:27:23.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95dc01e760 con 0x7f95ec060f10 2022-07-20T02:27:23.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec064d10 con 0x7f95ec060f10 2022-07-20T02:27:23.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec066180 con 0x7f95ec060f10 2022-07-20T02:27:23.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f95dc025030 con 0x7f95ec060f10 2022-07-20T02:27:23.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 0x7f95cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f95dc06cc00 con 0x7f95ec060f10 2022-07-20T02:27:23.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f15ac700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 0x7f95cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.676+0000 7f95f15ac700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 0x7f95cc055f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f95e8008000 tx=0x7f95e8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.680+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95dc072020 con 0x7f95ec060f10 2022-07-20T02:27:23.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.812+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f95ec0a7a40 con 0x7f95ec060f10 2022-07-20T02:27:23.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.816+0000 7f95daffd700 1 -- 172.21.15.26:0/3664461458 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f95dc0230a0 con 0x7f95ec060f10 2022-07-20T02:27:23.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 msgr2=0x7f95cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 0x7f95cc055f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f95e8008000 tx=0x7f95e8009040).stop 2022-07-20T02:27:23.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 msgr2=0x7f95ec0b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f95dc002f70 tx=0x7f95dc014000).stop 2022-07-20T02:27:23.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 shutdown_connections 2022-07-20T02:27:23.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95cc053a70 0x7f95cc055f30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec147d70 0x7f95ec0668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95ec0b2b30 0x7f95ec0663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 --2- 172.21.15.26:0/3664461458 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95ec060f10 0x7f95ec0b4ae0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 >> 172.21.15.26:0/3664461458 conn(0x7f95ec0b6110 msgr2=0x7f95ec0a96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:23.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 shutdown_connections 2022-07-20T02:27:23.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.820+0000 7f95f4011700 1 -- 172.21.15.26:0/3664461458 wait complete. 2022-07-20T02:27:23.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-20T02:27:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 Processor -- start 2022-07-20T02:27:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 -- start start 2022-07-20T02:27:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 0x7f14400b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14400aa1e0 con 0x7f14400af6d0 2022-07-20T02:27:23.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14400ae110 con 0x7f1440060950 2022-07-20T02:27:23.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14400a9f40 con 0x7f14400add30 2022-07-20T02:27:23.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f14400af6d0 0x7f14400afb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36022/0 (socket says 172.21.15.26:36022) 2022-07-20T02:27:23.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143f7fe700 1 -- 172.21.15.26:0/2873239640 learned_addr learned my addr 172.21.15.26:0/2873239640 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:23.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143effd700 1 --2- 172.21.15.26:0/2873239640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 0x7f14400b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773377928 0 0) 0x7f14400aa1e0 con 0x7f14400af6d0 2022-07-20T02:27:23.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1424003260 con 0x7f14400af6d0 2022-07-20T02:27:23.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115029986 0 0) 0x7f14400ae110 con 0x7f1440060950 2022-07-20T02:27:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14400aa1e0 con 0x7f1440060950 2022-07-20T02:27:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143effd700 1 -- 172.21.15.26:0/2873239640 >> v1:172.21.15.120:6789/0 conn(0x7f1440060950 legacy=0x7f1440060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143effd700 1 -- 172.21.15.26:0/2873239640 >> v1:172.21.15.26:6789/0 conn(0x7f14400af6d0 legacy=0x7f14400afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143effd700 1 -- 172.21.15.26:0/2873239640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400b1110 con 0x7f14400add30 2022-07-20T02:27:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143effd700 1 --2- 172.21.15.26:0/2873239640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 0x7f14400b2ab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f14300032f0 tx=0x7f14300091d0).ready entity=mon.2 client_cookie=e42fc2afe671d16d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.936+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f143000b600 con 0x7f14400add30 2022-07-20T02:27:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143000b760 con 0x7f14400add30 2022-07-20T02:27:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143dffb700 1 -- 172.21.15.26:0/2873239640 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f143000b8d0 con 0x7f14400add30 2022-07-20T02:27:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- 172.21.15.26:0/2873239640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 msgr2=0x7f14400b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 --2- 172.21.15.26:0/2873239640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 0x7f14400b2ab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f14300032f0 tx=0x7f14300091d0).stop 2022-07-20T02:27:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- 172.21.15.26:0/2873239640 shutdown_connections 2022-07-20T02:27:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 --2- 172.21.15.26:0/2873239640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14400add30 0x7f14400b2ab0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- 172.21.15.26:0/2873239640 >> 172.21.15.26:0/2873239640 conn(0x7f14400b3050 msgr2=0x7f14400b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- 172.21.15.26:0/2873239640 shutdown_connections 2022-07-20T02:27:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- 172.21.15.26:0/2873239640 wait complete. 2022-07-20T02:27:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 Processor -- start 2022-07-20T02:27:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- start start 2022-07-20T02:27:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1440060950 0x7f144014ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14400af6d0 0x7f144005cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1440153570 con 0x7f14400add30 2022-07-20T02:27:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144013d050 con 0x7f14400af6d0 2022-07-20T02:27:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.940+0000 7f143ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144013d2f0 con 0x7f1440060950 2022-07-20T02:27:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36746/0 (socket says 172.21.15.26:36746) 2022-07-20T02:27:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 -- 172.21.15.26:0/3418066930 learned_addr learned my addr 172.21.15.26:0/3418066930 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143effd700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1440060950 0x7f144014ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143f7fe700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14400af6d0 0x7f144005cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 -- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1440060950 msgr2=0x7f144014ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1440060950 0x7f144014ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 -- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14400af6d0 msgr2=0x7f144005cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14400af6d0 0x7f144005cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 -- 172.21.15.26:0/3418066930 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430009470 con 0x7f14400add30 2022-07-20T02:27:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143e7fc700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1434001110 tx=0x7f1434007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1434012400 con 0x7f14400add30 2022-07-20T02:27:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1434012560 con 0x7f14400add30 2022-07-20T02:27:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14340128c0 con 0x7f14400add30 2022-07-20T02:27:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14401537a0 con 0x7f14400add30 2022-07-20T02:27:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144013cb60 con 0x7f14400add30 2022-07-20T02:27:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.944+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1440153930 con 0x7f14400add30 2022-07-20T02:27:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.948+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1434019030 con 0x7f14400add30 2022-07-20T02:27:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.948+0000 7f142ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 0x7f141c055ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.948+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f143406b320 con 0x7f14400add30 2022-07-20T02:27:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.948+0000 7f143effd700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 0x7f141c055ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.948+0000 7f143effd700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 0x7f141c055ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f143001b000 tx=0x7f1430016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:23.952+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14340422b0 con 0x7f14400add30 2022-07-20T02:27:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.112+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f14400b13b0 con 0x7f14400add30 2022-07-20T02:27:24.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.532+0000 7f142ffff700 1 -- 172.21.15.26:0/3418066930 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v100) v1 ==== 139+0+0 (secure 0 0 0) 0x7f14340458d0 con 0x7f14400add30 2022-07-20T02:27:24.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 msgr2=0x7f141c055ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 0x7f141c055ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f143001b000 tx=0x7f1430016000).stop 2022-07-20T02:27:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 msgr2=0x7f1440060330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1434001110 tx=0x7f1434007b60).stop 2022-07-20T02:27:24.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 shutdown_connections 2022-07-20T02:27:24.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f14400add30 0x7f1440060330 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f141c053a30 0x7f141c055ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14400af6d0 0x7f144005cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 --2- 172.21.15.26:0/3418066930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1440060950 0x7f144014ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 >> 172.21.15.26:0/3418066930 conn(0x7f14400b3050 msgr2=0x7f1440066a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:24.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 shutdown_connections 2022-07-20T02:27:24.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.536+0000 7f143ffff700 1 -- 172.21.15.26:0/3418066930 wait complete. 2022-07-20T02:27:24.571 INFO:tasks.cephfs_test_runner:test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T02:27:24.572 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T02:27:24.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-20T02:27:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 Processor -- start 2022-07-20T02:27:24.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 -- start start 2022-07-20T02:27:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6dcc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6dcc0a7fc0 0x7f6dcc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc0b62d0 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dcc14aac0 con 0x7f6dcc0a7fc0 2022-07-20T02:27:24.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd213e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc14ad60 con 0x7f6dcc147be0 2022-07-20T02:27:24.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd113c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6dcc0b5ef0 0x7f6dcc1474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36032/0 (socket says 172.21.15.26:36032) 2022-07-20T02:27:24.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd113c700 1 -- 172.21.15.26:0/829121560 learned_addr learned my addr 172.21.15.26:0/829121560 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:24.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd193d700 1 --2- 172.21.15.26:0/829121560 >> v2:172.21.15.120:3300/0 conn(0x7f6dcc0a7fc0 0x7f6dcc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dd093b700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6dcc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129107137 0 0) 0x7f6dcc0b62d0 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.684+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0003260 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040922045 0 0) 0x7f6db0003260 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 msgr2=0x7f6dcc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6dcc0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 >> v2:172.21.15.120:3300/0 conn(0x7f6dcc0a7fc0 msgr2=0x7f6dcc0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 --2- 172.21.15.26:0/829121560 >> v2:172.21.15.120:3300/0 conn(0x7f6dcc0a7fc0 0x7f6dcc0af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc14b000 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f6dc40019e0 con 0x7f6dcc0b5ef0 2022-07-20T02:27:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 >> v1:172.21.15.26:6789/0 conn(0x7f6dcc0b5ef0 legacy=0x7f6dcc1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db0003750 0x7f6db0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6db0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db00049f0 0x7f6db0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc14aac0 con 0x7f6db0003750 2022-07-20T02:27:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc14ad60 con 0x7f6db00049f0 2022-07-20T02:27:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db000b750 con 0x7f6dcc147be0 2022-07-20T02:27:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db0003750 0x7f6db0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd093b700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6db0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd193d700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db00049f0 0x7f6db0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 -- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 msgr2=0x7f6db0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dcc147be0 0x7f6db0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 -- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db00049f0 msgr2=0x7f6db0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db00049f0 0x7f6db0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 -- 172.21.15.26:0/829121560 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0003260 con 0x7f6db0003750 2022-07-20T02:27:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd193d700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db00049f0 0x7f6db0004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd113c700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db0003750 0x7f6db0003b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6dc40016f0 tx=0x7f6dc40021d0).ready entity=mon.0 client_cookie=bde2204319e2adfa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dc400cc80 con 0x7f6db0003750 2022-07-20T02:27:24.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc400cde0 con 0x7f6db0003750 2022-07-20T02:27:24.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dc3fff700 1 -- 172.21.15.26:0/829121560 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dc4005b80 con 0x7f6db0003750 2022-07-20T02:27:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 -- 172.21.15.26:0/829121560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db0003750 msgr2=0x7f6db0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 --2- 172.21.15.26:0/829121560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db0003750 0x7f6db0003b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6dc40016f0 tx=0x7f6dc40021d0).stop 2022-07-20T02:27:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd193d700 1 -- 172.21.15.26:0/829121560 reap_dead start 2022-07-20T02:27:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 -- 172.21.15.26:0/829121560 shutdown_connections 2022-07-20T02:27:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 -- 172.21.15.26:0/829121560 >> 172.21.15.26:0/829121560 conn(0x7f6dcc060310 msgr2=0x7f6dcc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 -- 172.21.15.26:0/829121560 shutdown_connections 2022-07-20T02:27:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 -- 172.21.15.26:0/829121560 wait complete. 2022-07-20T02:27:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.688+0000 7f6dd213e700 1 Processor -- start 2022-07-20T02:27:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- start start 2022-07-20T02:27:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db0003750 0x7f6dcc062b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 0x7f6dcc0ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0641a0 con 0x7f6dcc0aa730 2022-07-20T02:27:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0643d0 con 0x7f6dcc0abec0 2022-07-20T02:27:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc064670 con 0x7f6db0003750 2022-07-20T02:27:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36766/0 (socket says 172.21.15.26:36766) 2022-07-20T02:27:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 -- 172.21.15.26:0/415347649 learned_addr learned my addr 172.21.15.26:0/415347649 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd113c700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db0003750 0x7f6dcc062b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd193d700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 0x7f6dcc0ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 -- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db0003750 msgr2=0x7f6dcc062b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db0003750 0x7f6dcc062b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 -- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 msgr2=0x7f6dcc0ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 0x7f6dcc0ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 -- 172.21.15.26:0/415347649 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc064910 con 0x7f6dcc0aa730 2022-07-20T02:27:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd193d700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 0x7f6dcc0ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd093b700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f6dbc00b000 tx=0x7f6dbc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dbc014680 con 0x7f6dcc0aa730 2022-07-20T02:27:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc0147e0 con 0x7f6dcc0aa730 2022-07-20T02:27:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc0aa3a0 con 0x7f6dcc0aa730 2022-07-20T02:27:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dbc014b40 con 0x7f6dcc0aa730 2022-07-20T02:27:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.692+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc0abb80 con 0x7f6dcc0aa730 2022-07-20T02:27:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc064aa0 con 0x7f6dcc0aa730 2022-07-20T02:27:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6dbc024070 con 0x7f6dcc0aa730 2022-07-20T02:27:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dc27fc700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 0x7f6db4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6dbc06c730 con 0x7f6dcc0aa730 2022-07-20T02:27:24.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dd113c700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 0x7f6db4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:24.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.696+0000 7f6dd113c700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 0x7f6db4055fc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6dc4019000 tx=0x7f6dc4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:24.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.700+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6dbc043730 con 0x7f6dcc0aa730 2022-07-20T02:27:24.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:24.872+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7f6dcc14b2a0 con 0x7f6dcc0aa730 2022-07-20T02:27:25.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.540+0000 7f6dc27fc700 1 -- 172.21.15.26:0/415347649 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v101) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6dbc046d50 con 0x7f6dcc0aa730 2022-07-20T02:27:25.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 msgr2=0x7f6db4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 0x7f6db4055fc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6dc4019000 tx=0x7f6dc4015000).stop 2022-07-20T02:27:25.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 msgr2=0x7f6dcc0ab000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f6dbc00b000 tx=0x7f6dbc008040).stop 2022-07-20T02:27:25.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 shutdown_connections 2022-07-20T02:27:25.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dcc0aa730 0x7f6dcc0ab000 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6db4053b00 0x7f6db4055fc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dcc0abec0 0x7f6dcc0ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 --2- 172.21.15.26:0/415347649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db0003750 0x7f6dcc062b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 >> 172.21.15.26:0/415347649 conn(0x7f6dcc060310 msgr2=0x7f6dcc0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:25.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 shutdown_connections 2022-07-20T02:27:25.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.544+0000 7f6dd213e700 1 -- 172.21.15.26:0/415347649 wait complete. 2022-07-20T02:27:25.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:27:25.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 Processor -- start 2022-07-20T02:27:25.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 -- start start 2022-07-20T02:27:25.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f600b9c80 0x7f6f601431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6f60062710 0x7f6f600b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f600b6fe0 0x7f6f601403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6012bee0 con 0x7f6f600b9c80 2022-07-20T02:27:25.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f600ba060 con 0x7f6f60062710 2022-07-20T02:27:25.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.684+0000 7f6f64a26700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f60061f80 con 0x7f6f600b6fe0 2022-07-20T02:27:25.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f600b9c80 0x7f6f601431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f600b9c80 0x7f6f601431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36772/0 (socket says 172.21.15.26:36772) 2022-07-20T02:27:25.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 -- 172.21.15.26:0/1498244193 learned_addr learned my addr 172.21.15.26:0/1498244193 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:25.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5ed9d700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f600b6fe0 0x7f6f601403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5dd9b700 1 --2- 172.21.15.26:0/1498244193 >> v2:172.21.15.120:3300/0 conn(0x7f6f60062710 0x7f6f600b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 -- 172.21.15.26:0/1498244193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f600b6fe0 msgr2=0x7f6f601403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:25.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f600b6fe0 0x7f6f601403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 -- 172.21.15.26:0/1498244193 >> v2:172.21.15.120:3300/0 conn(0x7f6f60062710 msgr2=0x7f6f600b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/1498244193 >> v2:172.21.15.120:3300/0 conn(0x7f6f60062710 0x7f6f600b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 -- 172.21.15.26:0/1498244193 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60062220 con 0x7f6f600b9c80 2022-07-20T02:27:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b9c80 0x7f6f601431a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6f50002f70 tx=0x7f6f50006eb0).ready entity=mon.0 client_cookie=2e8cbce7795d9d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5d59a700 1 -- 172.21.15.26:0/1498244193 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f5000d570 con 0x7f6f600b9c80 2022-07-20T02:27:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5d59a700 1 -- 172.21.15.26:0/1498244193 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50000e90 con 0x7f6f600b9c80 2022-07-20T02:27:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f5d59a700 1 -- 172.21.15.26:0/1498244193 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f5000d900 con 0x7f6f600b9c80 2022-07-20T02:27:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 -- 172.21.15.26:0/1498244193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b9c80 msgr2=0x7f6f601431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b9c80 0x7f6f601431a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6f50002f70 tx=0x7f6f50006eb0).stop 2022-07-20T02:27:25.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 -- 172.21.15.26:0/1498244193 shutdown_connections 2022-07-20T02:27:25.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f600b6fe0 0x7f6f601403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 --2- 172.21.15.26:0/1498244193 >> v2:172.21.15.120:3300/0 conn(0x7f6f60062710 0x7f6f600b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 --2- 172.21.15.26:0/1498244193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b9c80 0x7f6f601431a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 -- 172.21.15.26:0/1498244193 >> 172.21.15.26:0/1498244193 conn(0x7f6f60066f70 msgr2=0x7f6f6005faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:25.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 -- 172.21.15.26:0/1498244193 shutdown_connections 2022-07-20T02:27:25.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.688+0000 7f6f64a26700 1 -- 172.21.15.26:0/1498244193 wait complete. 2022-07-20T02:27:25.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 Processor -- start 2022-07-20T02:27:25.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- start start 2022-07-20T02:27:25.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60062710 0x7f6f60141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 0x7f6f601561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f60146970 con 0x7f6f600b6fe0 2022-07-20T02:27:25.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f60155860 con 0x7f6f600b9c80 2022-07-20T02:27:25.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601566a0 con 0x7f6f60062710 2022-07-20T02:27:25.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36780/0 (socket says 172.21.15.26:36780) 2022-07-20T02:27:25.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 -- 172.21.15.26:0/4057381073 learned_addr learned my addr 172.21.15.26:0/4057381073 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:25.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60062710 0x7f6f60141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5ed9d700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 0x7f6f601561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 -- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60062710 msgr2=0x7f6f60141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60062710 0x7f6f60141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 -- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 msgr2=0x7f6f601561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 0x7f6f601561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 -- 172.21.15.26:0/4057381073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5000b1e0 con 0x7f6f600b6fe0 2022-07-20T02:27:25.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5ed9d700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 0x7f6f601561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:25.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f5dd9b700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6f4800c000 tx=0x7f6f48006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:25.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f48010780 con 0x7f6f600b6fe0 2022-07-20T02:27:25.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f480108e0 con 0x7f6f600b6fe0 2022-07-20T02:27:25.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f60146e80 con 0x7f6f600b6fe0 2022-07-20T02:27:25.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f48010c40 con 0x7f6f600b6fe0 2022-07-20T02:27:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.692+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f60147120 con 0x7f6f600b6fe0 2022-07-20T02:27:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f600b6650 con 0x7f6f600b6fe0 2022-07-20T02:27:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6f4801c460 con 0x7f6f600b6fe0 2022-07-20T02:27:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f577fe700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 0x7f6f44055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6f4806c160 con 0x7f6f600b6fe0 2022-07-20T02:27:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 0x7f6f44055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.696+0000 7f6f5e59c700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 0x7f6f44055ea0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6f5001c000 tx=0x7f6f50015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.704+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f480431a0 con 0x7f6f600b6fe0 2022-07-20T02:27:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.832+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6f600574e0 con 0x7f6f600b6fe0 2022-07-20T02:27:25.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.836+0000 7f6f577fe700 1 -- 172.21.15.26:0/4057381073 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6f480467c0 con 0x7f6f600b6fe0 2022-07-20T02:27:25.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 msgr2=0x7f6f44055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 0x7f6f44055ea0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6f5001c000 tx=0x7f6f50015000).stop 2022-07-20T02:27:25.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 msgr2=0x7f6f60155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:25.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6f4800c000 tx=0x7f6f48006df0).stop 2022-07-20T02:27:25.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 shutdown_connections 2022-07-20T02:27:25.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f600b6fe0 0x7f6f60155cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f440539e0 0x7f6f44055ea0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f600b9c80 0x7f6f601561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 --2- 172.21.15.26:0/4057381073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f60062710 0x7f6f60141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:25.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 >> 172.21.15.26:0/4057381073 conn(0x7f6f60066f70 msgr2=0x7f6f600b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:25.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 shutdown_connections 2022-07-20T02:27:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:25.840+0000 7f6f64a26700 1 -- 172.21.15.26:0/4057381073 wait complete. 2022-07-20T02:27:25.855 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:27:25.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:25.893 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:25.922 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:25.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:27:25.987 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:27:25.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:25.989 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:25.989 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:27:26.019 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:27:26.020 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:27:26.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:26.021 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:26.022 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:27:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:27:26.041 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:27:26.050 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:27:26.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:27:26.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 Processor -- start 2022-07-20T02:27:26.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 -- start start 2022-07-20T02:27:26.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f149c0b5ef0 0x7f149c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:26.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149c0b62d0 con 0x7f149c0a8c40 2022-07-20T02:27:26.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149c14aac0 con 0x7f149c0a7fc0 2022-07-20T02:27:26.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f149c14ad60 con 0x7f149c0b5ef0 2022-07-20T02:27:26.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149bfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f149c0a8c40 0x7f149c151dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36070/0 (socket says 172.21.15.26:36070) 2022-07-20T02:27:26.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149bfff700 1 -- 172.21.15.26:0/22555666 learned_addr learned my addr 172.21.15.26:0/22555666 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:26.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149b7fe700 1 --2- 172.21.15.26:0/22555666 >> v2:172.21.15.120:3300/0 conn(0x7f149c0b5ef0 0x7f149c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:26.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149a7fc700 1 -- 172.21.15.26:0/22555666 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115379258 0 0) 0x7f149c0b62d0 con 0x7f149c0a8c40 2022-07-20T02:27:26.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149a7fc700 1 -- 172.21.15.26:0/22555666 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c003260 con 0x7f149c0a8c40 2022-07-20T02:27:26.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149b7fe700 1 -- 172.21.15.26:0/22555666 >> v1:172.21.15.120:6789/0 conn(0x7f149c0a7fc0 legacy=0x7f149c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149b7fe700 1 -- 172.21.15.26:0/22555666 >> v1:172.21.15.26:6789/0 conn(0x7f149c0a8c40 legacy=0x7f149c151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149b7fe700 1 -- 172.21.15.26:0/22555666 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c14b000 con 0x7f149c0b5ef0 2022-07-20T02:27:26.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149b7fe700 1 --2- 172.21.15.26:0/22555666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0b5ef0 0x7f149c1474d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1484002f70 tx=0x7f1484006df0).ready entity=mon.1 client_cookie=60563949f83ed6ce server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:26.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149a7fc700 1 -- 172.21.15.26:0/22555666 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148400d610 con 0x7f149c0b5ef0 2022-07-20T02:27:26.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149a7fc700 1 -- 172.21.15.26:0/22555666 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484000e90 con 0x7f149c0b5ef0 2022-07-20T02:27:26.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f149a7fc700 1 -- 172.21.15.26:0/22555666 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148400d960 con 0x7f149c0b5ef0 2022-07-20T02:27:26.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 -- 172.21.15.26:0/22555666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0b5ef0 msgr2=0x7f149c1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.172+0000 7f14a0937700 1 --2- 172.21.15.26:0/22555666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0b5ef0 0x7f149c1474d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1484002f70 tx=0x7f1484006df0).stop 2022-07-20T02:27:26.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- 172.21.15.26:0/22555666 shutdown_connections 2022-07-20T02:27:26.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 --2- 172.21.15.26:0/22555666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0b5ef0 0x7f149c1474d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- 172.21.15.26:0/22555666 >> 172.21.15.26:0/22555666 conn(0x7f149c060310 msgr2=0x7f149c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:26.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- 172.21.15.26:0/22555666 shutdown_connections 2022-07-20T02:27:26.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- 172.21.15.26:0/22555666 wait complete. 2022-07-20T02:27:26.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 Processor -- start 2022-07-20T02:27:26.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- start start 2022-07-20T02:27:26.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0a7fc0 0x7f149c0ab8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:26.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:26.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f149c0b5ef0 0x7f149c062ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:26.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c065570 con 0x7f149c0a8c40 2022-07-20T02:27:26.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c15aab0 con 0x7f149c0a7fc0 2022-07-20T02:27:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f14a0937700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c0657d0 con 0x7f149c0b5ef0 2022-07-20T02:27:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f149affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f149affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36794/0 (socket says 172.21.15.26:36794) 2022-07-20T02:27:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f149affd700 1 -- 172.21.15.26:0/3643805264 learned_addr learned my addr 172.21.15.26:0/3643805264 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f149b7fe700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0a7fc0 0x7f149c0ab8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.176+0000 7f149bfff700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f149c0b5ef0 0x7f149c062ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 -- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f149c0b5ef0 msgr2=0x7f149c062ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f149c0b5ef0 0x7f149c062ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 -- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0a7fc0 msgr2=0x7f149c0ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0a7fc0 0x7f149c0ab8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 -- 172.21.15.26:0/3643805264 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148400b180 con 0x7f149c0a8c40 2022-07-20T02:27:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149affd700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f149c06b790 tx=0x7f14940014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f149400f470 con 0x7f149c0a8c40 2022-07-20T02:27:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149400f5d0 con 0x7f149c0a8c40 2022-07-20T02:27:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c065a70 con 0x7f149c0a8c40 2022-07-20T02:27:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f149400f930 con 0x7f149c0a8c40 2022-07-20T02:27:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c063d70 con 0x7f149c0a8c40 2022-07-20T02:27:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c14b2a0 con 0x7f149c0a8c40 2022-07-20T02:27:26.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1494019030 con 0x7f149c0a8c40 2022-07-20T02:27:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 0x7f1480055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f149406b4b0 con 0x7f149c0a8c40 2022-07-20T02:27:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.180+0000 7f149b7fe700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 0x7f1480055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.184+0000 7f149b7fe700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 0x7f1480055ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f148401a000 tx=0x7f1484014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.188+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14940424b0 con 0x7f149c0a8c40 2022-07-20T02:27:26.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.316+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f149c0574e0 con 0x7f149c0a8c40 2022-07-20T02:27:26.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.320+0000 7f1498ff9700 1 -- 172.21.15.26:0/3643805264 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v37) v1 ==== 81+0+29267 (secure 0 0 0) 0x7f1494045ad0 con 0x7f149c0a8c40 2022-07-20T02:27:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:26.325 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:27:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 37, 2022-07-20T02:27:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:27:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:27:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:27:23.519509+0000", 2022-07-20T02:27:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:27:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:27:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:27:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:27:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:27:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:27:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:27:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:27:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:27:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:27:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:27:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:27:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:27:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:27:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 6, 2022-07-20T02:27:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:27:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:27:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:27:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:27:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:27:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:27:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:27:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:27:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:26.361 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:27:26.361 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:27:26.361 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:27:26.362 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:27:26.363 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:27:26.363 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:26.364 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:26.364 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:26.364 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:26.365 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:26.365 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:26.366 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:26.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.366 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:27:26.367 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:26.367 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:26.367 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:26.367 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:26.368 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:26.368 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:26.368 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:26.368 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:26.368 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:26.369 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:26.369 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:26.369 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:26.369 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:26.370 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:26.370 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:26.370 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:26.370 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:26.371 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:26.371 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:26.371 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:26.371 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:26.371 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:26.372 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:26.372 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:26.372 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:26.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:26.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.381 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:26.383 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:26.383 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:26.383 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:26.384 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:26.384 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:26.384 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:26.384 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:26.385 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:26.385 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:26.385 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:26.386 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:27:26.386 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:27:26.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.386 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:26.387 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:27:26.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.387 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:27:26.388 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:27:26.388 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:27:26.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:27:26.389 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:27:26.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:27:26.390 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:27:26.390 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:27:26.390 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:27:26.391 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:27:26.391 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:27:26.392 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:27:26.392 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:27:26.392 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:27:26.393 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:27:26.393 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:27:26.394 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:27:26.394 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:27:26.395 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:27:26.395 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:27:26.395 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:27:26.396 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:27:26.396 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:27:26.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.397 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:27:26.398 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:27:26.398 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:27:26.399 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:27:26.399 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:27:26.399 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:27:26.400 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:27:26.400 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:27:26.401 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:27:26.401 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:27:26.402 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:27:26.402 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:27:26.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:27:26.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:27:26.403 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:27:26.405 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:27:26.405 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:27:26.406 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:27:26.406 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:27:26.407 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:27:26.407 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:27:26.407 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:27:26.408 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:27:26.408 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:27:26.409 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:27:26.409 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:27:26.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:27:26.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.414 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:27:26.414 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:27:26.415 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:27:26.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:27:26.416 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:27:26.416 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:27:26.418 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:27:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:27:26.429 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:27:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:27:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:27:26.430 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:27:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:27:26.431 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:27:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:26.433 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:27:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.434 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:27:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:27:26.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.436 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.442 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.443 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.444 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.445 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:27:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:27:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:27:26.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:27:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.457 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:27:26.459 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:27:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:27:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:27:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:27:26.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:27:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:27:26.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:27:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:27:26.470 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.471 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:27:26.473 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:27:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.476 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.477 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.478 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.479 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.480 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:27:26.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:27:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.498 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:27:26.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:27:26.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:27:26.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:27:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:27:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:27:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:27:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:27:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:27:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:27:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:27:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.524 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:27:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:27:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.526 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:27:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:27:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:27:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:27:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:27:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:27:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:27:26.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:27:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:27:26.545 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:27:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:27:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:27:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:27:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:27:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.547 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.548 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.558 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:27:26.558 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:27:26.559 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.559 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.559 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.559 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.559 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.560 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.560 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.561 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:27:26.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:27:26.563 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.564 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:27:26.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:27:26.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.570 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:27:26.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:27:26.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.573 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:27:26.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:27:26.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:27:26.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:27:26.582 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:27:26.582 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:27:26.582 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:27:26.582 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.583 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.583 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.584 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:27:26.584 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:27:26.584 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.584 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.585 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.585 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.585 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.585 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.585 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.594 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.594 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.594 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:27:26.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:27:26.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.599 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:27:26.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:27:26.602 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.603 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:27:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:27:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:27:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:27:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:27:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:27:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:27:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:27:26.617 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:27:26.617 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.618 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.618 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.620 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:27:26.620 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:27:26.621 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.621 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.622 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.622 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.623 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.623 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.623 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.624 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.624 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.625 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.625 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:27:26.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:27:26.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:27:26.652 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:27:26.652 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:27:26.652 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:27:26.653 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.653 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.654 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:27:26.656 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:27:26.656 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.657 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.665 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.666 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:27:26.667 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:27:26.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:27:26.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.674 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:27:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:27:26.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.681 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:27:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:27:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:27:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:27:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:27:26.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:27:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:27:26.712 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:27:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:27:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:27:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:27:26.721 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:27:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:27:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:27:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:27:26.722 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:27:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:27:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:27:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 32, 2022-07-20T02:27:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:27:26.723 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:27:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:27:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:27:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:27:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:27:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:27:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:27:26.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:27:26.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:27:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:27:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:27:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:27:26.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:27:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:27:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:27:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:27:26.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:27:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.746 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:27:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:27:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:27:26.747 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:27:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:27:26.748 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:27:26.749 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:27:26.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:26.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.754 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:27:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:27:26.778 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.778 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.785 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.786 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.786 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:27:26.787 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.788 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:27:26.789 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.789 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.789 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.790 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.790 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.791 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:27:26.791 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.793 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:27:26.793 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.794 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.794 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.794 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.795 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.795 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:27:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.797 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:27:26.798 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.798 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.799 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.799 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.800 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.800 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:27:26.801 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.806 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:27:26.807 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.807 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.808 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.808 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.808 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.809 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:27:26.809 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:27:26.812 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.812 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.813 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.813 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.814 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.815 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:27:26.818 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:27:26.819 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.820 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.820 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.820 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.820 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.821 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:27:26.821 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:26.821 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:27:26.829 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:27:26.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:27:26.836 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:27:26.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:27:26.836 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:27:26.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:27:26.837 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:27:26.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.837 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:27:26.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:27:26.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:27:26.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:27:26.843 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:27:26.843 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:27:26.844 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6835/2813102162": "2022-07-21T02:27:19.695058+0000", 2022-07-20T02:27:26.844 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6834/2813102162": "2022-07-21T02:27:19.695058+0000" 2022-07-20T02:27:26.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.845 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:27:26.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:27:26.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:27:26.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:27:26.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:27:26.846 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:27:26.846 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:27:26.847 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:27:26.847 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:27:26.848 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:27:26.848 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:27:26.848 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:27:26.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.854 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:27:26.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:27:26.857 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:27:26.857 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:27:26.858 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:27:26.858 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:27:26.865 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:27:26.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:26.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:26.893 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:27:26.895 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:27:26.896 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:27:26.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:27:26.897 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:27:26.897 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:27:26.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 msgr2=0x7f1480055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 0x7f1480055ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f148401a000 tx=0x7f1484014000).stop 2022-07-20T02:27:26.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 msgr2=0x7f149c15a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:26.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f149c06b790 tx=0x7f14940014a0).stop 2022-07-20T02:27:26.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 shutdown_connections 2022-07-20T02:27:26.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f149c0a8c40 0x7f149c15a5c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f14800539e0 0x7f1480055ea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f149c0b5ef0 0x7f149c062ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 --2- 172.21.15.26:0/3643805264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f149c0a7fc0 0x7f149c0ab8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:26.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 >> 172.21.15.26:0/3643805264 conn(0x7f149c060310 msgr2=0x7f149c14c350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:26.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 shutdown_connections 2022-07-20T02:27:26.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:26.324+0000 7f14a0937700 1 -- 172.21.15.26:0/3643805264 wait complete. 2022-07-20T02:27:26.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:27.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 Processor -- start 2022-07-20T02:27:27.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 -- start start 2022-07-20T02:27:27.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 0x7f8284143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8284060f00 0x7f828413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82841474e0 con 0x7f8284147000 2022-07-20T02:27:27.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82840b4e40 con 0x7f8284060f00 2022-07-20T02:27:27.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8289333700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828405d3b0 con 0x7f828405cfd0 2022-07-20T02:27:27.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f82837fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8284147000 0x7f82840b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36082/0 (socket says 172.21.15.26:36082) 2022-07-20T02:27:27.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f82837fe700 1 -- 172.21.15.26:0/1995945651 learned_addr learned my addr 172.21.15.26:0/1995945651 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:27.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 --2- 172.21.15.26:0/1995945651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 0x7f8284143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8288b32700 1 --2- 172.21.15.26:0/1995945651 >> v2:172.21.15.120:3300/0 conn(0x7f8284060f00 0x7f828413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8282ffd700 1 -- 172.21.15.26:0/1995945651 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256954199 0 0) 0x7f82841474e0 con 0x7f8284147000 2022-07-20T02:27:27.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8282ffd700 1 -- 172.21.15.26:0/1995945651 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8270003260 con 0x7f8284147000 2022-07-20T02:27:27.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 -- 172.21.15.26:0/1995945651 >> v2:172.21.15.120:3300/0 conn(0x7f8284060f00 msgr2=0x7f828413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:27.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 --2- 172.21.15.26:0/1995945651 >> v2:172.21.15.120:3300/0 conn(0x7f8284060f00 0x7f828413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 -- 172.21.15.26:0/1995945651 >> v1:172.21.15.26:6789/0 conn(0x7f8284147000 legacy=0x7f82840b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 -- 172.21.15.26:0/1995945651 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82841431e0 con 0x7f828405cfd0 2022-07-20T02:27:27.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.012+0000 7f8283fff700 1 --2- 172.21.15.26:0/1995945651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 0x7f8284143bc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f8274002f70 tx=0x7f8274006df0).ready entity=mon.2 client_cookie=6b416dc7fc18ca13 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8282ffd700 1 -- 172.21.15.26:0/1995945651 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827400d4a0 con 0x7f828405cfd0 2022-07-20T02:27:27.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8282ffd700 1 -- 172.21.15.26:0/1995945651 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274000e90 con 0x7f828405cfd0 2022-07-20T02:27:27.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8282ffd700 1 -- 172.21.15.26:0/1995945651 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827400d7f0 con 0x7f828405cfd0 2022-07-20T02:27:27.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- 172.21.15.26:0/1995945651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 msgr2=0x7f8284143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 --2- 172.21.15.26:0/1995945651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 0x7f8284143bc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f8274002f70 tx=0x7f8274006df0).stop 2022-07-20T02:27:27.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- 172.21.15.26:0/1995945651 shutdown_connections 2022-07-20T02:27:27.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 --2- 172.21.15.26:0/1995945651 >> v2:172.21.15.120:3300/0 conn(0x7f8284060f00 0x7f828413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 --2- 172.21.15.26:0/1995945651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f828405cfd0 0x7f8284143bc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- 172.21.15.26:0/1995945651 >> 172.21.15.26:0/1995945651 conn(0x7f82840620c0 msgr2=0x7f828405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:27.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- 172.21.15.26:0/1995945651 shutdown_connections 2022-07-20T02:27:27.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- 172.21.15.26:0/1995945651 wait complete. 2022-07-20T02:27:27.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 Processor -- start 2022-07-20T02:27:27.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.016+0000 7f8289333700 1 -- start start 2022-07-20T02:27:27.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 0x7f82841393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8284147000 0x7f8284156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284140c30 con 0x7f82840b2280 2022-07-20T02:27:27.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b9080 con 0x7f8284147000 2022-07-20T02:27:27.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b38d0 con 0x7f8284060f00 2022-07-20T02:27:27.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8283fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 0x7f82841393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36810/0 (socket says 172.21.15.26:36810) 2022-07-20T02:27:27.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 -- 172.21.15.26:0/2521142399 learned_addr learned my addr 172.21.15.26:0/2521142399 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:27.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f82837fe700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8284147000 0x7f8284156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 -- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 msgr2=0x7f82841393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 0x7f82841393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 -- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8284147000 msgr2=0x7f8284156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8284147000 0x7f8284156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 -- 172.21.15.26:0/2521142399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827400b180 con 0x7f82840b2280 2022-07-20T02:27:27.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8283fff700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 0x7f82841393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:27.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8288b32700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f827c006e00 tx=0x7f827c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827c012600 con 0x7f82840b2280 2022-07-20T02:27:27.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c012760 con 0x7f82840b2280 2022-07-20T02:27:27.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840b3b70 con 0x7f82840b2280 2022-07-20T02:27:27.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827c012ac0 con 0x7f82840b2280 2022-07-20T02:27:27.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.020+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8284142f20 con 0x7f82840b2280 2022-07-20T02:27:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82840b2bf0 con 0x7f82840b2280 2022-07-20T02:27:27.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f827c022070 con 0x7f82840b2280 2022-07-20T02:27:27.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f82817fa700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 0x7f8264055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f827c06c450 con 0x7f82840b2280 2022-07-20T02:27:27.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f8283fff700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 0x7f8264055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.024+0000 7f8283fff700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 0x7f8264055fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f827401a000 tx=0x7f8274014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.028+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f827c043410 con 0x7f82840b2280 2022-07-20T02:27:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.188+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82840b2060 con 0x7f82840b2280 2022-07-20T02:27:27.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f82817fa700 1 -- 172.21.15.26:0/2521142399 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f827c016020 con 0x7f82840b2280 2022-07-20T02:27:27.202 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:27.221 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10},{"gid":5134,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3193611144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3193611144},{"type":"v1","addr":"172.21.15.26:6835","nonce":3193611144}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[]} 2022-07-20T02:27:27.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 msgr2=0x7f8264055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 0x7f8264055fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f827401a000 tx=0x7f8274014000).stop 2022-07-20T02:27:27.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 msgr2=0x7f8284142a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f827c006e00 tx=0x7f827c0091d0).stop 2022-07-20T02:27:27.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 shutdown_connections 2022-07-20T02:27:27.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f82840b2280 0x7f8284142a30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8264053b00 0x7f8264055fc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8284147000 0x7f8284156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 --2- 172.21.15.26:0/2521142399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8284060f00 0x7f82841393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 >> 172.21.15.26:0/2521142399 conn(0x7f828405cfd0 msgr2=0x7f8284139b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:27.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 shutdown_connections 2022-07-20T02:27:27.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.192+0000 7f8289333700 1 -- 172.21.15.26:0/2521142399 wait complete. 2022-07-20T02:27:27.227 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 15 2022-07-20T02:27:27.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:27.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 Processor -- start 2022-07-20T02:27:27.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 -- start start 2022-07-20T02:27:27.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e80b5d50 0x7fb8e8067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb8e8143070 0x7fb8e8143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e8146750 con 0x7fb8e8140320 2022-07-20T02:27:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e81469f0 con 0x7fb8e8143070 2022-07-20T02:27:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8eca74700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8146c90 con 0x7fb8e80b5d50 2022-07-20T02:27:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb8e8143070 0x7fb8e8143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb8e8143070 0x7fb8e8143490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36814/0 (socket says 172.21.15.26:36814) 2022-07-20T02:27:27.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 -- 172.21.15.26:0/1724365239 learned_addr learned my addr 172.21.15.26:0/1724365239 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:27.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e77fe700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e80b5d50 0x7fb8e8067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 -- 172.21.15.26:0/1724365239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e80b5d50 msgr2=0x7fb8e8067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e80b5d50 0x7fb8e8067490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 -- 172.21.15.26:0/1724365239 >> v1:172.21.15.120:6789/0 conn(0x7fb8e8140320 legacy=0x7fb8e8140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 -- 172.21.15.26:0/1724365239 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8146f30 con 0x7fb8e8143070 2022-07-20T02:27:27.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.336+0000 7fb8e6ffd700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e8143070 0x7fb8e8143490 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8d8002f70 tx=0x7fb8d8006eb0).ready entity=mon.0 client_cookie=efd99eb6b1230c6a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8e67fc700 1 -- 172.21.15.26:0/1724365239 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d800d5c0 con 0x7fb8e8143070 2022-07-20T02:27:27.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8e67fc700 1 -- 172.21.15.26:0/1724365239 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8000e90 con 0x7fb8e8143070 2022-07-20T02:27:27.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8e67fc700 1 -- 172.21.15.26:0/1724365239 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8d800d950 con 0x7fb8e8143070 2022-07-20T02:27:27.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 -- 172.21.15.26:0/1724365239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e8143070 msgr2=0x7fb8e8143490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e8143070 0x7fb8e8143490 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8d8002f70 tx=0x7fb8d8006eb0).stop 2022-07-20T02:27:27.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 -- 172.21.15.26:0/1724365239 shutdown_connections 2022-07-20T02:27:27.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e8143070 0x7fb8e8143490 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 --2- 172.21.15.26:0/1724365239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e80b5d50 0x7fb8e8067490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 -- 172.21.15.26:0/1724365239 >> 172.21.15.26:0/1724365239 conn(0x7fb8e8060310 msgr2=0x7fb8e80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:27.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 -- 172.21.15.26:0/1724365239 shutdown_connections 2022-07-20T02:27:27.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 -- 172.21.15.26:0/1724365239 wait complete. 2022-07-20T02:27:27.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.340+0000 7fb8eca74700 1 Processor -- start 2022-07-20T02:27:27.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- start start 2022-07-20T02:27:27.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e8140320 0x7fb8e8145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8e8143070 0x7fb8e80aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8144b90 con 0x7fb8e80b5d50 2022-07-20T02:27:27.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81445f0 con 0x7fb8e8143070 2022-07-20T02:27:27.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8144dc0 con 0x7fb8e8140320 2022-07-20T02:27:27.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36820/0 (socket says 172.21.15.26:36820) 2022-07-20T02:27:27.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 -- 172.21.15.26:0/3141507749 learned_addr learned my addr 172.21.15.26:0/3141507749 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:27.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e7fff700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8e8143070 0x7fb8e80aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e6ffd700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e8140320 0x7fb8e8145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 -- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e8140320 msgr2=0x7fb8e8145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e8140320 0x7fb8e8145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 -- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8e8143070 msgr2=0x7fb8e80aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8e8143070 0x7fb8e80aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 -- 172.21.15.26:0/3141507749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d800b1e0 con 0x7fb8e80b5d50 2022-07-20T02:27:27.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e77fe700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb8dc0030b0 tx=0x7fb8dc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc011780 con 0x7fb8e80b5d50 2022-07-20T02:27:27.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc0118e0 con 0x7fb8e80b5d50 2022-07-20T02:27:27.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc011c40 con 0x7fb8e80b5d50 2022-07-20T02:27:27.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80aa900 con 0x7fb8e80b5d50 2022-07-20T02:27:27.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.344+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e80ab290 con 0x7fb8e80b5d50 2022-07-20T02:27:27.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8143980 con 0x7fb8e80b5d50 2022-07-20T02:27:27.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb8dc01c460 con 0x7fb8e80b5d50 2022-07-20T02:27:27.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8e4ff9700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 0x7fb8c8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb8dc06c180 con 0x7fb8e80b5d50 2022-07-20T02:27:27.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8e6ffd700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 0x7fb8c8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.348+0000 7fb8e6ffd700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 0x7fb8c8055ed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb8d801c000 tx=0x7fb8d8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.352+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8dc043120 con 0x7fb8e80b5d50 2022-07-20T02:27:27.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.516+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8e80574e0 con 0x7fb8e80b5d50 2022-07-20T02:27:27.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.516+0000 7fb8e4ff9700 1 -- 172.21.15.26:0/3141507749 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1562 (secure 0 0 0) 0x7fb8dc046740 con 0x7fb8e80b5d50 2022-07-20T02:27:27.520 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:27.520 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/651213035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":651213035},{"type":"v1","addr":"172.21.15.120:6833","nonce":651213035}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":4682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/2842264428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2842264428},{"type":"v1","addr":"172.21.15.26:6837","nonce":2842264428}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":10},{"gid":5134,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3193611144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3193611144},{"type":"v1","addr":"172.21.15.26:6835","nonce":3193611144}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[]} 2022-07-20T02:27:27.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.516+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 msgr2=0x7fb8c8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 0x7fb8c8055ed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb8d801c000 tx=0x7fb8d8015000).stop 2022-07-20T02:27:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 msgr2=0x7fb8e81421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb8dc0030b0 tx=0x7fb8dc008040).stop 2022-07-20T02:27:27.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 shutdown_connections 2022-07-20T02:27:27.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8c8053a10 0x7fb8c8055ed0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8e8143070 0x7fb8e80aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8e8140320 0x7fb8e8145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 --2- 172.21.15.26:0/3141507749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8e80b5d50 0x7fb8e81421e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 >> 172.21.15.26:0/3141507749 conn(0x7fb8e8060310 msgr2=0x7fb8e8140760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:27.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 shutdown_connections 2022-07-20T02:27:27.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.520+0000 7fb8eca74700 1 -- 172.21.15.26:0/3141507749 wait complete. 2022-07-20T02:27:27.534 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 15 2022-07-20T02:27:27.537 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:27:27.537 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:27:27.538 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:27:27.539 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:27:27.539 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:27:27.540 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:27:27.540 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:27:27.541 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:27:27.542 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:27:27.550 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:27:27.565 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:27:27.565 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:27:27.570 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:27:27.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:27:27.572 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:27:27.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:27:27.581 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:27:27.581 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:27:27.585 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:27:27.586 DEBUG:teuthology.parallel:result is None 2022-07-20T02:27:27.625 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:27:27.627 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:27:27.627 DEBUG:teuthology.parallel:result is None 2022-07-20T02:27:27.628 DEBUG:teuthology.parallel:result is None 2022-07-20T02:27:27.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:27:27.663 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:27:27.701 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:27:27.707 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:27:27.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.736+0000 7f44b22df700 1 Processor -- start 2022-07-20T02:27:27.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.736+0000 7f44b22df700 1 -- start start 2022-07-20T02:27:27.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 0x7f44ac143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac1474e0 con 0x7f44ac05cfd0 2022-07-20T02:27:27.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac0b4e40 con 0x7f44ac060f00 2022-07-20T02:27:27.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac05d3b0 con 0x7f44ac147000 2022-07-20T02:27:27.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b12dd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f44ac05cfd0 0x7f44ac0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36132/0 (socket says 172.21.15.26:36132) 2022-07-20T02:27:27.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b12dd700 1 -- 172.21.15.26:0/1482039563 learned_addr learned my addr 172.21.15.26:0/1482039563 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:27.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b0adc700 1 --2- 172.21.15.26:0/1482039563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 0x7f44ac143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184101657 0 0) 0x7f44ac1474e0 con 0x7f44ac05cfd0 2022-07-20T02:27:27.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494003260 con 0x7f44ac05cfd0 2022-07-20T02:27:27.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966654198 0 0) 0x7f44ac0b4e40 con 0x7f44ac060f00 2022-07-20T02:27:27.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b0adc700 1 -- 172.21.15.26:0/1482039563 >> v1:172.21.15.120:6789/0 conn(0x7f44ac060f00 legacy=0x7f44ac13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b0adc700 1 -- 172.21.15.26:0/1482039563 >> v1:172.21.15.26:6789/0 conn(0x7f44ac05cfd0 legacy=0x7f44ac0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b0adc700 1 -- 172.21.15.26:0/1482039563 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac1431e0 con 0x7f44ac147000 2022-07-20T02:27:27.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b0adc700 1 --2- 172.21.15.26:0/1482039563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 0x7f44ac143bc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f449c002f70 tx=0x7f449c006df0).ready entity=mon.2 client_cookie=c1fd91c4d4e642e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f449c00d570 con 0x7f44ac147000 2022-07-20T02:27:27.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c000e90 con 0x7f44ac147000 2022-07-20T02:27:27.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f449bfff700 1 -- 172.21.15.26:0/1482039563 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f449c00d8c0 con 0x7f44ac147000 2022-07-20T02:27:27.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- 172.21.15.26:0/1482039563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 msgr2=0x7f44ac143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 --2- 172.21.15.26:0/1482039563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 0x7f44ac143bc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f449c002f70 tx=0x7f449c006df0).stop 2022-07-20T02:27:27.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- 172.21.15.26:0/1482039563 shutdown_connections 2022-07-20T02:27:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 --2- 172.21.15.26:0/1482039563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac147000 0x7f44ac143bc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- 172.21.15.26:0/1482039563 >> 172.21.15.26:0/1482039563 conn(0x7f44ac0620c0 msgr2=0x7f44ac05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- 172.21.15.26:0/1482039563 shutdown_connections 2022-07-20T02:27:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.740+0000 7f44b22df700 1 -- 172.21.15.26:0/1482039563 wait complete. 2022-07-20T02:27:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 Processor -- start 2022-07-20T02:27:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 -- start start 2022-07-20T02:27:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44ac147000 0x7f44ac0b9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac144490 0x7f44ac144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac1422f0 con 0x7f44ac147000 2022-07-20T02:27:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac142d70 con 0x7f44ac060f00 2022-07-20T02:27:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b22df700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac155530 con 0x7f44ac144490 2022-07-20T02:27:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b0adc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44ac147000 0x7f44ac0b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:56550/0 (socket says 172.21.15.26:56550) 2022-07-20T02:27:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 -- 172.21.15.26:0/2613839770 learned_addr learned my addr 172.21.15.26:0/2613839770 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b1ade700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac144490 0x7f44ac144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 -- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac144490 msgr2=0x7f44ac144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac144490 0x7f44ac144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 -- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44ac147000 msgr2=0x7f44ac0b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44ac147000 0x7f44ac0b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 -- 172.21.15.26:0/2613839770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c00b180 con 0x7f44ac060f00 2022-07-20T02:27:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f44b12dd700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f44a8001110 tx=0x7f44a8007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a800ce80 con 0x7f44ac060f00 2022-07-20T02:27:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.744+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a80070d0 con 0x7f44ac060f00 2022-07-20T02:27:27.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac144e30 con 0x7f44ac060f00 2022-07-20T02:27:27.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a8007430 con 0x7f44ac060f00 2022-07-20T02:27:27.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac1426d0 con 0x7f44ac060f00 2022-07-20T02:27:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac142bb0 con 0x7f44ac060f00 2022-07-20T02:27:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f44a8010070 con 0x7f44ac060f00 2022-07-20T02:27:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f449a7fc700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 0x7f448c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:27.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f44a806b2d0 con 0x7f44ac060f00 2022-07-20T02:27:27.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f44b0adc700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 0x7f448c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:27.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.748+0000 7f44b0adc700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 0x7f448c055f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f449c01a000 tx=0x7f449c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:27.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.756+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44a8042310 con 0x7f44ac060f00 2022-07-20T02:27:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:27.896+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f44ac143480 con 0x7f44ac060f00 2022-07-20T02:27:28.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.744+0000 7f449a7fc700 1 -- 172.21.15.26:0/2613839770 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44a8045930 con 0x7f44ac060f00 2022-07-20T02:27:28.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 msgr2=0x7f448c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:28.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 0x7f448c055f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f449c01a000 tx=0x7f449c014000).stop 2022-07-20T02:27:28.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 msgr2=0x7f44ac141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:28.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f44a8001110 tx=0x7f44a8007aa0).stop 2022-07-20T02:27:28.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 shutdown_connections 2022-07-20T02:27:28.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f44ac147000 0x7f44ac0b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f448c053a70 0x7f448c055f30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44ac144490 0x7f44ac144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 --2- 172.21.15.26:0/2613839770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44ac060f00 0x7f44ac141340 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 >> 172.21.15.26:0/2613839770 conn(0x7f44ac05cfd0 msgr2=0x7f44ac138f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:28.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 shutdown_connections 2022-07-20T02:27:28.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.748+0000 7f44b22df700 1 -- 172.21.15.26:0/2613839770 wait complete. 2022-07-20T02:27:28.764 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:27:28.765 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:27:28.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:27:28.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 Processor -- start 2022-07-20T02:27:28.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 -- start start 2022-07-20T02:27:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 0x7f09f8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09f8147000 0x7f09f80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f81474e0 con 0x7f09f8060f00 2022-07-20T02:27:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f805d3b0 con 0x7f09f8147000 2022-07-20T02:27:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09ff3ea700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f81431e0 con 0x7f09f805cfd0 2022-07-20T02:27:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09f8147000 0x7f09f80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f09f8147000 0x7f09f80b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36870/0 (socket says 172.21.15.26:36870) 2022-07-20T02:27:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 -- 172.21.15.26:0/2310830647 learned_addr learned my addr 172.21.15.26:0/2310830647 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fe3e8700 1 --2- 172.21.15.26:0/2310830647 >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 0x7f09f8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 -- 172.21.15.26:0/2310830647 >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 msgr2=0x7f09f8143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:28.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 --2- 172.21.15.26:0/2310830647 >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 0x7f09f8143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 -- 172.21.15.26:0/2310830647 >> v1:172.21.15.120:6789/0 conn(0x7f09f8060f00 legacy=0x7f09f813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:28.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 -- 172.21.15.26:0/2310830647 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8143480 con 0x7f09f8147000 2022-07-20T02:27:28.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fe3e8700 1 --2- 172.21.15.26:0/2310830647 >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 0x7f09f8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:28.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fdbe7700 1 --2- 172.21.15.26:0/2310830647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8147000 0x7f09f80b4730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f09f8068660 tx=0x7f09ec009290).ready entity=mon.0 client_cookie=e8bcaa8ad1d1ec66 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:28.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fd3e6700 1 -- 172.21.15.26:0/2310830647 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09ec00b830 con 0x7f09f8147000 2022-07-20T02:27:28.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fd3e6700 1 -- 172.21.15.26:0/2310830647 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec00b990 con 0x7f09f8147000 2022-07-20T02:27:28.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.868+0000 7f09fd3e6700 1 -- 172.21.15.26:0/2310830647 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09ec00bb40 con 0x7f09f8147000 2022-07-20T02:27:28.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- 172.21.15.26:0/2310830647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8147000 msgr2=0x7f09f80b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:28.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/2310830647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8147000 0x7f09f80b4730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f09f8068660 tx=0x7f09ec009290).stop 2022-07-20T02:27:28.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- 172.21.15.26:0/2310830647 shutdown_connections 2022-07-20T02:27:28.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/2310830647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8147000 0x7f09f80b4730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/2310830647 >> v2:172.21.15.120:3300/0 conn(0x7f09f805cfd0 0x7f09f8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- 172.21.15.26:0/2310830647 >> 172.21.15.26:0/2310830647 conn(0x7f09f80620c0 msgr2=0x7f09f805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:28.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- 172.21.15.26:0/2310830647 shutdown_connections 2022-07-20T02:27:28.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- 172.21.15.26:0/2310830647 wait complete. 2022-07-20T02:27:28.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 Processor -- start 2022-07-20T02:27:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- start start 2022-07-20T02:27:28.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09f8147000 0x7f09f80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09f80b9070 0x7f09f80af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f81501f0 con 0x7f09f8060f00 2022-07-20T02:27:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8155a60 con 0x7f09f8147000 2022-07-20T02:27:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09ff3ea700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f813d270 con 0x7f09f80b9070 2022-07-20T02:27:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36876/0 (socket says 172.21.15.26:36876) 2022-07-20T02:27:28.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 -- 172.21.15.26:0/495021310 learned_addr learned my addr 172.21.15.26:0/495021310 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:28.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fdbe7700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09f8147000 0x7f09f80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09febe9700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09f80b9070 0x7f09f80af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 -- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09f80b9070 msgr2=0x7f09f80af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:28.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09f80b9070 0x7f09f80af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 -- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09f8147000 msgr2=0x7f09f80aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09f8147000 0x7f09f80aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 -- 172.21.15.26:0/495021310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec009530 con 0x7f09f8060f00 2022-07-20T02:27:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.872+0000 7f09fe3e8700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f09f4007000 tx=0x7f09f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09f4010580 con 0x7f09f8060f00 2022-07-20T02:27:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f40106e0 con 0x7f09f8060f00 2022-07-20T02:27:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8156910 con 0x7f09f8060f00 2022-07-20T02:27:28.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09f4010a40 con 0x7f09f8060f00 2022-07-20T02:27:28.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8155f30 con 0x7f09f8060f00 2022-07-20T02:27:28.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8156640 con 0x7f09f8060f00 2022-07-20T02:27:28.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f09f4010ba0 con 0x7f09f8060f00 2022-07-20T02:27:28.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 0x7f09d8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:28.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f09f406be10 con 0x7f09f8060f00 2022-07-20T02:27:28.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09fdbe7700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 0x7f09d8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:28.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.876+0000 7f09fdbe7700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 0x7f09d8055ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f09ec01b000 tx=0x7f09ec016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:28.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:28.884+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f09f406f030 con 0x7f09f8060f00 2022-07-20T02:27:29.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f09f8143720 con 0x7f09f8060f00 2022-07-20T02:27:29.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09eb7fe700 1 -- 172.21.15.26:0/495021310 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f09f4042da0 con 0x7f09f8060f00 2022-07-20T02:27:29.056 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:29.056 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:27:29.056 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:27:29.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.058 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:27:29.058 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:27:29.059 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:27:29.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.073 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.073 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.075 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:27:29.076 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:27:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:27:29.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.080 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.082 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:27:29.082 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:27:29.083 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:27:29.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.085 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.086 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:27:29.087 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:27:29.087 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.088 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.088 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.089 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.091 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:27:29.091 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:27:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.093 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.093 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.115 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:27:29.115 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:27:29.115 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.117 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.118 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:27:29.118 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:27:29.119 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.119 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.119 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.119 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.121 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:27:29.121 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:27:29.121 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.121 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:27:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:27:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.126 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:27:29.127 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:27:29.127 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.128 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.129 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:27:29.129 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:27:29.129 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.130 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.130 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.130 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.131 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:27:29.131 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:27:29.131 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.132 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:27:29.132 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:27:29.133 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:27:29.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.134 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:27:29.134 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:27:29.134 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:27:29.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:27:29.135 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:27:29.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:27:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:27:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:27:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:27:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:27:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:27:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:27:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:27:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:27:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:27:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:27:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:27:29.142 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:27:29.142 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:27:29.142 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:27:29.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:29.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:27:29.143 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:27:29.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 msgr2=0x7f09d8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 0x7f09d8055ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f09ec01b000 tx=0x7f09ec016000).stop 2022-07-20T02:27:29.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 msgr2=0x7f09f81421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.052+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f09f4007000 tx=0x7f09f4008040).stop 2022-07-20T02:27:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 shutdown_connections 2022-07-20T02:27:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09d80539e0 0x7f09d8055ea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09f80b9070 0x7f09f80af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09f8147000 0x7f09f80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 --2- 172.21.15.26:0/495021310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09f8060f00 0x7f09f81421d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 >> 172.21.15.26:0/495021310 conn(0x7f09f805cfd0 msgr2=0x7f09f813a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 shutdown_connections 2022-07-20T02:27:29.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.056+0000 7f09ff3ea700 1 -- 172.21.15.26:0/495021310 wait complete. 2022-07-20T02:27:29.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:27:29.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.276+0000 7f94ef59e700 1 Processor -- start 2022-07-20T02:27:29.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.276+0000 7f94ef59e700 1 -- start start 2022-07-20T02:27:29.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ef59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94f0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ef59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94f0147000 0x7f94f00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ef59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f01474e0 con 0x7f94f0060f00 2022-07-20T02:27:29.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ef59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f005d3b0 con 0x7f94f0147000 2022-07-20T02:27:29.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ef59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01431e0 con 0x7f94f005cfd0 2022-07-20T02:27:29.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94eed9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f94f0060f00 0x7f94f013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36168/0 (socket says 172.21.15.26:36168) 2022-07-20T02:27:29.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94eed9d700 1 -- 172.21.15.26:0/2034771783 learned_addr learned my addr 172.21.15.26:0/2034771783 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:29.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94f0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94e7fff700 1 --2- 172.21.15.26:0/2034771783 >> v2:172.21.15.120:3300/0 conn(0x7f94f0147000 0x7f94f00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081601105 0 0) 0x7f94f01474e0 con 0x7f94f0060f00 2022-07-20T02:27:29.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d0003260 con 0x7f94f0060f00 2022-07-20T02:27:29.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776735321 0 0) 0x7f94d0003260 con 0x7f94f0060f00 2022-07-20T02:27:29.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 msgr2=0x7f94f0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94f0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 >> v2:172.21.15.120:3300/0 conn(0x7f94f0147000 msgr2=0x7f94f00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 --2- 172.21.15.26:0/2034771783 >> v2:172.21.15.120:3300/0 conn(0x7f94f0147000 0x7f94f00b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f0143480 con 0x7f94f0060f00 2022-07-20T02:27:29.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f94e80019e0 con 0x7f94f0060f00 2022-07-20T02:27:29.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 >> v1:172.21.15.26:6789/0 conn(0x7f94f0060f00 legacy=0x7f94f013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94d0003750 0x7f94d0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94d0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94d00049f0 0x7f94d0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f005d3b0 con 0x7f94d0003750 2022-07-20T02:27:29.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01431e0 con 0x7f94d00049f0 2022-07-20T02:27:29.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00042d0 con 0x7f94f005cfd0 2022-07-20T02:27:29.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94d0003750 0x7f94d0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94eed9d700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94d00049f0 0x7f94d0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94e7fff700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94d0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 -- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 msgr2=0x7f94d0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94d0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 -- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94d00049f0 msgr2=0x7f94d0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94d00049f0 0x7f94d0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 -- 172.21.15.26:0/2034771783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0003260 con 0x7f94d0003750 2022-07-20T02:27:29.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94eed9d700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94d00049f0 0x7f94d0004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:29.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94e7fff700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f005cfd0 0x7f94d0004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94ee59c700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94d0003750 0x7f94d0003b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f94e000c000 tx=0x7f94e0006df0).ready entity=mon.0 client_cookie=d0e486ed531defea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94e000e830 con 0x7f94d0003750 2022-07-20T02:27:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000e990 con 0x7f94d0003750 2022-07-20T02:27:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.280+0000 7f94edd9b700 1 -- 172.21.15.26:0/2034771783 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94e000ecf0 con 0x7f94d0003750 2022-07-20T02:27:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- 172.21.15.26:0/2034771783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94d0003750 msgr2=0x7f94d0003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 --2- 172.21.15.26:0/2034771783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94d0003750 0x7f94d0003b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f94e000c000 tx=0x7f94e0006df0).stop 2022-07-20T02:27:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94eed9d700 1 -- 172.21.15.26:0/2034771783 reap_dead start 2022-07-20T02:27:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- 172.21.15.26:0/2034771783 shutdown_connections 2022-07-20T02:27:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- 172.21.15.26:0/2034771783 >> 172.21.15.26:0/2034771783 conn(0x7f94f00620c0 msgr2=0x7f94f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- 172.21.15.26:0/2034771783 shutdown_connections 2022-07-20T02:27:29.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- 172.21.15.26:0/2034771783 wait complete. 2022-07-20T02:27:29.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 Processor -- start 2022-07-20T02:27:29.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- start start 2022-07-20T02:27:29.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f0155c90 0x7f94f00b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94f0138020 0x7f94f01384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00b7640 con 0x7f94f01469b0 2022-07-20T02:27:29.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01564d0 con 0x7f94f0138020 2022-07-20T02:27:29.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ef59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0156770 con 0x7f94f0155c90 2022-07-20T02:27:29.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94e7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94e7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36898/0 (socket says 172.21.15.26:36898) 2022-07-20T02:27:29.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94e7fff700 1 -- 172.21.15.26:0/758056151 learned_addr learned my addr 172.21.15.26:0/758056151 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:29.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.284+0000 7f94ee59c700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f0155c90 0x7f94f00b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94eed9d700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94f0138020 0x7f94f01384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 -- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f0155c90 msgr2=0x7f94f00b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f0155c90 0x7f94f00b8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 -- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94f0138020 msgr2=0x7f94f01384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94f0138020 0x7f94f01384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 -- 172.21.15.26:0/758056151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f0144fc0 con 0x7f94f01469b0 2022-07-20T02:27:29.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e7fff700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f94d800b000 tx=0x7f94d8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d8014680 con 0x7f94f01469b0 2022-07-20T02:27:29.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d80147e0 con 0x7f94f01469b0 2022-07-20T02:27:29.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d8014b40 con 0x7f94f01469b0 2022-07-20T02:27:29.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0145460 con 0x7f94f01469b0 2022-07-20T02:27:29.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f01389c0 con 0x7f94f01469b0 2022-07-20T02:27:29.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f0145150 con 0x7f94f01469b0 2022-07-20T02:27:29.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f94d8024070 con 0x7f94f01469b0 2022-07-20T02:27:29.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.288+0000 7f94e77fe700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 0x7f94d4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.292+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f94d806c420 con 0x7f94f01469b0 2022-07-20T02:27:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.292+0000 7f94ee59c700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 0x7f94d4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.292+0000 7f94ee59c700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 0x7f94d4055e40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f94e001a000 tx=0x7f94e0019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.296+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94d8043520 con 0x7f94f01469b0 2022-07-20T02:27:29.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.428+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94f0143720 con 0x7f94f01469b0 2022-07-20T02:27:29.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.428+0000 7f94e77fe700 1 -- 172.21.15.26:0/758056151 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+13462 (secure 0 0 0) 0x7f94d8046b40 con 0x7f94f01469b0 2022-07-20T02:27:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:29.439 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":41,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:28.736146+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:27:29.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 msgr2=0x7f94d4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 0x7f94d4055e40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f94e001a000 tx=0x7f94e0019000).stop 2022-07-20T02:27:29.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 msgr2=0x7f94f00b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f94d800b000 tx=0x7f94d8008040).stop 2022-07-20T02:27:29.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 shutdown_connections 2022-07-20T02:27:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f94f01469b0 0x7f94f00b7f90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f94d4053980 0x7f94d4055e40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94f0138020 0x7f94f01384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 --2- 172.21.15.26:0/758056151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94f0155c90 0x7f94f00b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 >> 172.21.15.26:0/758056151 conn(0x7f94d0003750 msgr2=0x7f94f013b060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 shutdown_connections 2022-07-20T02:27:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.432+0000 7f94ef59e700 1 -- 172.21.15.26:0/758056151 wait complete. 2022-07-20T02:27:29.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:27:29.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:27:29.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.584+0000 7f027d295700 1 Processor -- start 2022-07-20T02:27:29.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.584+0000 7f027d295700 1 -- start start 2022-07-20T02:27:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780b2ab0 0x7f0278061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02780af6d0 0x7f02780afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02780aa1e0 con 0x7f02780add30 2022-07-20T02:27:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02780ae110 con 0x7f02780af6d0 2022-07-20T02:27:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02780a9f40 con 0x7f02780b2ab0 2022-07-20T02:27:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02780af6d0 0x7f02780afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0277fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f02780add30 0x7f0278060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36184/0 (socket says 172.21.15.26:36184) 2022-07-20T02:27:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0277fff700 1 -- 172.21.15.26:0/1164498751 learned_addr learned my addr 172.21.15.26:0/1164498751 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f02777fe700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780b2ab0 0x7f0278061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0276ffd700 1 -- 172.21.15.26:0/1164498751 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699737842 0 0) 0x7f02780aa1e0 con 0x7f02780add30 2022-07-20T02:27:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0276ffd700 1 -- 172.21.15.26:0/1164498751 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0264003260 con 0x7f02780add30 2022-07-20T02:27:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 -- 172.21.15.26:0/1164498751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780b2ab0 msgr2=0x7f0278061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780b2ab0 0x7f0278061060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 -- 172.21.15.26:0/1164498751 >> v1:172.21.15.26:6789/0 conn(0x7f02780add30 legacy=0x7f0278060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 -- 172.21.15.26:0/1164498751 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02780b1110 con 0x7f02780af6d0 2022-07-20T02:27:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027ca94700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780af6d0 0x7f02780afb60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f02680032f0 tx=0x7f026800b470).ready entity=mon.0 client_cookie=689ee6ff8e2537ea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0276ffd700 1 -- 172.21.15.26:0/1164498751 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026800dd70 con 0x7f02780af6d0 2022-07-20T02:27:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0276ffd700 1 -- 172.21.15.26:0/1164498751 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0268010470 con 0x7f02780af6d0 2022-07-20T02:27:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f0276ffd700 1 -- 172.21.15.26:0/1164498751 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02680107d0 con 0x7f02780af6d0 2022-07-20T02:27:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- 172.21.15.26:0/1164498751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780af6d0 msgr2=0x7f02780afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780af6d0 0x7f02780afb60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f02680032f0 tx=0x7f026800b470).stop 2022-07-20T02:27:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- 172.21.15.26:0/1164498751 shutdown_connections 2022-07-20T02:27:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780af6d0 0x7f02780afb60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 --2- 172.21.15.26:0/1164498751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780b2ab0 0x7f0278061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- 172.21.15.26:0/1164498751 >> 172.21.15.26:0/1164498751 conn(0x7f02780b3050 msgr2=0x7f02780b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- 172.21.15.26:0/1164498751 shutdown_connections 2022-07-20T02:27:29.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.588+0000 7f027d295700 1 -- 172.21.15.26:0/1164498751 wait complete. 2022-07-20T02:27:29.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 Processor -- start 2022-07-20T02:27:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- start start 2022-07-20T02:27:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780add30 0x7f0278064f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02780af6d0 0x7f0278144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0278060ab0 con 0x7f02780b2ab0 2022-07-20T02:27:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0278060d50 con 0x7f02780af6d0 2022-07-20T02:27:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0278065630 con 0x7f02780add30 2022-07-20T02:27:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36914/0 (socket says 172.21.15.26:36914) 2022-07-20T02:27:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 -- 172.21.15.26:0/835404537 learned_addr learned my addr 172.21.15.26:0/835404537 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f02777fe700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02780af6d0 0x7f0278144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f0277fff700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780add30 0x7f0278064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 -- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780add30 msgr2=0x7f0278064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780add30 0x7f0278064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 -- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02780af6d0 msgr2=0x7f0278144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02780af6d0 0x7f0278144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 -- 172.21.15.26:0/835404537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026800b710 con 0x7f02780b2ab0 2022-07-20T02:27:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027ca94700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f02680032f0 tx=0x7f02780af5f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026801d070 con 0x7f02780b2ab0 2022-07-20T02:27:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0268018800 con 0x7f02780b2ab0 2022-07-20T02:27:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02780658d0 con 0x7f02780b2ab0 2022-07-20T02:27:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0268018b60 con 0x7f02780b2ab0 2022-07-20T02:27:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.592+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027805cae0 con 0x7f02780b2ab0 2022-07-20T02:27:29.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0278063d80 con 0x7f02780b2ab0 2022-07-20T02:27:29.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0268029070 con 0x7f02780b2ab0 2022-07-20T02:27:29.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f02757fa700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 0x7f0258055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f026806d410 con 0x7f02780b2ab0 2022-07-20T02:27:29.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f0277fff700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 0x7f0258055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.596+0000 7f0277fff700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 0x7f0258055e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f027000d000 tx=0x7f0270009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.604+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0268044510 con 0x7f02780b2ab0 2022-07-20T02:27:29.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.732+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f02780b13b0 con 0x7f02780b2ab0 2022-07-20T02:27:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.732+0000 7f02757fa700 1 -- 172.21.15.26:0/835404537 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0268047b30 con 0x7f02780b2ab0 2022-07-20T02:27:29.755 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:27:29.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 msgr2=0x7f0258055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 0x7f0258055e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f027000d000 tx=0x7f0270009040).stop 2022-07-20T02:27:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 msgr2=0x7f027813eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f02680032f0 tx=0x7f02780af5f0).stop 2022-07-20T02:27:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 shutdown_connections 2022-07-20T02:27:29.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02780b2ab0 0x7f027813eb90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0258053950 0x7f0258055e10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02780af6d0 0x7f0278144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 --2- 172.21.15.26:0/835404537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02780add30 0x7f0278064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 >> 172.21.15.26:0/835404537 conn(0x7f02780b3050 msgr2=0x7f0278066a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 shutdown_connections 2022-07-20T02:27:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.736+0000 7f027d295700 1 -- 172.21.15.26:0/835404537 wait complete. 2022-07-20T02:27:29.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:27:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 Processor -- start 2022-07-20T02:27:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 -- start start 2022-07-20T02:27:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdbac05c720 0x7fdbac147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac05d4d0 con 0x7fdbac0b1200 2022-07-20T02:27:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbac0a9ff0 con 0x7fdbac05c720 2022-07-20T02:27:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbb0ca1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac067e60 con 0x7fdbac0afe40 2022-07-20T02:27:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbabfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdbac0b1200 0x7fdbac0abbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36202/0 (socket says 172.21.15.26:36202) 2022-07-20T02:27:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbabfff700 1 -- 172.21.15.26:0/749189796 learned_addr learned my addr 172.21.15.26:0/749189796 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbab7fe700 1 --2- 172.21.15.26:0/749189796 >> v2:172.21.15.120:3300/0 conn(0x7fdbac05c720 0x7fdbac147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/749189796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaa7fc700 1 -- 172.21.15.26:0/749189796 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601419814 0 0) 0x7fdbac05d4d0 con 0x7fdbac0b1200 2022-07-20T02:27:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaa7fc700 1 -- 172.21.15.26:0/749189796 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb8c003260 con 0x7fdbac0b1200 2022-07-20T02:27:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 -- 172.21.15.26:0/749189796 >> v2:172.21.15.120:3300/0 conn(0x7fdbac05c720 msgr2=0x7fdbac147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/749189796 >> v2:172.21.15.120:3300/0 conn(0x7fdbac05c720 0x7fdbac147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 -- 172.21.15.26:0/749189796 >> v1:172.21.15.26:6789/0 conn(0x7fdbac0b1200 legacy=0x7fdbac0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 -- 172.21.15.26:0/749189796 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac067bc0 con 0x7fdbac0afe40 2022-07-20T02:27:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.892+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/749189796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac147a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdb94002f70 tx=0x7fdb94006df0).ready entity=mon.2 client_cookie=54a1299beee23880 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbaa7fc700 1 -- 172.21.15.26:0/749189796 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9400d4a0 con 0x7fdbac0afe40 2022-07-20T02:27:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbaa7fc700 1 -- 172.21.15.26:0/749189796 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94000e90 con 0x7fdbac0afe40 2022-07-20T02:27:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbaa7fc700 1 -- 172.21.15.26:0/749189796 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9400d7f0 con 0x7fdbac0afe40 2022-07-20T02:27:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/749189796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 msgr2=0x7fdbac147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/749189796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac147a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdb94002f70 tx=0x7fdb94006df0).stop 2022-07-20T02:27:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/749189796 shutdown_connections 2022-07-20T02:27:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/749189796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac147a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/749189796 >> v2:172.21.15.120:3300/0 conn(0x7fdbac05c720 0x7fdbac147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/749189796 >> 172.21.15.26:0/749189796 conn(0x7fdbac060310 msgr2=0x7fdbac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/749189796 shutdown_connections 2022-07-20T02:27:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/749189796 wait complete. 2022-07-20T02:27:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 Processor -- start 2022-07-20T02:27:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.896+0000 7fdbb0ca1700 1 -- start start 2022-07-20T02:27:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac066d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbac0b1200 0x7fdbac067250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac0652c0 con 0x7fdbac05c720 2022-07-20T02:27:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac067740 con 0x7fdbac0b1200 2022-07-20T02:27:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac0654f0 con 0x7fdbac0afe40 2022-07-20T02:27:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36924/0 (socket says 172.21.15.26:36924) 2022-07-20T02:27:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 -- 172.21.15.26:0/3965913003 learned_addr learned my addr 172.21.15.26:0/3965913003 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac066d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbabfff700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbac0b1200 0x7fdbac067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 -- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 msgr2=0x7fdbac066d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac066d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 -- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbac0b1200 msgr2=0x7fdbac067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbac0b1200 0x7fdbac067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 -- 172.21.15.26:0/3965913003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9400b180 con 0x7fdbac05c720 2022-07-20T02:27:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbab7fe700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fdb9c00c000 tx=0x7fdb9c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9c012600 con 0x7fdbac05c720 2022-07-20T02:27:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbac065790 con 0x7fdbac05c720 2022-07-20T02:27:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c012760 con 0x7fdbac05c720 2022-07-20T02:27:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9c012ac0 con 0x7fdbac05c720 2022-07-20T02:27:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.900+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbac064700 con 0x7fdbac05c720 2022-07-20T02:27:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbac066a70 con 0x7fdbac05c720 2022-07-20T02:27:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb9c022070 con 0x7fdbac05c720 2022-07-20T02:27:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdba8ff9700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 0x7fdb90055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fdb9c06c340 con 0x7fdbac05c720 2022-07-20T02:27:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 0x7fdb90055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.904+0000 7fdbaaffd700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 0x7fdb90055f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdb9401a000 tx=0x7fdb94014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:29.908+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb9c0433a0 con 0x7fdbac05c720 2022-07-20T02:27:30.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.036+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdbac0aa250 con 0x7fdbac05c720 2022-07-20T02:27:30.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.036+0000 7fdba8ff9700 1 -- 172.21.15.26:0/3965913003 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v41) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdb9c0469c0 con 0x7fdbac05c720 2022-07-20T02:27:30.046 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:27:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 msgr2=0x7fdb90055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 0x7fdb90055f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdb9401a000 tx=0x7fdb94014000).stop 2022-07-20T02:27:30.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 msgr2=0x7fdbac14d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fdb9c00c000 tx=0x7fdb9c0091d0).stop 2022-07-20T02:27:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 shutdown_connections 2022-07-20T02:27:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb90053a70 0x7fdb90055f30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbac0b1200 0x7fdbac067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbac0afe40 0x7fdbac066d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 --2- 172.21.15.26:0/3965913003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbac05c720 0x7fdbac14d530 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 >> 172.21.15.26:0/3965913003 conn(0x7fdbac060310 msgr2=0x7fdbac0a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 shutdown_connections 2022-07-20T02:27:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.040+0000 7fdbb0ca1700 1 -- 172.21.15.26:0/3965913003 wait complete. 2022-07-20T02:27:30.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:27:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.168+0000 7fbae52a2700 1 Processor -- start 2022-07-20T02:27:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.168+0000 7fbae52a2700 1 -- start start 2022-07-20T02:27:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbae01468c0 0x7fbae00b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae0061e00 con 0x7fbae0061a20 2022-07-20T02:27:30.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae012cc40 con 0x7fbae01468c0 2022-07-20T02:27:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0147040 con 0x7fbae01432e0 2022-07-20T02:27:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:58448/0 (socket says 172.21.15.26:58448) 2022-07-20T02:27:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 -- 172.21.15.26:0/310549349 learned_addr learned my addr 172.21.15.26:0/310549349 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbadf7fe700 1 --2- 172.21.15.26:0/310549349 >> v2:172.21.15.120:3300/0 conn(0x7fbae01468c0 0x7fbae00b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 -- 172.21.15.26:0/310549349 >> v2:172.21.15.120:3300/0 conn(0x7fbae01468c0 msgr2=0x7fbae00b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 --2- 172.21.15.26:0/310549349 >> v2:172.21.15.120:3300/0 conn(0x7fbae01468c0 0x7fbae00b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 -- 172.21.15.26:0/310549349 >> v1:172.21.15.120:6789/0 conn(0x7fbae0061a20 legacy=0x7fbae012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 -- 172.21.15.26:0/310549349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae01472e0 con 0x7fbae01432e0 2022-07-20T02:27:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbade7fc700 1 --2- 172.21.15.26:0/310549349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbac80032f0 tx=0x7fbac8009290).ready entity=mon.2 client_cookie=289668e7b3297659 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbaddffb700 1 -- 172.21.15.26:0/310549349 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbac800b8a0 con 0x7fbae01432e0 2022-07-20T02:27:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbaddffb700 1 -- 172.21.15.26:0/310549349 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac800ba00 con 0x7fbae01432e0 2022-07-20T02:27:30.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbaddffb700 1 -- 172.21.15.26:0/310549349 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbac800bb70 con 0x7fbae01432e0 2022-07-20T02:27:30.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- 172.21.15.26:0/310549349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 msgr2=0x7fbae012c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 --2- 172.21.15.26:0/310549349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbac80032f0 tx=0x7fbac8009290).stop 2022-07-20T02:27:30.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- 172.21.15.26:0/310549349 shutdown_connections 2022-07-20T02:27:30.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 --2- 172.21.15.26:0/310549349 >> v2:172.21.15.120:3300/0 conn(0x7fbae01468c0 0x7fbae00b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 --2- 172.21.15.26:0/310549349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae01432e0 0x7fbae012c750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- 172.21.15.26:0/310549349 >> 172.21.15.26:0/310549349 conn(0x7fbae005faf0 msgr2=0x7fbae0066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:30.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- 172.21.15.26:0/310549349 shutdown_connections 2022-07-20T02:27:30.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.172+0000 7fbae52a2700 1 -- 172.21.15.26:0/310549349 wait complete. 2022-07-20T02:27:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 Processor -- start 2022-07-20T02:27:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 -- start start 2022-07-20T02:27:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 0x7fbae012a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbae01468c0 0x7fbae00b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00b08c0 con 0x7fbae01468c0 2022-07-20T02:27:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00b5a40 con 0x7fbae01432e0 2022-07-20T02:27:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbae52a2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00b9ea0 con 0x7fbae0061a20 2022-07-20T02:27:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbade7fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbadf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbae01468c0 0x7fbae00b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbadeffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 0x7fbae012a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbade7fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:56630/0 (socket says 172.21.15.26:56630) 2022-07-20T02:27:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.176+0000 7fbade7fc700 1 -- 172.21.15.26:0/2766251586 learned_addr learned my addr 172.21.15.26:0/2766251586 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 -- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 msgr2=0x7fbae00b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 -- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbae01468c0 msgr2=0x7fbae00b8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbae01468c0 0x7fbae00b8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 -- 172.21.15.26:0/2766251586 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8009530 con 0x7fbae0061a20 2022-07-20T02:27:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbade7fc700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbadeffd700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 0x7fbae012a070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fbae00b7930 tx=0x7fbad0007b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad0007030 con 0x7fbae0061a20 2022-07-20T02:27:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad0007190 con 0x7fbae0061a20 2022-07-20T02:27:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae00b7af0 con 0x7fbae0061a20 2022-07-20T02:27:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad0006510 con 0x7fbae0061a20 2022-07-20T02:27:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae00b91b0 con 0x7fbae0061a20 2022-07-20T02:27:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae0147580 con 0x7fbae0061a20 2022-07-20T02:27:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbad0010030 con 0x7fbae0061a20 2022-07-20T02:27:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 0x7fbac4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fbad006b8f0 con 0x7fbae0061a20 2022-07-20T02:27:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.180+0000 7fbade7fc700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 0x7fbac4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.184+0000 7fbade7fc700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 0x7fbac4055ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fbac801b000 tx=0x7fbac8016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.192+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbad00428e0 con 0x7fbae0061a20 2022-07-20T02:27:30.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.320+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbae00574e0 con 0x7fbae0061a20 2022-07-20T02:27:30.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.320+0000 7fbad7fff700 1 -- 172.21.15.26:0/2766251586 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+143 (secure 0 0 0) 0x7fbad0045f00 con 0x7fbae0061a20 2022-07-20T02:27:30.326 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:30.326 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:27:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:30.327 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:27:30.328 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:27:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:27:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:27:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:27:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:27:30.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:27:30.330 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:27:30.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 msgr2=0x7fbac4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 0x7fbac4055ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fbac801b000 tx=0x7fbac8016000).stop 2022-07-20T02:27:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 msgr2=0x7fbae012a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 0x7fbae012a070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fbae00b7930 tx=0x7fbad0007b50).stop 2022-07-20T02:27:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 shutdown_connections 2022-07-20T02:27:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbae01468c0 0x7fbae00b8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbac4053a10 0x7fbac4055ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbae01432e0 0x7fbae00b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 --2- 172.21.15.26:0/2766251586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbae0061a20 0x7fbae012a070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 >> 172.21.15.26:0/2766251586 conn(0x7fbae005faf0 msgr2=0x7fbae012c320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:30.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 shutdown_connections 2022-07-20T02:27:30.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.324+0000 7fbae52a2700 1 -- 172.21.15.26:0/2766251586 wait complete. 2022-07-20T02:27:30.338 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:27:30.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:27:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.464+0000 7fae6d7e9700 1 Processor -- start 2022-07-20T02:27:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.464+0000 7fae6d7e9700 1 -- start start 2022-07-20T02:27:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6d7e9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae680b3260 0x7fae68067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6d7e9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6805ff70 con 0x7fae68061860 2022-07-20T02:27:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6d7e9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae680606e0 con 0x7fae680980d0 2022-07-20T02:27:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6d7e9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae680b3640 con 0x7fae680b3260 2022-07-20T02:27:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae5ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fae68061860 0x7fae680979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36226/0 (socket says 172.21.15.26:36226) 2022-07-20T02:27:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae5ffff700 1 -- 172.21.15.26:0/3851463738 learned_addr learned my addr 172.21.15.26:0/3851463738 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> v2:172.21.15.120:3300/0 conn(0x7fae680b3260 0x7fae68067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933495210 0 0) 0x7fae6805ff70 con 0x7fae68061860 2022-07-20T02:27:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48003260 con 0x7fae68061860 2022-07-20T02:27:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071569940 0 0) 0x7fae680606e0 con 0x7fae680980d0 2022-07-20T02:27:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae6805ff70 con 0x7fae680980d0 2022-07-20T02:27:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3920772045 0 0) 0x7fae48003260 con 0x7fae68061860 2022-07-20T02:27:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 >> v2:172.21.15.120:3300/0 conn(0x7fae680b3260 msgr2=0x7fae68067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 --2- 172.21.15.26:0/3851463738 >> v2:172.21.15.120:3300/0 conn(0x7fae680b3260 0x7fae68067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 >> v1:172.21.15.120:6789/0 conn(0x7fae680980d0 legacy=0x7fae68142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680601a0 con 0x7fae68061860 2022-07-20T02:27:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fae50001660 con 0x7fae68061860 2022-07-20T02:27:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> v2:172.21.15.120:3300/0 conn(0x7fae680b3260 0x7fae68067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 >> v1:172.21.15.26:6789/0 conn(0x7fae68061860 legacy=0x7fae680979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 0x7fae48003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae48004500 0x7fae48004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae48004e70 0x7fae4800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae680b3640 con 0x7fae480038e0 2022-07-20T02:27:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4800bcf0 con 0x7fae48004e70 2022-07-20T02:27:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4800beb0 con 0x7fae48004500 2022-07-20T02:27:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 0x7fae48003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae5ffff700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae48004500 0x7fae48004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6cfe8700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae48004e70 0x7fae4800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 -- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae48004500 msgr2=0x7fae48004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:30.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae48004500 0x7fae48004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 -- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae48004e70 msgr2=0x7fae4800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:30.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae48004e70 0x7fae4800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae6cfe8700 1 -- 172.21.15.26:0/3851463738 reap_dead start 2022-07-20T02:27:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 -- 172.21.15.26:0/3851463738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae48003260 con 0x7fae480038e0 2022-07-20T02:27:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae67fff700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 0x7fae48003ce0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fae5800c000 tx=0x7fae58006df0).ready entity=mon.0 client_cookie=2aef34ef513c496c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.468+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5800e800 con 0x7fae480038e0 2022-07-20T02:27:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5800e960 con 0x7fae480038e0 2022-07-20T02:27:30.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae677fe700 1 -- 172.21.15.26:0/3851463738 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5800ecc0 con 0x7fae480038e0 2022-07-20T02:27:30.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 msgr2=0x7fae48003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 0x7fae48003ce0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fae5800c000 tx=0x7fae58006df0).stop 2022-07-20T02:27:30.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3851463738 shutdown_connections 2022-07-20T02:27:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3851463738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae480038e0 0x7fae48003ce0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3851463738 >> 172.21.15.26:0/3851463738 conn(0x7fae680b67f0 msgr2=0x7fae680b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3851463738 shutdown_connections 2022-07-20T02:27:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3851463738 wait complete. 2022-07-20T02:27:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 Processor -- start 2022-07-20T02:27:30.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- start start 2022-07-20T02:27:30.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 0x7fae6813d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae6813ba30 0x7fae6813be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809aeb0 con 0x7fae680b4cd0 2022-07-20T02:27:30.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809a4d0 con 0x7fae6813ba30 2022-07-20T02:27:30.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6d7e9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809a770 con 0x7fae480038e0 2022-07-20T02:27:30.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36962/0 (socket says 172.21.15.26:36962) 2022-07-20T02:27:30.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 -- 172.21.15.26:0/3641913555 learned_addr learned my addr 172.21.15.26:0/3641913555 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:30.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae67fff700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 0x7fae6813d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae5ffff700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae6813ba30 0x7fae6813be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 -- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 msgr2=0x7fae6813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 0x7fae6813d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 -- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae6813ba30 msgr2=0x7fae6813be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:30.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae6813ba30 0x7fae6813be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:30.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae6cfe8700 1 -- 172.21.15.26:0/3641913555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6809aa10 con 0x7fae680b4cd0 2022-07-20T02:27:30.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.472+0000 7fae67fff700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 0x7fae6813d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:30.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae6cfe8700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fae680978a0 tx=0x7fae60007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae60006d60 con 0x7fae680b4cd0 2022-07-20T02:27:30.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae60007070 con 0x7fae680b4cd0 2022-07-20T02:27:30.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae680b4340 con 0x7fae680b4cd0 2022-07-20T02:27:30.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae600073d0 con 0x7fae680b4cd0 2022-07-20T02:27:30.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6813db70 con 0x7fae680b4cd0 2022-07-20T02:27:30.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6809aba0 con 0x7fae680b4cd0 2022-07-20T02:27:30.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae6001b070 con 0x7fae680b4cd0 2022-07-20T02:27:30.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 0x7fae4c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:30.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fae6006b0a0 con 0x7fae680b4cd0 2022-07-20T02:27:30.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.476+0000 7fae67fff700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 0x7fae4c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:30.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.480+0000 7fae67fff700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 0x7fae4c055ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fae5801d000 tx=0x7fae58019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:30.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.488+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae600421a0 con 0x7fae680b4cd0 2022-07-20T02:27:30.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:30.612+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fae680574e0 con 0x7fae680b4cd0 2022-07-20T02:27:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.588+0000 7fae65ffb700 1 -- 172.21.15.26:0/3641913555 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7fae600457c0 con 0x7fae680b4cd0 2022-07-20T02:27:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 msgr2=0x7fae4c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 0x7fae4c055ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fae5801d000 tx=0x7fae58019000).stop 2022-07-20T02:27:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 msgr2=0x7fae680b5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fae680978a0 tx=0x7fae60007d80).stop 2022-07-20T02:27:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 shutdown_connections 2022-07-20T02:27:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae680b4cd0 0x7fae680b5150 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae4c053b30 0x7fae4c055ff0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae6813ba30 0x7fae6813be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 --2- 172.21.15.26:0/3641913555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae480038e0 0x7fae6813d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 >> 172.21.15.26:0/3641913555 conn(0x7fae680b67f0 msgr2=0x7fae680945d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 shutdown_connections 2022-07-20T02:27:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.592+0000 7fae6d7e9700 1 -- 172.21.15.26:0/3641913555 wait complete. 2022-07-20T02:27:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:27:31.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:27:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 Processor -- start 2022-07-20T02:27:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 -- start start 2022-07-20T02:27:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00a00b2ac0 0x7f00a00aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00b2ea0 con 0x7f00a0148580 2022-07-20T02:27:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a0147900 con 0x7f00a00a7620 2022-07-20T02:27:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f00a5004700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a0147ba0 con 0x7f00a00b2ac0 2022-07-20T02:27:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009f59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f00a0148580 0x7f00a0066f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36250/0 (socket says 172.21.15.26:36250) 2022-07-20T02:27:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009f59e700 1 -- 172.21.15.26:0/380960152 learned_addr learned my addr 172.21.15.26:0/380960152 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009ed9d700 1 --2- 172.21.15.26:0/380960152 >> v2:172.21.15.120:3300/0 conn(0x7f00a00b2ac0 0x7f00a00aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009dd9b700 1 -- 172.21.15.26:0/380960152 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523716445 0 0) 0x7f00a00b2ea0 con 0x7f00a0148580 2022-07-20T02:27:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009dd9b700 1 -- 172.21.15.26:0/380960152 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0084003260 con 0x7f00a0148580 2022-07-20T02:27:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009ed9d700 1 -- 172.21.15.26:0/380960152 >> v1:172.21.15.120:6789/0 conn(0x7f00a00a7620 legacy=0x7f00a0066850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009ed9d700 1 -- 172.21.15.26:0/380960152 >> v1:172.21.15.26:6789/0 conn(0x7f00a0148580 legacy=0x7f00a0066f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009ed9d700 1 -- 172.21.15.26:0/380960152 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0147e40 con 0x7f00a00b2ac0 2022-07-20T02:27:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009ed9d700 1 --2- 172.21.15.26:0/380960152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00b2ac0 0x7f00a00aff70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f00a00a88a0 tx=0x7f0090006df0).ready entity=mon.1 client_cookie=aa4e46bd48d5e583 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009dd9b700 1 -- 172.21.15.26:0/380960152 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009000d610 con 0x7f00a00b2ac0 2022-07-20T02:27:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009dd9b700 1 -- 172.21.15.26:0/380960152 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0090000e90 con 0x7f00a00b2ac0 2022-07-20T02:27:31.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.724+0000 7f009dd9b700 1 -- 172.21.15.26:0/380960152 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009000d960 con 0x7f00a00b2ac0 2022-07-20T02:27:31.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- 172.21.15.26:0/380960152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00b2ac0 msgr2=0x7f00a00aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 --2- 172.21.15.26:0/380960152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00b2ac0 0x7f00a00aff70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f00a00a88a0 tx=0x7f0090006df0).stop 2022-07-20T02:27:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- 172.21.15.26:0/380960152 shutdown_connections 2022-07-20T02:27:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 --2- 172.21.15.26:0/380960152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00b2ac0 0x7f00a00aff70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- 172.21.15.26:0/380960152 >> 172.21.15.26:0/380960152 conn(0x7f00a00b60a0 msgr2=0x7f00a00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- 172.21.15.26:0/380960152 shutdown_connections 2022-07-20T02:27:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- 172.21.15.26:0/380960152 wait complete. 2022-07-20T02:27:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 Processor -- start 2022-07-20T02:27:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.728+0000 7f00a5004700 1 -- start start 2022-07-20T02:27:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00a7620 0x7f00a00af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00a0148580 0x7f00a0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0062990 con 0x7f00a00b2ac0 2022-07-20T02:27:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0062c30 con 0x7f00a00a7620 2022-07-20T02:27:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0062ed0 con 0x7f00a0148580 2022-07-20T02:27:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36974/0 (socket says 172.21.15.26:36974) 2022-07-20T02:27:31.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 -- 172.21.15.26:0/3950029054 learned_addr learned my addr 172.21.15.26:0/3950029054 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:31.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009ed9d700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00a7620 0x7f00a00af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009f59e700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00a0148580 0x7f00a0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 -- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00a0148580 msgr2=0x7f00a0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00a0148580 0x7f00a0159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 -- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00a7620 msgr2=0x7f00a00af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00a7620 0x7f00a00af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 -- 172.21.15.26:0/3950029054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009000b180 con 0x7f00a00b2ac0 2022-07-20T02:27:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f009e59c700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f00a00618d0 tx=0x7f0094007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0094012400 con 0x7f00a00b2ac0 2022-07-20T02:27:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094012560 con 0x7f00a00b2ac0 2022-07-20T02:27:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0063170 con 0x7f00a00b2ac0 2022-07-20T02:27:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00940128c0 con 0x7f00a00b2ac0 2022-07-20T02:27:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.732+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a00adbe0 con 0x7f00a00b2ac0 2022-07-20T02:27:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a01480e0 con 0x7f00a00b2ac0 2022-07-20T02:27:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0094019030 con 0x7f00a00b2ac0 2022-07-20T02:27:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f008ffff700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 0x7f007c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f009ed9d700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 0x7f007c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f009406b180 con 0x7f00a00b2ac0 2022-07-20T02:27:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.736+0000 7f009ed9d700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 0x7f007c055f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f009001a000 tx=0x7f0090014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.744+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00940420b0 con 0x7f00a00b2ac0 2022-07-20T02:27:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:31.872+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f00a00574e0 con 0x7f00a00b2ac0 2022-07-20T02:27:32.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.592+0000 7f008ffff700 1 -- 172.21.15.26:0/3950029054 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v43) v1 ==== 125+0+0 (secure 0 0 0) 0x7f00940456d0 con 0x7f00a00b2ac0 2022-07-20T02:27:32.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 msgr2=0x7f007c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 0x7f007c055f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f009001a000 tx=0x7f0090014000).stop 2022-07-20T02:27:32.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 msgr2=0x7f00a00adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f00a00618d0 tx=0x7f0094007c50).stop 2022-07-20T02:27:32.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 shutdown_connections 2022-07-20T02:27:32.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00a00b2ac0 0x7f00a00adfb0 secure :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0x7f00a00618d0 tx=0x7f0094007c50).stop 2022-07-20T02:27:32.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f007c053aa0 0x7f007c055f60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00a0148580 0x7f00a0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 --2- 172.21.15.26:0/3950029054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a00a7620 0x7f00a00af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 >> 172.21.15.26:0/3950029054 conn(0x7f00a00b60a0 msgr2=0x7f00a0060030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:32.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 shutdown_connections 2022-07-20T02:27:32.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.596+0000 7f00a5004700 1 -- 172.21.15.26:0/3950029054 wait complete. 2022-07-20T02:27:32.609 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:27:32.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:27:32.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.748+0000 7fb1970f3700 1 Processor -- start 2022-07-20T02:27:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.748+0000 7fb1970f3700 1 -- start start 2022-07-20T02:27:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1900adf90 0x7fb1900b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900ae370 con 0x7fb1900b14b0 2022-07-20T02:27:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190062050 con 0x7fb1900614b0 2022-07-20T02:27:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900b28b0 con 0x7fb1900adf90 2022-07-20T02:27:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1900b14b0 0x7fb190060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36262/0 (socket says 172.21.15.26:36262) 2022-07-20T02:27:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 -- 172.21.15.26:0/3383980647 learned_addr learned my addr 172.21.15.26:0/3383980647 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb194e8f700 1 --2- 172.21.15.26:0/3383980647 >> v2:172.21.15.120:3300/0 conn(0x7fb1900adf90 0x7fb1900b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535381588 0 0) 0x7fb1900ae370 con 0x7fb1900b14b0 2022-07-20T02:27:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174003260 con 0x7fb1900b14b0 2022-07-20T02:27:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663507930 0 0) 0x7fb174003260 con 0x7fb1900b14b0 2022-07-20T02:27:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 >> v2:172.21.15.120:3300/0 conn(0x7fb1900adf90 msgr2=0x7fb1900b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 --2- 172.21.15.26:0/3383980647 >> v2:172.21.15.120:3300/0 conn(0x7fb1900adf90 0x7fb1900b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 >> v1:172.21.15.120:6789/0 conn(0x7fb1900614b0 legacy=0x7fb190061940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900b0610 con 0x7fb1900b14b0 2022-07-20T02:27:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fb184001680 con 0x7fb1900b14b0 2022-07-20T02:27:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 >> v1:172.21.15.26:6789/0 conn(0x7fb1900b14b0 legacy=0x7fb190060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174003750 0x7fb174003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb174004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb174004e70 0x7fb17400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900b28b0 con 0x7fb174004500 2022-07-20T02:27:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17400bd20 con 0x7fb174003750 2022-07-20T02:27:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17400bee0 con 0x7fb174004e70 2022-07-20T02:27:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb174004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb194e8f700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174003750 0x7fb174003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 -- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb174004e70 msgr2=0x7fb17400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb174004e70 0x7fb17400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:27:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 -- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174003750 msgr2=0x7fb174003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174003750 0x7fb174003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 -- 172.21.15.26:0/3383980647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb174003260 con 0x7fb174004500 2022-07-20T02:27:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb195690700 1 -- 172.21.15.26:0/3383980647 reap_dead start 2022-07-20T02:27:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb174004980 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fb184001390 tx=0x7fb190067420).ready entity=mon.0 client_cookie=b5dd139eb2f4d0b3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb184007170 con 0x7fb174004500 2022-07-20T02:27:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1840072d0 con 0x7fb174004500 2022-07-20T02:27:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb18f7fe700 1 -- 172.21.15.26:0/3383980647 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb18400c8f0 con 0x7fb174004500 2022-07-20T02:27:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 msgr2=0x7fb174004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb174004980 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fb184001390 tx=0x7fb190067420).stop 2022-07-20T02:27:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- 172.21.15.26:0/3383980647 shutdown_connections 2022-07-20T02:27:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3383980647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb174004980 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- 172.21.15.26:0/3383980647 >> 172.21.15.26:0/3383980647 conn(0x7fb1900b66e0 msgr2=0x7fb1900b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- 172.21.15.26:0/3383980647 shutdown_connections 2022-07-20T02:27:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.752+0000 7fb1970f3700 1 -- 172.21.15.26:0/3383980647 wait complete. 2022-07-20T02:27:32.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 Processor -- start 2022-07-20T02:27:32.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- start start 2022-07-20T02:27:32.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900654c0 0x7fb190151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 0x7fb19013e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb190064900 con 0x7fb174004500 2022-07-20T02:27:32.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb190066bf0 con 0x7fb1900654c0 2022-07-20T02:27:32.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb190064b30 con 0x7fb1900641e0 2022-07-20T02:27:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36992/0 (socket says 172.21.15.26:36992) 2022-07-20T02:27:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 -- 172.21.15.26:0/3745452364 learned_addr learned my addr 172.21.15.26:0/3745452364 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900654c0 0x7fb190151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb195690700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 0x7fb19013e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 -- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 msgr2=0x7fb19013e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 0x7fb19013e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 -- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900654c0 msgr2=0x7fb190151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900654c0 0x7fb190151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 -- 172.21.15.26:0/3745452364 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190066e90 con 0x7fb174004500 2022-07-20T02:27:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb195690700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 0x7fb19013e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb194e8f700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb17c00b000 tx=0x7fb17c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb17c014680 con 0x7fb174004500 2022-07-20T02:27:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c0147e0 con 0x7fb174004500 2022-07-20T02:27:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb190145170 con 0x7fb174004500 2022-07-20T02:27:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb17c014b40 con 0x7fb174004500 2022-07-20T02:27:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb190151c90 con 0x7fb174004500 2022-07-20T02:27:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.756+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb190065310 con 0x7fb174004500 2022-07-20T02:27:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.760+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb17c024070 con 0x7fb174004500 2022-07-20T02:27:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.760+0000 7fb18dffb700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 0x7fb170055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.760+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fb17c06c850 con 0x7fb174004500 2022-07-20T02:27:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.760+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 0x7fb170055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.760+0000 7fb18ffff700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 0x7fb170055f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb184019000 tx=0x7fb184015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.764+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb17c043690 con 0x7fb174004500 2022-07-20T02:27:32.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.928+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fb1900574e0 con 0x7fb174004500 2022-07-20T02:27:32.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.940+0000 7fb18dffb700 1 -- 172.21.15.26:0/3745452364 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v17) v1 ==== 187+0+0 (secure 0 0 0) 0x7fb17c046cb0 con 0x7fb174004500 2022-07-20T02:27:32.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.944+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 msgr2=0x7fb170055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.944+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 0x7fb170055f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb184019000 tx=0x7fb184015000).stop 2022-07-20T02:27:32.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.944+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 msgr2=0x7fb190144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:32.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.944+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb17c00b000 tx=0x7fb17c008040).stop 2022-07-20T02:27:32.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 shutdown_connections 2022-07-20T02:27:32.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb170053a70 0x7fb170055f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1900641e0 0x7fb19013e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1900654c0 0x7fb190151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 --2- 172.21.15.26:0/3745452364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb174004500 0x7fb190144b00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:32.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 >> 172.21.15.26:0/3745452364 conn(0x7fb1900b66e0 msgr2=0x7fb1900601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:32.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 shutdown_connections 2022-07-20T02:27:32.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:32.948+0000 7fb1970f3700 1 -- 172.21.15.26:0/3745452364 wait complete. 2022-07-20T02:27:32.957 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 7 and data pool 8 2022-07-20T02:27:32.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:27:33.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.072+0000 7ff076179700 1 Processor -- start 2022-07-20T02:27:33.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.072+0000 7ff076179700 1 -- start start 2022-07-20T02:27:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff0701474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0700b62d0 con 0x7ff0700a7fc0 2022-07-20T02:27:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07014aac0 con 0x7ff0700a8c40 2022-07-20T02:27:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07014ad60 con 0x7ff0700b5ef0 2022-07-20T02:27:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff074976700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff0700a7fc0 0x7ff0700aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36290/0 (socket says 172.21.15.26:36290) 2022-07-20T02:27:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff074976700 1 -- 172.21.15.26:0/2935278056 learned_addr learned my addr 172.21.15.26:0/2935278056 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075177700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff0701474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844448993 0 0) 0x7ff0700b62d0 con 0x7ff0700a7fc0 2022-07-20T02:27:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff054003260 con 0x7ff0700a7fc0 2022-07-20T02:27:33.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769482843 0 0) 0x7ff07014aac0 con 0x7ff0700a8c40 2022-07-20T02:27:33.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0700b62d0 con 0x7ff0700a8c40 2022-07-20T02:27:33.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061937775 0 0) 0x7ff054003260 con 0x7ff0700a7fc0 2022-07-20T02:27:33.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 msgr2=0x7ff0701474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:33.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff0701474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 >> v1:172.21.15.120:6789/0 conn(0x7ff0700a8c40 legacy=0x7ff070151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07014b000 con 0x7ff0700a7fc0 2022-07-20T02:27:33.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7ff06c001660 con 0x7ff0700a7fc0 2022-07-20T02:27:33.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 >> v1:172.21.15.26:6789/0 conn(0x7ff0700a7fc0 legacy=0x7ff0700aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff054003910 0x7ff054003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff054004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0540049f0 0x7ff054004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07014ad60 con 0x7ff0540049f0 2022-07-20T02:27:33.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05400b780 con 0x7ff054003910 2022-07-20T02:27:33.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05400b940 con 0x7ff0700b5ef0 2022-07-20T02:27:33.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0540049f0 0x7ff054004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075177700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff054003910 0x7ff054003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff074976700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff054004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 -- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 msgr2=0x7ff054004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700b5ef0 0x7ff054004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 -- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff054003910 msgr2=0x7ff054003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff054003910 0x7ff054003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 -- 172.21.15.26:0/2935278056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff054003260 con 0x7ff0540049f0 2022-07-20T02:27:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075177700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff054003910 0x7ff054003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0540049f0 0x7ff054004e70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7ff068001490 tx=0x7ff068007d80).ready entity=mon.0 client_cookie=50cbc1581ed744e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff068001780 con 0x7ff0540049f0 2022-07-20T02:27:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068006890 con 0x7ff0540049f0 2022-07-20T02:27:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff067fff700 1 -- 172.21.15.26:0/2935278056 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff068006bf0 con 0x7ff0540049f0 2022-07-20T02:27:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- 172.21.15.26:0/2935278056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0540049f0 msgr2=0x7ff054004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 --2- 172.21.15.26:0/2935278056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0540049f0 0x7ff054004e70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7ff068001490 tx=0x7ff068007d80).stop 2022-07-20T02:27:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff075978700 1 -- 172.21.15.26:0/2935278056 reap_dead start 2022-07-20T02:27:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- 172.21.15.26:0/2935278056 shutdown_connections 2022-07-20T02:27:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- 172.21.15.26:0/2935278056 >> 172.21.15.26:0/2935278056 conn(0x7ff070060310 msgr2=0x7ff0700b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- 172.21.15.26:0/2935278056 shutdown_connections 2022-07-20T02:27:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.076+0000 7ff076179700 1 -- 172.21.15.26:0/2935278056 wait complete. 2022-07-20T02:27:33.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 Processor -- start 2022-07-20T02:27:33.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 -- start start 2022-07-20T02:27:33.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 0x7ff07014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700b46e0 con 0x7ff0700a97a0 2022-07-20T02:27:33.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700a9c50 con 0x7ff0540049f0 2022-07-20T02:27:33.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.080+0000 7ff076179700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff070146240 con 0x7ff0700ac150 2022-07-20T02:27:33.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff074976700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff074976700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:58540/0 (socket says 172.21.15.26:58540) 2022-07-20T02:27:33.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff074976700 1 -- 172.21.15.26:0/4132257275 learned_addr learned my addr 172.21.15.26:0/4132257275 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:33.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37026/0 (socket says 172.21.15.26:37026) 2022-07-20T02:27:33.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075177700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 0x7ff07014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 -- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 msgr2=0x7ff0700a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 -- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 msgr2=0x7ff07014cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 0x7ff07014cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 -- 172.21.15.26:0/4132257275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff070146bc0 con 0x7ff0700a97a0 2022-07-20T02:27:33.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff074976700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:33.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075177700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 0x7ff07014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:33.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff075978700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff0700afd10 tx=0x7ff068016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff068001780 con 0x7ff0700a97a0 2022-07-20T02:27:33.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06800c9e0 con 0x7ff0700a97a0 2022-07-20T02:27:33.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff070146e60 con 0x7ff0700a97a0 2022-07-20T02:27:33.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff06800cd80 con 0x7ff0700a97a0 2022-07-20T02:27:33.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701482f0 con 0x7ff0700a97a0 2022-07-20T02:27:33.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07014b2a0 con 0x7ff0700a97a0 2022-07-20T02:27:33.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.084+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff068025030 con 0x7ff0700a97a0 2022-07-20T02:27:33.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.088+0000 7ff0667fc700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.088+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff06806cd40 con 0x7ff0700a97a0 2022-07-20T02:27:33.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.088+0000 7ff075177700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.088+0000 7ff075177700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 0x7ff058055f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff06000b000 tx=0x7ff060008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.092+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff068043b00 con 0x7ff0700a97a0 2022-07-20T02:27:33.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.220+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0700574e0 con 0x7ff0700a97a0 2022-07-20T02:27:33.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.224+0000 7ff0667fc700 1 -- 172.21.15.26:0/4132257275 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16332 (secure 0 0 0) 0x7ff068023070 con 0x7ff0700a97a0 2022-07-20T02:27:33.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:33.227 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":44,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:32.931756+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:27:30.671084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T02:27:31.927380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:27:33.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 msgr2=0x7ff058055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 0x7ff058055f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff06000b000 tx=0x7ff060008040).stop 2022-07-20T02:27:33.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 msgr2=0x7ff0700b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff0700afd10 tx=0x7ff068016000).stop 2022-07-20T02:27:33.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 shutdown_connections 2022-07-20T02:27:33.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff0700a97a0 0x7ff0700b30a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0700ac150 0x7ff0700a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 --2- 172.21.15.26:0/4132257275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0540049f0 0x7ff07014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 >> 172.21.15.26:0/4132257275 conn(0x7ff070060310 msgr2=0x7ff0701486d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 shutdown_connections 2022-07-20T02:27:33.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.228+0000 7ff076179700 1 -- 172.21.15.26:0/4132257275 wait complete. 2022-07-20T02:27:33.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:27:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 Processor -- start 2022-07-20T02:27:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- start start 2022-07-20T02:27:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 0x7f69ec143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69ec060f00 0x7f69ec13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ec1474e0 con 0x7f69ec147000 2022-07-20T02:27:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ec0b4e40 con 0x7f69ec060f00 2022-07-20T02:27:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ec05d3b0 con 0x7f69ec05cfd0 2022-07-20T02:27:33.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69ec060f00 0x7f69ec13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69ec060f00 0x7f69ec13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37034/0 (socket says 172.21.15.26:37034) 2022-07-20T02:27:33.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 -- 172.21.15.26:0/815413630 learned_addr learned my addr 172.21.15.26:0/815413630 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:33.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f21ae700 1 --2- 172.21.15.26:0/815413630 >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 0x7f69ec143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 -- 172.21.15.26:0/815413630 >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 msgr2=0x7f69ec143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:33.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 --2- 172.21.15.26:0/815413630 >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 0x7f69ec143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 -- 172.21.15.26:0/815413630 >> v1:172.21.15.120:6789/0 conn(0x7f69ec147000 legacy=0x7f69ec0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 -- 172.21.15.26:0/815413630 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec1431e0 con 0x7f69ec060f00 2022-07-20T02:27:33.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f21ae700 1 --2- 172.21.15.26:0/815413630 >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 0x7f69ec143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:33.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f29af700 1 --2- 172.21.15.26:0/815413630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec060f00 0x7f69ec13c9b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f69e40032f0 tx=0x7f69e4009290).ready entity=mon.0 client_cookie=2a79fa3a324eb1a2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f11ac700 1 -- 172.21.15.26:0/815413630 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e400b870 con 0x7f69ec060f00 2022-07-20T02:27:33.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f11ac700 1 -- 172.21.15.26:0/815413630 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e400b9d0 con 0x7f69ec060f00 2022-07-20T02:27:33.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f11ac700 1 -- 172.21.15.26:0/815413630 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e400bb40 con 0x7f69ec060f00 2022-07-20T02:27:33.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- 172.21.15.26:0/815413630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec060f00 msgr2=0x7f69ec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 --2- 172.21.15.26:0/815413630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec060f00 0x7f69ec13c9b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f69e40032f0 tx=0x7f69e4009290).stop 2022-07-20T02:27:33.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- 172.21.15.26:0/815413630 shutdown_connections 2022-07-20T02:27:33.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 --2- 172.21.15.26:0/815413630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec060f00 0x7f69ec13c9b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 --2- 172.21.15.26:0/815413630 >> v2:172.21.15.120:3300/0 conn(0x7f69ec05cfd0 0x7f69ec143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- 172.21.15.26:0/815413630 >> 172.21.15.26:0/815413630 conn(0x7f69ec0620c0 msgr2=0x7f69ec05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.364+0000 7f69f31b0700 1 -- 172.21.15.26:0/815413630 shutdown_connections 2022-07-20T02:27:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 -- 172.21.15.26:0/815413630 wait complete. 2022-07-20T02:27:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 Processor -- start 2022-07-20T02:27:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 -- start start 2022-07-20T02:27:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 0x7f69ec146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69ec0b80f0 0x7f69ec1386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec138bd0 con 0x7f69ec147000 2022-07-20T02:27:33.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec0b7480 con 0x7f69ec060f00 2022-07-20T02:27:33.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f31b0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec0b0540 con 0x7f69ec0b80f0 2022-07-20T02:27:33.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f29af700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69ec0b80f0 0x7f69ec1386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f21ae700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 0x7f69ec146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37040/0 (socket says 172.21.15.26:37040) 2022-07-20T02:27:33.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 -- 172.21.15.26:0/2233162025 learned_addr learned my addr 172.21.15.26:0/2233162025 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:33.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f21ae700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 0x7f69ec146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:56728/0 (socket says 172.21.15.26:56728) 2022-07-20T02:27:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 -- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69ec0b80f0 msgr2=0x7f69ec1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69ec0b80f0 0x7f69ec1386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 -- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 msgr2=0x7f69ec146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 0x7f69ec146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 -- 172.21.15.26:0/2233162025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4009530 con 0x7f69ec147000 2022-07-20T02:27:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.368+0000 7f69f19ad700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f69ec068660 tx=0x7f69e8007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e8007030 con 0x7f69ec147000 2022-07-20T02:27:33.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8007190 con 0x7f69ec147000 2022-07-20T02:27:33.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e8006510 con 0x7f69ec147000 2022-07-20T02:27:33.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec0b76b0 con 0x7f69ec147000 2022-07-20T02:27:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec13d1c0 con 0x7f69ec147000 2022-07-20T02:27:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f69e8010030 con 0x7f69ec147000 2022-07-20T02:27:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec0b0300 con 0x7f69ec147000 2022-07-20T02:27:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 0x7f69cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69f21ae700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 0x7f69cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f69e806bba0 con 0x7f69ec147000 2022-07-20T02:27:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.372+0000 7f69f21ae700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 0x7f69cc055f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f69dc00c000 tx=0x7f69dc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.380+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f69ec0b0300 con 0x7f69ec147000 2022-07-20T02:27:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.508+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69ec0b0300 con 0x7f69ec147000 2022-07-20T02:27:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.508+0000 7f69e37fe700 1 -- 172.21.15.26:0/2233162025 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16332 (secure 0 0 0) 0x7f69ec0b0300 con 0x7f69ec147000 2022-07-20T02:27:33.516 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:33.516 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":44,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:32.931756+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:27:30.671084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T02:27:31.927380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:27:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 msgr2=0x7f69cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 0x7f69cc055f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f69dc00c000 tx=0x7f69dc008040).stop 2022-07-20T02:27:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 msgr2=0x7f69ec1381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f69ec068660 tx=0x7f69e8007be0).stop 2022-07-20T02:27:33.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 shutdown_connections 2022-07-20T02:27:33.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69ec147000 0x7f69ec1381f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69cc053a70 0x7f69cc055f30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69ec0b80f0 0x7f69ec1386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 --2- 172.21.15.26:0/2233162025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec060f00 0x7f69ec146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 >> 172.21.15.26:0/2233162025 conn(0x7f69ec05cfd0 msgr2=0x7f69ec0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 shutdown_connections 2022-07-20T02:27:33.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.512+0000 7f69f31b0700 1 -- 172.21.15.26:0/2233162025 wait complete. 2022-07-20T02:27:33.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:27:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.656+0000 7f54aca13700 1 Processor -- start 2022-07-20T02:27:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.656+0000 7f54aca13700 1 -- start start 2022-07-20T02:27:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54aca13700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54a80a95a0 0x7f54a8060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54aca13700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a8067dc0 con 0x7f54a80b3260 2022-07-20T02:27:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54aca13700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80b3640 con 0x7f54a8095b10 2022-07-20T02:27:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54aca13700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a80615c0 con 0x7f54a80a95a0 2022-07-20T02:27:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a6d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f54a80b3260 0x7f54a80676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36324/0 (socket says 172.21.15.26:36324) 2022-07-20T02:27:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a6d9d700 1 -- 172.21.15.26:0/3323253599 learned_addr learned my addr 172.21.15.26:0/3323253599 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a659c700 1 --2- 172.21.15.26:0/3323253599 >> v2:172.21.15.120:3300/0 conn(0x7f54a80a95a0 0x7f54a8060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314089257 0 0) 0x7f54a8067dc0 con 0x7f54a80b3260 2022-07-20T02:27:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c003260 con 0x7f54a80b3260 2022-07-20T02:27:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634305258 0 0) 0x7f54a80b3640 con 0x7f54a8095b10 2022-07-20T02:27:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a659c700 1 -- 172.21.15.26:0/3323253599 >> v1:172.21.15.120:6789/0 conn(0x7f54a8095b10 legacy=0x7f54a8095fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a659c700 1 -- 172.21.15.26:0/3323253599 >> v1:172.21.15.26:6789/0 conn(0x7f54a80b3260 legacy=0x7f54a80676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a659c700 1 -- 172.21.15.26:0/3323253599 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8061860 con 0x7f54a80a95a0 2022-07-20T02:27:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a659c700 1 --2- 172.21.15.26:0/3323253599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a80a95a0 0x7f54a8060950 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f549c002f70 tx=0x7f549c006df0).ready entity=mon.1 client_cookie=f5c8dc3bd4c62b98 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549c00d610 con 0x7f54a80a95a0 2022-07-20T02:27:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c000e90 con 0x7f54a80a95a0 2022-07-20T02:27:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.660+0000 7f54a5d9b700 1 -- 172.21.15.26:0/3323253599 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549c00d960 con 0x7f54a80a95a0 2022-07-20T02:27:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- 172.21.15.26:0/3323253599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a80a95a0 msgr2=0x7f54a8060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 --2- 172.21.15.26:0/3323253599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a80a95a0 0x7f54a8060950 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f549c002f70 tx=0x7f549c006df0).stop 2022-07-20T02:27:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- 172.21.15.26:0/3323253599 shutdown_connections 2022-07-20T02:27:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 --2- 172.21.15.26:0/3323253599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a80a95a0 0x7f54a8060950 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- 172.21.15.26:0/3323253599 >> 172.21.15.26:0/3323253599 conn(0x7f54a80b67f0 msgr2=0x7f54a80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- 172.21.15.26:0/3323253599 shutdown_connections 2022-07-20T02:27:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- 172.21.15.26:0/3323253599 wait complete. 2022-07-20T02:27:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 Processor -- start 2022-07-20T02:27:33.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.664+0000 7f54aca13700 1 -- start start 2022-07-20T02:27:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 0x7f54a80af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b3260 0x7f54a80aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8143f80 con 0x7f54a80a95a0 2022-07-20T02:27:33.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8144aa0 con 0x7f54a8095b10 2022-07-20T02:27:33.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8144d40 con 0x7f54a80b3260 2022-07-20T02:27:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37052/0 (socket says 172.21.15.26:37052) 2022-07-20T02:27:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 -- 172.21.15.26:0/684002743 learned_addr learned my addr 172.21.15.26:0/684002743 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a6d9d700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 0x7f54a80af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a759e700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b3260 0x7f54a80aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 -- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b3260 msgr2=0x7f54a80aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b3260 0x7f54a80aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 -- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 msgr2=0x7f54a80af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 0x7f54a80af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 -- 172.21.15.26:0/684002743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c00b180 con 0x7f54a80a95a0 2022-07-20T02:27:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a6d9d700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 0x7f54a80af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54a659c700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f549c002f70 tx=0x7f549c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549c016070 con 0x7f54a80a95a0 2022-07-20T02:27:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c00de90 con 0x7f54a80a95a0 2022-07-20T02:27:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8144fe0 con 0x7f54a80a95a0 2022-07-20T02:27:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549c013d20 con 0x7f54a80a95a0 2022-07-20T02:27:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.668+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81537f0 con 0x7f54a80a95a0 2022-07-20T02:27:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8061b00 con 0x7f54a80a95a0 2022-07-20T02:27:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f549c00f030 con 0x7f54a80a95a0 2022-07-20T02:27:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f5497fff700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 0x7f5484055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f549c06cc90 con 0x7f54a80a95a0 2022-07-20T02:27:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f54a6d9d700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 0x7f5484055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.672+0000 7f54a6d9d700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 0x7f5484055ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f549800d000 tx=0x7f5498009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.680+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f549c043940 con 0x7f54a80a95a0 2022-07-20T02:27:33.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.840+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f54a80574e0 con 0x7f54a80a95a0 2022-07-20T02:27:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.948+0000 7f5497fff700 1 -- 172.21.15.26:0/684002743 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f549c0230b0 con 0x7f54a80a95a0 2022-07-20T02:27:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 msgr2=0x7f5484055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 0x7f5484055ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f549800d000 tx=0x7f5498009040).stop 2022-07-20T02:27:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 msgr2=0x7f54a8140570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f549c002f70 tx=0x7f549c014000).stop 2022-07-20T02:27:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 shutdown_connections 2022-07-20T02:27:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54a80a95a0 0x7f54a8140570 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f54840539e0 0x7f5484055ea0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b3260 0x7f54a80aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 --2- 172.21.15.26:0/684002743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54a8095b10 0x7f54a80af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 >> 172.21.15.26:0/684002743 conn(0x7f54a80b67f0 msgr2=0x7f54a80b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 shutdown_connections 2022-07-20T02:27:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:33.952+0000 7f54aca13700 1 -- 172.21.15.26:0/684002743 wait complete. 2022-07-20T02:27:33.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:34.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 Processor -- start 2022-07-20T02:27:34.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 -- start start 2022-07-20T02:27:34.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c3005ce30 0x7f8c300b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8c300b0ed0 0x7f8c301474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c30144740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c30061080 con 0x7f8c3005ce30 2022-07-20T02:27:34.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c300b3960 con 0x7f8c300b0ed0 2022-07-20T02:27:34.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c3673a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300b3c00 con 0x7f8c300b5270 2022-07-20T02:27:34.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c3005ce30 0x7f8c300b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c3005ce30 0x7f8c300b14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37058/0 (socket says 172.21.15.26:37058) 2022-07-20T02:27:34.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 -- 172.21.15.26:0/1372215190 learned_addr learned my addr 172.21.15.26:0/1372215190 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:34.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35f39700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c30144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c34f37700 1 --2- 172.21.15.26:0/1372215190 >> v2:172.21.15.120:3300/0 conn(0x7f8c300b0ed0 0x7f8c301474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 -- 172.21.15.26:0/1372215190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 msgr2=0x7f8c30144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:34.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c30144740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 -- 172.21.15.26:0/1372215190 >> v2:172.21.15.120:3300/0 conn(0x7f8c300b0ed0 msgr2=0x7f8c301474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.076+0000 7f8c35738700 1 --2- 172.21.15.26:0/1372215190 >> v2:172.21.15.120:3300/0 conn(0x7f8c300b0ed0 0x7f8c301474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35738700 1 -- 172.21.15.26:0/1372215190 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300b4f80 con 0x7f8c3005ce30 2022-07-20T02:27:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35738700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c3005ce30 0x7f8c300b14f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f8c20002f70 tx=0x7f8c20006eb0).ready entity=mon.0 client_cookie=986ec9b5414096de server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c27fff700 1 -- 172.21.15.26:0/1372215190 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c2000d570 con 0x7f8c3005ce30 2022-07-20T02:27:34.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c27fff700 1 -- 172.21.15.26:0/1372215190 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20000e90 con 0x7f8c3005ce30 2022-07-20T02:27:34.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c27fff700 1 -- 172.21.15.26:0/1372215190 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c2000d900 con 0x7f8c3005ce30 2022-07-20T02:27:34.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- 172.21.15.26:0/1372215190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c3005ce30 msgr2=0x7f8c300b14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c3005ce30 0x7f8c300b14f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f8c20002f70 tx=0x7f8c20006eb0).stop 2022-07-20T02:27:34.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- 172.21.15.26:0/1372215190 shutdown_connections 2022-07-20T02:27:34.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c30144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- 172.21.15.26:0/1372215190 >> v2:172.21.15.120:3300/0 conn(0x7f8c300b0ed0 0x7f8c301474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- 172.21.15.26:0/1372215190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c3005ce30 0x7f8c300b14f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- 172.21.15.26:0/1372215190 >> 172.21.15.26:0/1372215190 conn(0x7f8c300620c0 msgr2=0x7f8c3005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- 172.21.15.26:0/1372215190 shutdown_connections 2022-07-20T02:27:34.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- 172.21.15.26:0/1372215190 wait complete. 2022-07-20T02:27:34.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 Processor -- start 2022-07-20T02:27:34.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- start start 2022-07-20T02:27:34.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c300675f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c30063340 0x7f8c300637c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c301576d0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30157270 con 0x7f8c30063340 2022-07-20T02:27:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c3673a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30145470 con 0x7f8c300b5270 2022-07-20T02:27:34.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35738700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35738700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37064/0 (socket says 172.21.15.26:37064) 2022-07-20T02:27:34.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35738700 1 -- 172.21.15.26:0/2062488049 learned_addr learned my addr 172.21.15.26:0/2062488049 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:34.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c35f39700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c30063340 0x7f8c300637c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.080+0000 7f8c34f37700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c300675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 -- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 msgr2=0x7f8c300675f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c300675f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 -- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c30063340 msgr2=0x7f8c300637c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c30063340 0x7f8c300637c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 -- 172.21.15.26:0/2062488049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2000b1e0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c35738700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8c30157810 tx=0x7f8c20015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c200068b0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20011bf0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c30063cb0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c2001e460 con 0x7f8c300b0ed0 2022-07-20T02:27:34.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c30146320 con 0x7f8c300b0ed0 2022-07-20T02:27:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3005d210 con 0x7f8c300b0ed0 2022-07-20T02:27:34.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.084+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8c20025030 con 0x7f8c300b0ed0 2022-07-20T02:27:34.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.088+0000 7f8c267fc700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 0x7f8c10055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.088+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8c2006d1e0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.088+0000 7f8c34f37700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 0x7f8c10055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.088+0000 7f8c34f37700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 0x7f8c10055f30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8c2c00d000 tx=0x7f8c2c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.092+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c20043fa0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.248+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c300574e0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c267fc700 1 -- 172.21.15.26:0/2062488049 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f8c200475c0 con 0x7f8c300b0ed0 2022-07-20T02:27:34.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:34.266 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:34.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 msgr2=0x7f8c10055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 0x7f8c10055f30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8c2c00d000 tx=0x7f8c2c008040).stop 2022-07-20T02:27:34.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 msgr2=0x7f8c30067100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8c30157810 tx=0x7f8c20015000).stop 2022-07-20T02:27:34.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 shutdown_connections 2022-07-20T02:27:34.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c10053a70 0x7f8c10055f30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c30063340 0x7f8c300637c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c300b5270 0x7f8c300675f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 --2- 172.21.15.26:0/2062488049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c300b0ed0 0x7f8c30067100 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 >> 172.21.15.26:0/2062488049 conn(0x7f8c3005ce30 msgr2=0x7f8c30149160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 shutdown_connections 2022-07-20T02:27:34.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.252+0000 7f8c3673a700 1 -- 172.21.15.26:0/2062488049 wait complete. 2022-07-20T02:27:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:34.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:27:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.396+0000 7f440880e700 1 Processor -- start 2022-07-20T02:27:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f440880e700 1 -- start start 2022-07-20T02:27:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f440880e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 0x7f4404143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f440880e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44041474e0 con 0x7f440405cfd0 2022-07-20T02:27:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f440880e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44040b4e40 con 0x7f4404147000 2022-07-20T02:27:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f440880e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440405d3b0 con 0x7f4404060f00 2022-07-20T02:27:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f44037fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f440405cfd0 0x7f44040b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36352/0 (socket says 172.21.15.26:36352) 2022-07-20T02:27:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f44037fe700 1 -- 172.21.15.26:0/3248243754 learned_addr learned my addr 172.21.15.26:0/3248243754 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4403fff700 1 --2- 172.21.15.26:0/3248243754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 0x7f4404143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4403fff700 1 -- 172.21.15.26:0/3248243754 >> v1:172.21.15.120:6789/0 conn(0x7f4404147000 legacy=0x7f440413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4402ffd700 1 -- 172.21.15.26:0/3248243754 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592670752 0 0) 0x7f44041474e0 con 0x7f440405cfd0 2022-07-20T02:27:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4403fff700 1 -- 172.21.15.26:0/3248243754 >> v1:172.21.15.26:6789/0 conn(0x7f440405cfd0 legacy=0x7f44040b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4403fff700 1 -- 172.21.15.26:0/3248243754 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44041431e0 con 0x7f4404060f00 2022-07-20T02:27:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4403fff700 1 --2- 172.21.15.26:0/3248243754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 0x7f4404143bc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f43ec001110 tx=0x7f43fc0091d0).ready entity=mon.2 client_cookie=ce3984440b526bae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4402ffd700 1 -- 172.21.15.26:0/3248243754 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43fc00b630 con 0x7f4404060f00 2022-07-20T02:27:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4402ffd700 1 -- 172.21.15.26:0/3248243754 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc00b790 con 0x7f4404060f00 2022-07-20T02:27:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.400+0000 7f4402ffd700 1 -- 172.21.15.26:0/3248243754 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43fc00b900 con 0x7f4404060f00 2022-07-20T02:27:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- 172.21.15.26:0/3248243754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 msgr2=0x7f4404143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 --2- 172.21.15.26:0/3248243754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 0x7f4404143bc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f43ec001110 tx=0x7f43fc0091d0).stop 2022-07-20T02:27:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- 172.21.15.26:0/3248243754 shutdown_connections 2022-07-20T02:27:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 --2- 172.21.15.26:0/3248243754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404060f00 0x7f4404143bc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- 172.21.15.26:0/3248243754 >> 172.21.15.26:0/3248243754 conn(0x7f44040620c0 msgr2=0x7f440405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- 172.21.15.26:0/3248243754 shutdown_connections 2022-07-20T02:27:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- 172.21.15.26:0/3248243754 wait complete. 2022-07-20T02:27:34.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 Processor -- start 2022-07-20T02:27:34.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- start start 2022-07-20T02:27:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4404060f00 0x7f440413c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404155850 0x7f4404155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440413d160 con 0x7f4404147000 2022-07-20T02:27:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44040b2a20 con 0x7f4404060f00 2022-07-20T02:27:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f440880e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4404152d40 con 0x7f4404155850 2022-07-20T02:27:34.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f43fbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f43fbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37080/0 (socket says 172.21.15.26:37080) 2022-07-20T02:27:34.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f44037fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4404060f00 0x7f440413c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f43fbfff700 1 -- 172.21.15.26:0/567647954 learned_addr learned my addr 172.21.15.26:0/567647954 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:34.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.404+0000 7f4403fff700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404155850 0x7f4404155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 -- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404155850 msgr2=0x7f4404155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404155850 0x7f4404155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 -- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4404060f00 msgr2=0x7f440413c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4404060f00 0x7f440413c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 -- 172.21.15.26:0/567647954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc009470 con 0x7f4404147000 2022-07-20T02:27:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f43fbfff700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4404152ed0 tx=0x7f43ec007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43ec007030 con 0x7f4404147000 2022-07-20T02:27:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ec007190 con 0x7f4404147000 2022-07-20T02:27:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43ec006510 con 0x7f4404147000 2022-07-20T02:27:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4404140860 con 0x7f4404147000 2022-07-20T02:27:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4404153250 con 0x7f4404147000 2022-07-20T02:27:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404146270 con 0x7f4404147000 2022-07-20T02:27:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43ec010030 con 0x7f4404147000 2022-07-20T02:27:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 0x7f43e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.408+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f43ec06bdb0 con 0x7f4404147000 2022-07-20T02:27:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.412+0000 7f44037fe700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 0x7f43e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.412+0000 7f44037fe700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 0x7f43e8055f60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f43f400d000 tx=0x7f43f4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.416+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43ec042b10 con 0x7f4404147000 2022-07-20T02:27:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.540+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4404143480 con 0x7f4404147000 2022-07-20T02:27:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.540+0000 7f44017fa700 1 -- 172.21.15.26:0/567647954 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f43ec046130 con 0x7f4404147000 2022-07-20T02:27:34.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:34.549 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:33.936082+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:27:30.671084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T02:27:31.927380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:27:34.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 msgr2=0x7f43e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 0x7f43e8055f60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f43f400d000 tx=0x7f43f4009040).stop 2022-07-20T02:27:34.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 msgr2=0x7f440413cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4404152ed0 tx=0x7f43ec007bd0).stop 2022-07-20T02:27:34.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 shutdown_connections 2022-07-20T02:27:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4404147000 0x7f440413cc70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43e8053aa0 0x7f43e8055f60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4404155850 0x7f4404155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 --2- 172.21.15.26:0/567647954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4404060f00 0x7f440413c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 >> 172.21.15.26:0/567647954 conn(0x7f440405cfd0 msgr2=0x7f4404139cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 shutdown_connections 2022-07-20T02:27:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.544+0000 7f440880e700 1 -- 172.21.15.26:0/567647954 wait complete. 2022-07-20T02:27:34.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:27:34.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 Processor -- start 2022-07-20T02:27:34.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- start start 2022-07-20T02:27:34.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f99080b9c80 0x7f99081431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9908062710 0x7f99080b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080ba060 con 0x7f99080b6fe0 2022-07-20T02:27:34.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990812bee0 con 0x7f99080b9c80 2022-07-20T02:27:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9908061f80 con 0x7f9908062710 2022-07-20T02:27:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f99080b9c80 0x7f99081431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f99080b9c80 0x7f99081431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37086/0 (socket says 172.21.15.26:37086) 2022-07-20T02:27:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 -- 172.21.15.26:0/3292949975 learned_addr learned my addr 172.21.15.26:0/3292949975 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990c824700 1 --2- 172.21.15.26:0/3292949975 >> v2:172.21.15.120:3300/0 conn(0x7f9908062710 0x7f99080b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 -- 172.21.15.26:0/3292949975 >> v2:172.21.15.120:3300/0 conn(0x7f9908062710 msgr2=0x7f99080b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 --2- 172.21.15.26:0/3292949975 >> v2:172.21.15.120:3300/0 conn(0x7f9908062710 0x7f99080b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 -- 172.21.15.26:0/3292949975 >> v1:172.21.15.26:6789/0 conn(0x7f99080b6fe0 legacy=0x7f99081403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 -- 172.21.15.26:0/3292949975 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908062220 con 0x7f99080b9c80 2022-07-20T02:27:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990d025700 1 --2- 172.21.15.26:0/3292949975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99080b9c80 0x7f99081431a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904006eb0).ready entity=mon.0 client_cookie=24eae0105c8d9dca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f98fbfff700 1 -- 172.21.15.26:0/3292949975 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400d5c0 con 0x7f99080b9c80 2022-07-20T02:27:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f98fbfff700 1 -- 172.21.15.26:0/3292949975 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904000e90 con 0x7f99080b9c80 2022-07-20T02:27:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f98fbfff700 1 -- 172.21.15.26:0/3292949975 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400d950 con 0x7f99080b9c80 2022-07-20T02:27:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- 172.21.15.26:0/3292949975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99080b9c80 msgr2=0x7f99081431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 --2- 172.21.15.26:0/3292949975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99080b9c80 0x7f99081431a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904006eb0).stop 2022-07-20T02:27:34.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- 172.21.15.26:0/3292949975 shutdown_connections 2022-07-20T02:27:34.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 --2- 172.21.15.26:0/3292949975 >> v2:172.21.15.120:3300/0 conn(0x7f9908062710 0x7f99080b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 --2- 172.21.15.26:0/3292949975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f99080b9c80 0x7f99081431a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.684+0000 7f990f289700 1 -- 172.21.15.26:0/3292949975 >> 172.21.15.26:0/3292949975 conn(0x7f9908066f70 msgr2=0x7f990805faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.688+0000 7f990f289700 1 -- 172.21.15.26:0/3292949975 shutdown_connections 2022-07-20T02:27:34.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.688+0000 7f990f289700 1 -- 172.21.15.26:0/3292949975 wait complete. 2022-07-20T02:27:34.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.688+0000 7f990f289700 1 Processor -- start 2022-07-20T02:27:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.688+0000 7f990f289700 1 -- start start 2022-07-20T02:27:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 0x7f9908155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99080b9c80 0x7f9908156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080b2b60 con 0x7f9908062710 2022-07-20T02:27:34.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080b2620 con 0x7f99080b9c80 2022-07-20T02:27:34.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080b28c0 con 0x7f99080b6fe0 2022-07-20T02:27:34.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37092/0 (socket says 172.21.15.26:37092) 2022-07-20T02:27:34.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 -- 172.21.15.26:0/3264911887 learned_addr learned my addr 172.21.15.26:0/3264911887 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:34.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990c824700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 0x7f9908155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d826700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99080b9c80 0x7f9908156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 -- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 msgr2=0x7f9908155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 0x7f9908155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 -- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99080b9c80 msgr2=0x7f9908156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99080b9c80 0x7f9908156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 -- 172.21.15.26:0/3264911887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990400b1e0 con 0x7f9908062710 2022-07-20T02:27:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990c824700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 0x7f9908155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990d025700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9904011990 con 0x7f9908062710 2022-07-20T02:27:34.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904011af0 con 0x7f9908062710 2022-07-20T02:27:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99080b7cc0 con 0x7f9908062710 2022-07-20T02:27:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9904011e50 con 0x7f9908062710 2022-07-20T02:27:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9908156510 con 0x7f9908062710 2022-07-20T02:27:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.692+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99080b6650 con 0x7f9908062710 2022-07-20T02:27:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.696+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9904025030 con 0x7f9908062710 2022-07-20T02:27:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.696+0000 7f98fa7fc700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 0x7f98e8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.696+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f990406d3a0 con 0x7f9908062710 2022-07-20T02:27:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.696+0000 7f990c824700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 0x7f98e8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.696+0000 7f990c824700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 0x7f98e8055ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f98fc00b000 tx=0x7f98fc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.700+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9904044050 con 0x7f9908062710 2022-07-20T02:27:34.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.868+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f99080574e0 con 0x7f9908062710 2022-07-20T02:27:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f98fa7fc700 1 -- 172.21.15.26:0/3264911887 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f99040230e0 con 0x7f9908062710 2022-07-20T02:27:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 msgr2=0x7f98e8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 0x7f98e8055ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f98fc00b000 tx=0x7f98fc008040).stop 2022-07-20T02:27:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 msgr2=0x7f99080b77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904015000).stop 2022-07-20T02:27:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 shutdown_connections 2022-07-20T02:27:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f98e80539e0 0x7f98e8055ea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99080b9c80 0x7f9908156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99080b6fe0 0x7f9908155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 --2- 172.21.15.26:0/3264911887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9908062710 0x7f99080b77d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 >> 172.21.15.26:0/3264911887 conn(0x7f9908066f70 msgr2=0x7f99080b8540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 shutdown_connections 2022-07-20T02:27:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:34.876+0000 7f990f289700 1 -- 172.21.15.26:0/3264911887 wait complete. 2022-07-20T02:27:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:27:34.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:27:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 Processor -- start 2022-07-20T02:27:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- start start 2022-07-20T02:27:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7e24062710 0x7f7e241431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240ba060 con 0x7f7e240b9c80 2022-07-20T02:27:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24061f80 con 0x7f7e240b5b70 2022-07-20T02:27:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2412bee0 con 0x7f7e24062710 2022-07-20T02:27:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7e24062710 0x7f7e241431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7e24062710 0x7f7e241431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37102/0 (socket says 172.21.15.26:37102) 2022-07-20T02:27:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 -- 172.21.15.26:0/2480688412 learned_addr learned my addr 172.21.15.26:0/2480688412 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 -- 172.21.15.26:0/2480688412 >> v1:172.21.15.120:6789/0 conn(0x7f7e240b5b70 legacy=0x7f7e241403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 -- 172.21.15.26:0/2480688412 >> v1:172.21.15.26:6789/0 conn(0x7f7e240b9c80 legacy=0x7f7e240b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 -- 172.21.15.26:0/2480688412 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e24062220 con 0x7f7e24062710 2022-07-20T02:27:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e1bfff700 1 -- 172.21.15.26:0/2480688412 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818284257 0 0) 0x7f7e240ba060 con 0x7f7e240b9c80 2022-07-20T02:27:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e28da8700 1 --2- 172.21.15.26:0/2480688412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e24062710 0x7f7e241431a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f7e14002f70 tx=0x7f7e14006eb0).ready entity=mon.0 client_cookie=8ae8ff64620c9a95 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e1bfff700 1 -- 172.21.15.26:0/2480688412 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1400d660 con 0x7f7e24062710 2022-07-20T02:27:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e1bfff700 1 -- 172.21.15.26:0/2480688412 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14000e90 con 0x7f7e24062710 2022-07-20T02:27:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e1bfff700 1 -- 172.21.15.26:0/2480688412 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1400d9b0 con 0x7f7e24062710 2022-07-20T02:27:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- 172.21.15.26:0/2480688412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e24062710 msgr2=0x7f7e241431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/2480688412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e24062710 0x7f7e241431a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f7e14002f70 tx=0x7f7e14006eb0).stop 2022-07-20T02:27:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- 172.21.15.26:0/2480688412 shutdown_connections 2022-07-20T02:27:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/2480688412 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e24062710 0x7f7e241431a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- 172.21.15.26:0/2480688412 >> 172.21.15.26:0/2480688412 conn(0x7f7e24066f70 msgr2=0x7f7e2405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- 172.21.15.26:0/2480688412 shutdown_connections 2022-07-20T02:27:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.048+0000 7f7e2b80d700 1 -- 172.21.15.26:0/2480688412 wait complete. 2022-07-20T02:27:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 Processor -- start 2022-07-20T02:27:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- start start 2022-07-20T02:27:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e24062710 0x7f7e241448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e240b5b70 0x7f7e24146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24147640 con 0x7f7e240b9c80 2022-07-20T02:27:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e241453e0 con 0x7f7e24062710 2022-07-20T02:27:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240b3580 con 0x7f7e240b5b70 2022-07-20T02:27:35.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37110/0 (socket says 172.21.15.26:37110) 2022-07-20T02:27:35.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 -- 172.21.15.26:0/1166769402 learned_addr learned my addr 172.21.15.26:0/1166769402 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:35.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e28da8700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e240b5b70 0x7f7e24146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e295a9700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e24062710 0x7f7e241448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 -- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e240b5b70 msgr2=0x7f7e24146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e240b5b70 0x7f7e24146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 -- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e24062710 msgr2=0x7f7e241448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e24062710 0x7f7e241448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 -- 172.21.15.26:0/1166769402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1400b1e0 con 0x7f7e240b9c80 2022-07-20T02:27:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e29daa700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7e240676f0 tx=0x7f7e1c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1c00f470 con 0x7f7e240b9c80 2022-07-20T02:27:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c00f5d0 con 0x7f7e240b9c80 2022-07-20T02:27:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1c00f930 con 0x7f7e240b9c80 2022-07-20T02:27:35.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240b3820 con 0x7f7e240b9c80 2022-07-20T02:27:35.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.052+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e241450d0 con 0x7f7e240b9c80 2022-07-20T02:27:35.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e24155980 con 0x7f7e240b9c80 2022-07-20T02:27:35.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7e1c019030 con 0x7f7e240b9c80 2022-07-20T02:27:35.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e1a7fc700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 0x7f7e0c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7e1c06b630 con 0x7f7e240b9c80 2022-07-20T02:27:35.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e295a9700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 0x7f7e0c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.056+0000 7f7e295a9700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 0x7f7e0c055e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7e2000d000 tx=0x7f7e20007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.064+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7e1c0423f0 con 0x7f7e240b9c80 2022-07-20T02:27:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.244+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7e240b6650 con 0x7f7e240b9c80 2022-07-20T02:27:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.252+0000 7f7e1a7fc700 1 -- 172.21.15.26:0/1166769402 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7e1c045a10 con 0x7f7e240b9c80 2022-07-20T02:27:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 msgr2=0x7f7e0c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 0x7f7e0c055e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7e2000d000 tx=0x7f7e20007a60).stop 2022-07-20T02:27:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 msgr2=0x7f7e24146fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7e240676f0 tx=0x7f7e1c007f40).stop 2022-07-20T02:27:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 shutdown_connections 2022-07-20T02:27:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7e240b9c80 0x7f7e24146fc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7e0c053950 0x7f7e0c055e10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7e240b5b70 0x7f7e24146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 --2- 172.21.15.26:0/1166769402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e24062710 0x7f7e241448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 >> 172.21.15.26:0/1166769402 conn(0x7f7e24066f70 msgr2=0x7f7e24142500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:35.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 shutdown_connections 2022-07-20T02:27:35.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.256+0000 7f7e2b80d700 1 -- 172.21.15.26:0/1166769402 wait complete. 2022-07-20T02:27:35.276 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:27:35.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 Processor -- start 2022-07-20T02:27:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- start start 2022-07-20T02:27:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdbd405cfd0 0x7fdbd4143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd41474e0 con 0x7fdbd4147000 2022-07-20T02:27:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40b4e40 con 0x7fdbd4060f00 2022-07-20T02:27:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd405d3b0 con 0x7fdbd405cfd0 2022-07-20T02:27:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd9475700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdbd405cfd0 0x7fdbd4143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd8c74700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdbd4147000 0x7fdbd40b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36396/0 (socket says 172.21.15.26:36396) 2022-07-20T02:27:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd8c74700 1 -- 172.21.15.26:0/2216426619 learned_addr learned my addr 172.21.15.26:0/2216426619 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd9475700 1 -- 172.21.15.26:0/2216426619 >> v1:172.21.15.120:6789/0 conn(0x7fdbd4060f00 legacy=0x7fdbd413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd9475700 1 -- 172.21.15.26:0/2216426619 >> v1:172.21.15.26:6789/0 conn(0x7fdbd4147000 legacy=0x7fdbd40b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbc3fff700 1 -- 172.21.15.26:0/2216426619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495269202 0 0) 0x7fdbd41474e0 con 0x7fdbd4147000 2022-07-20T02:27:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd9475700 1 -- 172.21.15.26:0/2216426619 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd41431e0 con 0x7fdbd405cfd0 2022-07-20T02:27:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbd9475700 1 --2- 172.21.15.26:0/2216426619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd405cfd0 0x7fdbd4143bc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fdbd4068700 tx=0x7fdbd0006df0).ready entity=mon.0 client_cookie=93313e77bf35d937 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbc3fff700 1 -- 172.21.15.26:0/2216426619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd000d5c0 con 0x7fdbd405cfd0 2022-07-20T02:27:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbc3fff700 1 -- 172.21.15.26:0/2216426619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0000e90 con 0x7fdbd405cfd0 2022-07-20T02:27:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbc3fff700 1 -- 172.21.15.26:0/2216426619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd000d910 con 0x7fdbd405cfd0 2022-07-20T02:27:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- 172.21.15.26:0/2216426619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd405cfd0 msgr2=0x7fdbd4143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 --2- 172.21.15.26:0/2216426619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd405cfd0 0x7fdbd4143bc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fdbd4068700 tx=0x7fdbd0006df0).stop 2022-07-20T02:27:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- 172.21.15.26:0/2216426619 shutdown_connections 2022-07-20T02:27:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 --2- 172.21.15.26:0/2216426619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd405cfd0 0x7fdbd4143bc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- 172.21.15.26:0/2216426619 >> 172.21.15.26:0/2216426619 conn(0x7fdbd40620c0 msgr2=0x7fdbd405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- 172.21.15.26:0/2216426619 shutdown_connections 2022-07-20T02:27:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.388+0000 7fdbda477700 1 -- 172.21.15.26:0/2216426619 wait complete. 2022-07-20T02:27:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 Processor -- start 2022-07-20T02:27:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- start start 2022-07-20T02:27:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbd4060f00 0x7fdbd41421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbd4147000 0x7fdbd40b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd4152790 con 0x7fdbd4146020 2022-07-20T02:27:35.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40b7520 con 0x7fdbd4147000 2022-07-20T02:27:35.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40b77c0 con 0x7fdbd4060f00 2022-07-20T02:27:35.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37122/0 (socket says 172.21.15.26:37122) 2022-07-20T02:27:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 -- 172.21.15.26:0/2276187738 learned_addr learned my addr 172.21.15.26:0/2276187738 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9475700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbd4060f00 0x7fdbd41421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd8c74700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbd4147000 0x7fdbd40b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 -- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbd4060f00 msgr2=0x7fdbd41421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbd4060f00 0x7fdbd41421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 -- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbd4147000 msgr2=0x7fdbd40b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbd4147000 0x7fdbd40b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 -- 172.21.15.26:0/2276187738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd000b180 con 0x7fdbd4146020 2022-07-20T02:27:35.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbd9c76700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fdbcc002320 tx=0x7fdbcc0023f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbcc006aa0 con 0x7fdbd4146020 2022-07-20T02:27:35.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc006c00 con 0x7fdbd4146020 2022-07-20T02:27:35.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd4156310 con 0x7fdbd4146020 2022-07-20T02:27:35.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbcc007030 con 0x7fdbd4146020 2022-07-20T02:27:35.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.392+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd41467d0 con 0x7fdbd4146020 2022-07-20T02:27:35.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdbcc010030 con 0x7fdbd4146020 2022-07-20T02:27:35.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd40b7980 con 0x7fdbd4146020 2022-07-20T02:27:35.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbc27fc700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 0x7fdbb4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:35.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdbcc06bfe0 con 0x7fdbd4146020 2022-07-20T02:27:35.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbd9475700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 0x7fdbb4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:35.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.396+0000 7fdbd9475700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 0x7fdbb4055ea0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fdbd0018000 tx=0x7fdbd0012000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:35.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.400+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdbcc042da0 con 0x7fdbd4146020 2022-07-20T02:27:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.560+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbd41564a0 con 0x7fdbd4146020 2022-07-20T02:27:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.560+0000 7fdbc27fc700 1 -- 172.21.15.26:0/2276187738 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7fdbcc0463c0 con 0x7fdbd4146020 2022-07-20T02:27:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:35.565 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 msgr2=0x7fdbb4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 0x7fdbb4055ea0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fdbd0018000 tx=0x7fdbd0012000).stop 2022-07-20T02:27:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 msgr2=0x7fdbd41522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fdbcc002320 tx=0x7fdbcc0023f0).stop 2022-07-20T02:27:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 shutdown_connections 2022-07-20T02:27:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbd4146020 0x7fdbd41522a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbb40539e0 0x7fdbb4055ea0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbd4147000 0x7fdbd40b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 --2- 172.21.15.26:0/2276187738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbd4060f00 0x7fdbd41421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 >> 172.21.15.26:0/2276187738 conn(0x7fdbd405cfd0 msgr2=0x7fdbd40b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 shutdown_connections 2022-07-20T02:27:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:35.564+0000 7fdbda477700 1 -- 172.21.15.26:0/2276187738 wait complete. 2022-07-20T02:27:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:35.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:27:32.932112+0000', 'modified': '2022-07-20T02:27:33.936408+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'c', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3286368975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3286368975}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3286368975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:27:35.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:27:35.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:27:36.030 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":3.073938557,"mdsmap_epoch":18,"osdmap_epoch":45,"osdmap_epoch_barrier":44,"uptime":8.3019439080000001} 2022-07-20T02:27:36.031 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 45, "osdmap_epoch_barrier": 44, "rank_uptime": 3.073938557, "state": "up:active", "uptime": 8.301943908, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:27:36.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:27:36.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 Processor -- start 2022-07-20T02:27:36.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 -- start start 2022-07-20T02:27:36.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc0d81432e0 0x7fc0d812c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d80b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d8061e00 con 0x7fc0d8061a20 2022-07-20T02:27:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d812cc40 con 0x7fc0d81432e0 2022-07-20T02:27:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8147040 con 0x7fc0d81468c0 2022-07-20T02:27:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc0d81432e0 0x7fc0d812c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d77fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc0d8061a20 0x7fc0d812c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36408/0 (socket says 172.21.15.26:36408) 2022-07-20T02:27:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d77fe700 1 -- 172.21.15.26:0/2071243848 learned_addr learned my addr 172.21.15.26:0/2071243848 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d7fff700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d80b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d67fc700 1 -- 172.21.15.26:0/2071243848 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514086183 0 0) 0x7fc0d8061e00 con 0x7fc0d8061a20 2022-07-20T02:27:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 -- 172.21.15.26:0/2071243848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 msgr2=0x7fc0d80b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d80b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 -- 172.21.15.26:0/2071243848 >> v1:172.21.15.26:6789/0 conn(0x7fc0d8061a20 legacy=0x7fc0d812c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 -- 172.21.15.26:0/2071243848 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d81472e0 con 0x7fc0d81432e0 2022-07-20T02:27:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d6ffd700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d81432e0 0x7fc0d812c750 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc0c0002f70 tx=0x7fc0c0006df0).ready entity=mon.0 client_cookie=c12882809080d144 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d67fc700 1 -- 172.21.15.26:0/2071243848 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c000d570 con 0x7fc0d81432e0 2022-07-20T02:27:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d67fc700 1 -- 172.21.15.26:0/2071243848 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c0000e90 con 0x7fc0d81432e0 2022-07-20T02:27:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0d67fc700 1 -- 172.21.15.26:0/2071243848 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c000d8c0 con 0x7fc0d81432e0 2022-07-20T02:27:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 -- 172.21.15.26:0/2071243848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d81432e0 msgr2=0x7fc0d812c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.144+0000 7fc0dde52700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d81432e0 0x7fc0d812c750 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc0c0002f70 tx=0x7fc0c0006df0).stop 2022-07-20T02:27:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 -- 172.21.15.26:0/2071243848 shutdown_connections 2022-07-20T02:27:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d80b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 --2- 172.21.15.26:0/2071243848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d81432e0 0x7fc0d812c750 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 -- 172.21.15.26:0/2071243848 >> 172.21.15.26:0/2071243848 conn(0x7fc0d805faf0 msgr2=0x7fc0d8066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 -- 172.21.15.26:0/2071243848 shutdown_connections 2022-07-20T02:27:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 -- 172.21.15.26:0/2071243848 wait complete. 2022-07-20T02:27:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 Processor -- start 2022-07-20T02:27:36.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.148+0000 7fc0dde52700 1 -- start start 2022-07-20T02:27:36.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0d81432e0 0x7fc0d812d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d812dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80b84b0 con 0x7fc0d8061a20 2022-07-20T02:27:36.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80b96f0 con 0x7fc0d81432e0 2022-07-20T02:27:36.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80b9990 con 0x7fc0d81468c0 2022-07-20T02:27:36.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37132/0 (socket says 172.21.15.26:37132) 2022-07-20T02:27:36.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 -- 172.21.15.26:0/3915470808 learned_addr learned my addr 172.21.15.26:0/3915470808 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:36.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d7fff700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d812dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d6ffd700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0d81432e0 0x7fc0d812d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 -- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 msgr2=0x7fc0d812dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d812dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 -- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0d81432e0 msgr2=0x7fc0d812d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0d81432e0 0x7fc0d812d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 -- 172.21.15.26:0/3915470808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c000b180 con 0x7fc0d8061a20 2022-07-20T02:27:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d77fe700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc0c8001360 tx=0x7fc0c8001490).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c8005ba0 con 0x7fc0d8061a20 2022-07-20T02:27:36.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8005d00 con 0x7fc0d8061a20 2022-07-20T02:27:36.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d80b0840 con 0x7fc0d8061a20 2022-07-20T02:27:36.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c80072c0 con 0x7fc0d8061a20 2022-07-20T02:27:36.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d80b8010 con 0x7fc0d8061a20 2022-07-20T02:27:36.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.152+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d80b9b50 con 0x7fc0d8061a20 2022-07-20T02:27:36.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.156+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc0c801a060 con 0x7fc0d8061a20 2022-07-20T02:27:36.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.156+0000 7fc0d4ff9700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 0x7fc0b0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.156+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc0c806bba0 con 0x7fc0d8061a20 2022-07-20T02:27:36.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.156+0000 7fc0d6ffd700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 0x7fc0b0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.156+0000 7fc0d6ffd700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 0x7fc0b0055f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc0c001c000 tx=0x7fc0c0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.160+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0c8042850 con 0x7fc0d8061a20 2022-07-20T02:27:36.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.284+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d8147580 con 0x7fc0d8061a20 2022-07-20T02:27:36.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.288+0000 7fc0d4ff9700 1 -- 172.21.15.26:0/3915470808 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16319 (secure 0 0 0) 0x7fc0c8018030 con 0x7fc0d8061a20 2022-07-20T02:27:36.291 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:36.292 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:33.936082+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:27:30.671084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T02:27:31.927380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:27:36.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.288+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 msgr2=0x7fc0b0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.288+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 0x7fc0b0055f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc0c001c000 tx=0x7fc0c0015000).stop 2022-07-20T02:27:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.288+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 msgr2=0x7fc0d80b8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.288+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc0c8001360 tx=0x7fc0c8001490).stop 2022-07-20T02:27:36.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 shutdown_connections 2022-07-20T02:27:36.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc0b0053aa0 0x7fc0b0055f60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0d81468c0 0x7fc0d812dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0d81432e0 0x7fc0d812d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 --2- 172.21.15.26:0/3915470808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8061a20 0x7fc0d80b8ed0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 >> 172.21.15.26:0/3915470808 conn(0x7fc0d805faf0 msgr2=0x7fc0d805cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 shutdown_connections 2022-07-20T02:27:36.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.292+0000 7fc0dde52700 1 -- 172.21.15.26:0/3915470808 wait complete. 2022-07-20T02:27:36.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:27:30.671084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:27:31.927380+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:27:36.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:27:36.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.412+0000 7fef7d5d0700 1 Processor -- start 2022-07-20T02:27:36.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.412+0000 7fef7d5d0700 1 -- start start 2022-07-20T02:27:36.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 0x7fef7814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef780aa1e0 con 0x7fef780b2ab0 2022-07-20T02:27:36.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef780ae110 con 0x7fef780add30 2022-07-20T02:27:36.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef780a9f40 con 0x7fef780b06e0 2022-07-20T02:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef777fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fef780b2ab0 0x7fef78061060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36420/0 (socket says 172.21.15.26:36420) 2022-07-20T02:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef777fe700 1 -- 172.21.15.26:0/2045449438 learned_addr learned my addr 172.21.15.26:0/2045449438 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7cdcf700 1 --2- 172.21.15.26:0/2045449438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 0x7fef7814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098256042 0 0) 0x7fef780aa1e0 con 0x7fef780b2ab0 2022-07-20T02:27:36.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef64003260 con 0x7fef780b2ab0 2022-07-20T02:27:36.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699721925 0 0) 0x7fef64003260 con 0x7fef780b2ab0 2022-07-20T02:27:36.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7cdcf700 1 -- 172.21.15.26:0/2045449438 >> v1:172.21.15.120:6789/0 conn(0x7fef780add30 legacy=0x7fef78060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7cdcf700 1 -- 172.21.15.26:0/2045449438 >> v1:172.21.15.26:6789/0 conn(0x7fef780b2ab0 legacy=0x7fef78061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7cdcf700 1 -- 172.21.15.26:0/2045449438 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef780b1110 con 0x7fef780b06e0 2022-07-20T02:27:36.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7cdcf700 1 --2- 172.21.15.26:0/2045449438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 0x7fef7814a9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fef68001800 tx=0x7fef700091d0).ready entity=mon.2 client_cookie=d0b18666ab1ab3af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef68001660 con 0x7fef780b06e0 2022-07-20T02:27:36.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7000b580 con 0x7fef780b06e0 2022-07-20T02:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef76ffd700 1 -- 172.21.15.26:0/2045449438 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7000b6f0 con 0x7fef780b06e0 2022-07-20T02:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- 172.21.15.26:0/2045449438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 msgr2=0x7fef7814a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/2045449438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 0x7fef7814a9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fef68001800 tx=0x7fef700091d0).stop 2022-07-20T02:27:36.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- 172.21.15.26:0/2045449438 shutdown_connections 2022-07-20T02:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/2045449438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b06e0 0x7fef7814a9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- 172.21.15.26:0/2045449438 >> 172.21.15.26:0/2045449438 conn(0x7fef780b3050 msgr2=0x7fef780b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- 172.21.15.26:0/2045449438 shutdown_connections 2022-07-20T02:27:36.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.416+0000 7fef7d5d0700 1 -- 172.21.15.26:0/2045449438 wait complete. 2022-07-20T02:27:36.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 Processor -- start 2022-07-20T02:27:36.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- start start 2022-07-20T02:27:36.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 0x7fef78064ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b2ab0 0x7fef7813c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef78065860 con 0x7fef780add30 2022-07-20T02:27:36.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780ab920 con 0x7fef780b06e0 2022-07-20T02:27:36.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef78063fd0 con 0x7fef780b2ab0 2022-07-20T02:27:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37146/0 (socket says 172.21.15.26:37146) 2022-07-20T02:27:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 -- 172.21.15.26:0/691449142 learned_addr learned my addr 172.21.15.26:0/691449142 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7cdcf700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b2ab0 0x7fef7813c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef777fe700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 0x7fef78064ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 -- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b2ab0 msgr2=0x7fef7813c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b2ab0 0x7fef7813c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 -- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 msgr2=0x7fef78064ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 0x7fef78064ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 -- 172.21.15.26:0/691449142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70009470 con 0x7fef780add30 2022-07-20T02:27:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef777fe700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 0x7fef78064ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef77fff700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fef780aab70 tx=0x7fef6c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c007240 con 0x7fef780add30 2022-07-20T02:27:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c0073a0 con 0x7fef780add30 2022-07-20T02:27:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7805ff20 con 0x7fef780add30 2022-07-20T02:27:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c006700 con 0x7fef780add30 2022-07-20T02:27:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.420+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef78065a50 con 0x7fef780add30 2022-07-20T02:27:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef78064110 con 0x7fef780add30 2022-07-20T02:27:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef6c010030 con 0x7fef780add30 2022-07-20T02:27:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef757fa700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fef6c06be30 con 0x7fef780add30 2022-07-20T02:27:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef777fe700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.424+0000 7fef777fe700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 0x7fef58055ea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fef6800b000 tx=0x7fef68000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.432+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef6c042bf0 con 0x7fef780add30 2022-07-20T02:27:36.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fef780b13b0 con 0x7fef780add30 2022-07-20T02:27:36.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef757fa700 1 -- 172.21.15.26:0/691449142 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v45) v1 ==== 110+0+10 (secure 0 0 0) 0x7fef6c046210 con 0x7fef780add30 2022-07-20T02:27:36.572 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:27:36.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 msgr2=0x7fef58055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 0x7fef58055ea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fef6800b000 tx=0x7fef68000bc0).stop 2022-07-20T02:27:36.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 msgr2=0x7fef7805cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fef780aab70 tx=0x7fef6c007c00).stop 2022-07-20T02:27:36.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 shutdown_connections 2022-07-20T02:27:36.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef780b2ab0 0x7fef7813c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef780b06e0 0x7fef78064ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 --2- 172.21.15.26:0/691449142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef780add30 0x7fef7805cde0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 >> 172.21.15.26:0/691449142 conn(0x7fef780b3050 msgr2=0x7fef7805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 shutdown_connections 2022-07-20T02:27:36.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.560+0000 7fef7d5d0700 1 -- 172.21.15.26:0/691449142 wait complete. 2022-07-20T02:27:36.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:27:36.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 Processor -- start 2022-07-20T02:27:36.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 -- start start 2022-07-20T02:27:36.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85240618a0 0x7f8524147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85240b22d0 0x7f8524143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8524143d60 0x7f85240b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8524061c80 con 0x7f8524143d60 2022-07-20T02:27:36.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85241441f0 con 0x7f85240b22d0 2022-07-20T02:27:36.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8523fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85240b2810 con 0x7f85240618a0 2022-07-20T02:27:36.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f85237fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8524143d60 0x7f85240b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f85237fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8524143d60 0x7f85240b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37154/0 (socket says 172.21.15.26:37154) 2022-07-20T02:27:36.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f85237fe700 1 -- 172.21.15.26:0/3119143481 learned_addr learned my addr 172.21.15.26:0/3119143481 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:36.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f851bfff700 1 --2- 172.21.15.26:0/3119143481 >> v2:172.21.15.120:3300/0 conn(0x7f85240b22d0 0x7f8524143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.720+0000 7f8522ffd700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85240618a0 0x7f8524147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 -- 172.21.15.26:0/3119143481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85240618a0 msgr2=0x7f8524147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85240618a0 0x7f8524147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 -- 172.21.15.26:0/3119143481 >> v2:172.21.15.120:3300/0 conn(0x7f85240b22d0 msgr2=0x7f8524143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 --2- 172.21.15.26:0/3119143481 >> v2:172.21.15.120:3300/0 conn(0x7f85240b22d0 0x7f8524143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 -- 172.21.15.26:0/3119143481 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240b2ab0 con 0x7f8524143d60 2022-07-20T02:27:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85237fe700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8524143d60 0x7f85240b35b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f851c0032f0 tx=0x7f851c009290).ready entity=mon.0 client_cookie=f63cdafe51ab57c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85227fc700 1 -- 172.21.15.26:0/3119143481 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f851c00b680 con 0x7f8524143d60 2022-07-20T02:27:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85227fc700 1 -- 172.21.15.26:0/3119143481 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c00b7e0 con 0x7f8524143d60 2022-07-20T02:27:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f85227fc700 1 -- 172.21.15.26:0/3119143481 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f851c00b950 con 0x7f8524143d60 2022-07-20T02:27:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- 172.21.15.26:0/3119143481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8524143d60 msgr2=0x7f85240b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8524143d60 0x7f85240b35b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f851c0032f0 tx=0x7f851c009290).stop 2022-07-20T02:27:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- 172.21.15.26:0/3119143481 shutdown_connections 2022-07-20T02:27:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8524143d60 0x7f85240b35b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 --2- 172.21.15.26:0/3119143481 >> v2:172.21.15.120:3300/0 conn(0x7f85240b22d0 0x7f8524143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 --2- 172.21.15.26:0/3119143481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85240618a0 0x7f8524147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- 172.21.15.26:0/3119143481 >> 172.21.15.26:0/3119143481 conn(0x7f8524066f70 msgr2=0x7f8524067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- 172.21.15.26:0/3119143481 shutdown_connections 2022-07-20T02:27:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- 172.21.15.26:0/3119143481 wait complete. 2022-07-20T02:27:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 Processor -- start 2022-07-20T02:27:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.724+0000 7f8523fff700 1 -- start start 2022-07-20T02:27:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85240618a0 0x7f8524147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8524143d60 0x7f85240b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524145710 con 0x7f85240b22d0 2022-07-20T02:27:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524146230 con 0x7f85240618a0 2022-07-20T02:27:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85241464d0 con 0x7f8524143d60 2022-07-20T02:27:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37164/0 (socket says 172.21.15.26:37164) 2022-07-20T02:27:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 -- 172.21.15.26:0/1287336292 learned_addr learned my addr 172.21.15.26:0/1287336292 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8522ffd700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85240618a0 0x7f8524147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f85237fe700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8524143d60 0x7f85240b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 -- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8524143d60 msgr2=0x7f85240b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8524143d60 0x7f85240b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 -- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85240618a0 msgr2=0x7f8524147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85240618a0 0x7f8524147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 -- 172.21.15.26:0/1287336292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c009530 con 0x7f85240b22d0 2022-07-20T02:27:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f851bfff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f850c00c000 tx=0x7f850c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c010780 con 0x7f85240b22d0 2022-07-20T02:27:36.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c0108e0 con 0x7f85240b22d0 2022-07-20T02:27:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524145900 con 0x7f85240b22d0 2022-07-20T02:27:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c010c40 con 0x7f85240b22d0 2022-07-20T02:27:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.728+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8524146a30 con 0x7f85240b22d0 2022-07-20T02:27:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f850c01c460 con 0x7f85240b22d0 2022-07-20T02:27:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524146690 con 0x7f85240b22d0 2022-07-20T02:27:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8520ff9700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 0x7f8508055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f850c06c3e0 con 0x7f85240b22d0 2022-07-20T02:27:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8522ffd700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 0x7f8508055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.732+0000 7f8522ffd700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 0x7f8508055ed0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f851400b000 tx=0x7f8514006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.740+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f850c043140 con 0x7f85240b22d0 2022-07-20T02:27:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.864+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8524145a90 con 0x7f85240b22d0 2022-07-20T02:27:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.864+0000 7f8520ff9700 1 -- 172.21.15.26:0/1287336292 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v45) v1 ==== 92+0+10 (secure 0 0 0) 0x7f850c046760 con 0x7f85240b22d0 2022-07-20T02:27:36.886 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:27:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 msgr2=0x7f8508055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 0x7f8508055ed0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f851400b000 tx=0x7f8514006df0).stop 2022-07-20T02:27:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 msgr2=0x7f85240af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f850c00c000 tx=0x7f850c006df0).stop 2022-07-20T02:27:36.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 shutdown_connections 2022-07-20T02:27:36.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85240b22d0 0x7f85240af450 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8508053a10 0x7f8508055ed0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8524143d60 0x7f85240b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 --2- 172.21.15.26:0/1287336292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85240618a0 0x7f8524147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:36.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 >> 172.21.15.26:0/1287336292 conn(0x7f8524066f70 msgr2=0x7f8524141fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:36.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 shutdown_connections 2022-07-20T02:27:36.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:36.868+0000 7f8523fff700 1 -- 172.21.15.26:0/1287336292 wait complete. 2022-07-20T02:27:36.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:27:37.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 Processor -- start 2022-07-20T02:27:37.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 -- start start 2022-07-20T02:27:37.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8f005cfd0 0x7fb8f00b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f005d3b0 con 0x7fb8f00b30e0 2022-07-20T02:27:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f00b5640 con 0x7fb8f00b4410 2022-07-20T02:27:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f8072700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f00b58e0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f660f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb8f00b30e0 0x7fb8f00b6440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36450/0 (socket says 172.21.15.26:36450) 2022-07-20T02:27:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f660f700 1 -- 172.21.15.26:0/3349298789 learned_addr learned my addr 172.21.15.26:0/3349298789 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f5e0e700 1 --2- 172.21.15.26:0/3349298789 >> v2:172.21.15.120:3300/0 conn(0x7fb8f005cfd0 0x7fb8f00b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641622970 0 0) 0x7fb8f005d3b0 con 0x7fb8f00b30e0 2022-07-20T02:27:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d8003260 con 0x7fb8f00b30e0 2022-07-20T02:27:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051204268 0 0) 0x7fb8f00b5640 con 0x7fb8f00b4410 2022-07-20T02:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f005d3b0 con 0x7fb8f00b4410 2022-07-20T02:27:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160942045 0 0) 0x7fb8d8003260 con 0x7fb8f00b30e0 2022-07-20T02:27:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/3349298789 >> v1:172.21.15.120:6789/0 conn(0x7fb8f00b4410 legacy=0x7fb8f00b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/3349298789 >> v1:172.21.15.26:6789/0 conn(0x7fb8f00b30e0 legacy=0x7fb8f00b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/3349298789 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00b5b80 con 0x7fb8f005cfd0 2022-07-20T02:27:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f5e0e700 1 --2- 172.21.15.26:0/3349298789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b4a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb8e8001490 tx=0x7fb8ec006df0).ready entity=mon.1 client_cookie=c9ee1f0f38bf143 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e80019e0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec000e90 con 0x7fb8f005cfd0 2022-07-20T02:27:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.020+0000 7fb8f4e0c700 1 -- 172.21.15.26:0/3349298789 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8ec012460 con 0x7fb8f005cfd0 2022-07-20T02:27:37.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- 172.21.15.26:0/3349298789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f005cfd0 msgr2=0x7fb8f00b4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 --2- 172.21.15.26:0/3349298789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b4a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb8e8001490 tx=0x7fb8ec006df0).stop 2022-07-20T02:27:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- 172.21.15.26:0/3349298789 shutdown_connections 2022-07-20T02:27:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 --2- 172.21.15.26:0/3349298789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b4a70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- 172.21.15.26:0/3349298789 >> 172.21.15.26:0/3349298789 conn(0x7fb8f0066f70 msgr2=0x7fb8f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- 172.21.15.26:0/3349298789 shutdown_connections 2022-07-20T02:27:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- 172.21.15.26:0/3349298789 wait complete. 2022-07-20T02:27:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 Processor -- start 2022-07-20T02:27:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- start start 2022-07-20T02:27:37.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00b30e0 0x7fb8f0149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00b4410 0x7fb8f015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b9980 con 0x7fb8f005cfd0 2022-07-20T02:27:37.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b9c20 con 0x7fb8f00b4410 2022-07-20T02:27:37.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f8072700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b9ec0 con 0x7fb8f00b30e0 2022-07-20T02:27:37.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.024+0000 7fb8f5e0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37174/0 (socket says 172.21.15.26:37174) 2022-07-20T02:27:37.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/2171761165 learned_addr learned my addr 172.21.15.26:0/2171761165 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:37.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f660f700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00b4410 0x7fb8f015a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f560d700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00b30e0 0x7fb8f0149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00b30e0 msgr2=0x7fb8f0149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00b30e0 0x7fb8f0149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00b4410 msgr2=0x7fb8f015a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00b4410 0x7fb8f015a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 -- 172.21.15.26:0/2171761165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec00b180 con 0x7fb8f005cfd0 2022-07-20T02:27:37.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f5e0e700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb8e8001490 tx=0x7fb8ec014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8ec016070 con 0x7fb8f005cfd0 2022-07-20T02:27:37.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec00bec0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f015ae70 con 0x7fb8f005cfd0 2022-07-20T02:27:37.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8ec012460 con 0x7fb8f005cfd0 2022-07-20T02:27:37.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f014a280 con 0x7fb8f005cfd0 2022-07-20T02:27:37.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb8ec00f030 con 0x7fb8f005cfd0 2022-07-20T02:27:37.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.028+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f00b84d0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.032+0000 7fb8deffd700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 0x7fb8d0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.032+0000 7fb8f560d700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 0x7fb8d0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.032+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb8ec06cc40 con 0x7fb8f005cfd0 2022-07-20T02:27:37.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.032+0000 7fb8f560d700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 0x7fb8d0055fc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb8e000d000 tx=0x7fb8e0007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.036+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8f00b84d0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.164+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8f00b84d0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.164+0000 7fb8deffd700 1 -- 172.21.15.26:0/2171761165 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb8f00b84d0 con 0x7fb8f005cfd0 2022-07-20T02:27:37.172 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:27:37.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 msgr2=0x7fb8d0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 0x7fb8d0055fc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb8e000d000 tx=0x7fb8e0007ad0).stop 2022-07-20T02:27:37.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 msgr2=0x7fb8f00b7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb8e8001490 tx=0x7fb8ec014000).stop 2022-07-20T02:27:37.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 shutdown_connections 2022-07-20T02:27:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb8d0053b00 0x7fb8d0055fc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8f00b4410 0x7fb8f015a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8f00b30e0 0x7fb8f0149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 --2- 172.21.15.26:0/2171761165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f005cfd0 0x7fb8f00b7910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 >> 172.21.15.26:0/2171761165 conn(0x7fb8f0066f70 msgr2=0x7fb8f00af570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 shutdown_connections 2022-07-20T02:27:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.168+0000 7fb8f8072700 1 -- 172.21.15.26:0/2171761165 wait complete. 2022-07-20T02:27:37.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:27:37.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 Processor -- start 2022-07-20T02:27:37.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 -- start start 2022-07-20T02:27:37.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51d409b9f0 0x7f51d40b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40613a0 con 0x7f51d40a9690 2022-07-20T02:27:37.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40615f0 con 0x7f51d409b9f0 2022-07-20T02:27:37.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51dac77700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40b1980 con 0x7f51d40b31a0 2022-07-20T02:27:37.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51d409b9f0 0x7f51d40b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51d409b9f0 0x7f51d40b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37186/0 (socket says 172.21.15.26:37186) 2022-07-20T02:27:37.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 -- 172.21.15.26:0/1767674160 learned_addr learned my addr 172.21.15.26:0/1767674160 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:37.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d8a13700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 -- 172.21.15.26:0/1767674160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 msgr2=0x7f51d40b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:37.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 -- 172.21.15.26:0/1767674160 >> v1:172.21.15.120:6789/0 conn(0x7f51d40a9690 legacy=0x7f51d4067070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.324+0000 7f51d9214700 1 -- 172.21.15.26:0/1767674160 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40b3580 con 0x7f51d409b9f0 2022-07-20T02:27:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51d9214700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d40b2310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f51cc0032f0 tx=0x7f51cc009290).ready entity=mon.0 client_cookie=5ceadbd90a64fda2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51d37fe700 1 -- 172.21.15.26:0/1767674160 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc00b8a0 con 0x7f51d409b9f0 2022-07-20T02:27:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51d37fe700 1 -- 172.21.15.26:0/1767674160 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc00ba00 con 0x7f51d409b9f0 2022-07-20T02:27:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51d37fe700 1 -- 172.21.15.26:0/1767674160 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc00bb70 con 0x7f51d409b9f0 2022-07-20T02:27:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- 172.21.15.26:0/1767674160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 msgr2=0x7f51d40b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d40b2310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f51cc0032f0 tx=0x7f51cc009290).stop 2022-07-20T02:27:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- 172.21.15.26:0/1767674160 shutdown_connections 2022-07-20T02:27:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d40b2310 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 --2- 172.21.15.26:0/1767674160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- 172.21.15.26:0/1767674160 >> 172.21.15.26:0/1767674160 conn(0x7f51d40b6920 msgr2=0x7f51d40b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- 172.21.15.26:0/1767674160 shutdown_connections 2022-07-20T02:27:37.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- 172.21.15.26:0/1767674160 wait complete. 2022-07-20T02:27:37.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 Processor -- start 2022-07-20T02:27:37.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.328+0000 7f51dac77700 1 -- start start 2022-07-20T02:27:37.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d40a9690 0x7f51d40aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4092130 con 0x7f51d409b9f0 2022-07-20T02:27:37.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4091630 con 0x7f51d40a9690 2022-07-20T02:27:37.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4151260 con 0x7f51d40b31a0 2022-07-20T02:27:37.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37188/0 (socket says 172.21.15.26:37188) 2022-07-20T02:27:37.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 -- 172.21.15.26:0/3632305537 learned_addr learned my addr 172.21.15.26:0/3632305537 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:37.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d3fff700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d40a9690 0x7f51d40aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d9214700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 -- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 msgr2=0x7f51d40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 -- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d40a9690 msgr2=0x7f51d40aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d40a9690 0x7f51d40aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 -- 172.21.15.26:0/3632305537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc009530 con 0x7f51d409b9f0 2022-07-20T02:27:37.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d9214700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:37.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d8a13700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f51d40671e0 tx=0x7f51c8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51c8010780 con 0x7f51d409b9f0 2022-07-20T02:27:37.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c80108e0 con 0x7f51d409b9f0 2022-07-20T02:27:37.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d4091e80 con 0x7f51d409b9f0 2022-07-20T02:27:37.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51c8010c40 con 0x7f51d409b9f0 2022-07-20T02:27:37.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d4091be0 con 0x7f51d409b9f0 2022-07-20T02:27:37.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d4097610 con 0x7f51d409b9f0 2022-07-20T02:27:37.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51c801c460 con 0x7f51d409b9f0 2022-07-20T02:27:37.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.332+0000 7f51d1ffb700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 0x7f51b4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.336+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f51c806c540 con 0x7f51d409b9f0 2022-07-20T02:27:37.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.336+0000 7f51d3fff700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 0x7f51b4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.336+0000 7f51d3fff700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 0x7f51b4055f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f51c000b000 tx=0x7f51c0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.340+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f51c8043200 con 0x7f51d409b9f0 2022-07-20T02:27:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.468+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f51d40574e0 con 0x7f51d409b9f0 2022-07-20T02:27:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.468+0000 7f51d1ffb700 1 -- 172.21.15.26:0/3632305537 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+10 (secure 0 0 0) 0x7f51c8046820 con 0x7f51d409b9f0 2022-07-20T02:27:37.473 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:27:37.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 msgr2=0x7f51b4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 0x7f51b4055f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f51c000b000 tx=0x7f51c0009040).stop 2022-07-20T02:27:37.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 msgr2=0x7f51d4066500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f51d40671e0 tx=0x7f51c8006df0).stop 2022-07-20T02:27:37.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 shutdown_connections 2022-07-20T02:27:37.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51b4053a70 0x7f51b4055f30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d40b31a0 0x7f51d40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d40a9690 0x7f51d40aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 --2- 172.21.15.26:0/3632305537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51d409b9f0 0x7f51d4066500 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 >> 172.21.15.26:0/3632305537 conn(0x7f51d40b6920 msgr2=0x7f51d40b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 shutdown_connections 2022-07-20T02:27:37.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.472+0000 7f51dac77700 1 -- 172.21.15.26:0/3632305537 wait complete. 2022-07-20T02:27:37.499 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:27:37.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:37.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 Processor -- start 2022-07-20T02:27:37.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 -- start start 2022-07-20T02:27:37.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6d2c0980d0 0x7f6d2c142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c05ff70 con 0x7f6d2c0b3260 2022-07-20T02:27:37.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c0b3640 con 0x7f6d2c0980d0 2022-07-20T02:27:37.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0601a0 con 0x7f6d2c061860 2022-07-20T02:27:37.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2affd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6d2c0b3260 0x7f6d2c0979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36478/0 (socket says 172.21.15.26:36478) 2022-07-20T02:27:37.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6d2c0980d0 0x7f6d2c142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2affd700 1 -- 172.21.15.26:0/1263527822 learned_addr learned my addr 172.21.15.26:0/1263527822 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2a7fc700 1 --2- 172.21.15.26:0/1263527822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d29ffb700 1 -- 172.21.15.26:0/1263527822 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156616307 0 0) 0x7f6d2c05ff70 con 0x7f6d2c0b3260 2022-07-20T02:27:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d29ffb700 1 -- 172.21.15.26:0/1263527822 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d10003260 con 0x7f6d2c0b3260 2022-07-20T02:27:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2a7fc700 1 -- 172.21.15.26:0/1263527822 >> v2:172.21.15.26:3300/0 conn(0x7f6d2c0980d0 msgr2=0x7f6d2c142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2a7fc700 1 --2- 172.21.15.26:0/1263527822 >> v2:172.21.15.26:3300/0 conn(0x7f6d2c0980d0 0x7f6d2c142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2a7fc700 1 -- 172.21.15.26:0/1263527822 >> v1:172.21.15.26:6789/0 conn(0x7f6d2c0b3260 legacy=0x7f6d2c0979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.640+0000 7f6d2a7fc700 1 -- 172.21.15.26:0/1263527822 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0606e0 con 0x7f6d2c061860 2022-07-20T02:27:37.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2a7fc700 1 --2- 172.21.15.26:0/1263527822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c067900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6d1c0032f0 tx=0x7f6d1c0091d0).ready entity=mon.2 client_cookie=e4b50509bdeb0dfa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d29ffb700 1 -- 172.21.15.26:0/1263527822 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00b600 con 0x7f6d2c061860 2022-07-20T02:27:37.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d29ffb700 1 -- 172.21.15.26:0/1263527822 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c00b760 con 0x7f6d2c061860 2022-07-20T02:27:37.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d29ffb700 1 -- 172.21.15.26:0/1263527822 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00b8d0 con 0x7f6d2c061860 2022-07-20T02:27:37.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- 172.21.15.26:0/1263527822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 msgr2=0x7f6d2c067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/1263527822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c067900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6d1c0032f0 tx=0x7f6d1c0091d0).stop 2022-07-20T02:27:37.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- 172.21.15.26:0/1263527822 shutdown_connections 2022-07-20T02:27:37.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/1263527822 >> v2:172.21.15.26:3300/0 conn(0x7f6d2c0980d0 0x7f6d2c142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/1263527822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c067900 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- 172.21.15.26:0/1263527822 >> 172.21.15.26:0/1263527822 conn(0x7f6d2c0b67f0 msgr2=0x7f6d2c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- 172.21.15.26:0/1263527822 shutdown_connections 2022-07-20T02:27:37.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- 172.21.15.26:0/1263527822 wait complete. 2022-07-20T02:27:37.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 Processor -- start 2022-07-20T02:27:37.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.644+0000 7f6d2bfff700 1 -- start start 2022-07-20T02:27:37.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c0b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c0980d0 0x7f6d2c1543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c062ff0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c063760 con 0x7f6d2c0980d0 2022-07-20T02:27:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c1548b0 con 0x7f6d2c061860 2022-07-20T02:27:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37206/0 (socket says 172.21.15.26:37206) 2022-07-20T02:27:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 -- 172.21.15.26:0/3268584749 learned_addr learned my addr 172.21.15.26:0/3268584749 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2affd700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c0b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2a7fc700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c0980d0 0x7f6d2c1543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 -- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 msgr2=0x7f6d2c0b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c0b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 -- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c0980d0 msgr2=0x7f6d2c1543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c0980d0 0x7f6d2c1543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 -- 172.21.15.26:0/3268584749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c009470 con 0x7f6d2c0b3260 2022-07-20T02:27:37.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2b7fe700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f6d2400c000 tx=0x7f6d24006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d24011750 con 0x7f6d2c0b3260 2022-07-20T02:27:37.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d240118b0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c063220 con 0x7f6d2c0b3260 2022-07-20T02:27:37.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c09a3d0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d24011c10 con 0x7f6d2c0b3260 2022-07-20T02:27:37.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0574e0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.648+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d24011d70 con 0x7f6d2c0b3260 2022-07-20T02:27:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.652+0000 7f6d1bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 0x7f6d08055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.652+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6d2406c5a0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.652+0000 7f6d2affd700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 0x7f6d08055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.652+0000 7f6d2affd700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 0x7f6d08055ff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f6d2000d000 tx=0x7f6d20009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.656+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d240432c0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.816+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d2c1408d0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d1bfff700 1 -- 172.21.15.26:0/3268584749 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f6d240468e0 con 0x7f6d2c0b3260 2022-07-20T02:27:37.831 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:37.831 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:37.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 msgr2=0x7f6d08055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 0x7f6d08055ff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f6d2000d000 tx=0x7f6d20009040).stop 2022-07-20T02:27:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 msgr2=0x7f6d2c066340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f6d2400c000 tx=0x7f6d24006df0).stop 2022-07-20T02:27:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 shutdown_connections 2022-07-20T02:27:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d2c0b3260 0x7f6d2c066340 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d08053b30 0x7f6d08055ff0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c0980d0 0x7f6d2c1543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 --2- 172.21.15.26:0/3268584749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d2c061860 0x7f6d2c0b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 >> 172.21.15.26:0/3268584749 conn(0x7f6d2c0b67f0 msgr2=0x7f6d2c141f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:37.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 shutdown_connections 2022-07-20T02:27:37.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:37.820+0000 7f6d2bfff700 1 -- 172.21.15.26:0/3268584749 wait complete. 2022-07-20T02:27:37.861 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:38.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:27:32.932112+0000', 'modified': '2022-07-20T02:27:33.936408+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'c', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3286368975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3286368975}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3286368975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:27:38.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:27:38.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:27:38.300 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":5.3539789979999997,"mdsmap_epoch":18,"osdmap_epoch":45,"osdmap_epoch_barrier":44,"uptime":10.581972468} 2022-07-20T02:27:38.301 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 45, "osdmap_epoch_barrier": 44, "rank_uptime": 5.353978998, "state": "up:active", "uptime": 10.581972468, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:27:38.302 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:27:38.302 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:27:38.303 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:27:38.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.304 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:38.325 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:27:38.384 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:27:38.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:27:38.401 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:27:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:27:38.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.404 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:27:38.454 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:27:38.454 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:38.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:27:38.493 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:27:38.493 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:27:38.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:27:38.528+0000 7f29c57e8080 -1 init, newargv = 0x5574e4ab38f0 newargc=9ceph-fuse[22395 2022-07-20T02:27:38.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:]: starting ceph client 2022-07-20T02:27:38.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22395]: starting fuse 2022-07-20T02:27:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:27:38.548 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:27:38.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.549 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:27:38.597 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:27:38.598 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:27:38.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.599 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:27:38.599 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:27:38.600 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:27:38.600 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:27:38.600 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:27:38.601 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.601 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:27:38.602 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:27:38.603 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:27:38.603 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.604 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:27:38.604 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:27:38.605 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:27:38.605 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:27:38.605 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.606 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:27:38.606 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:27:38.607 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:27:38.607 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:27:38.608 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:27:38.608 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.608 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:27:38.609 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:27:38.691 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22395.asok 2022-07-20T02:27:38.692 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22395.asok 2022-07-20T02:27:38.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22395.asok status 2022-07-20T02:27:38.830 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:27:38.830 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:27:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:27:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:27:38.831 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:27:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:27:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:27:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22395", 2022-07-20T02:27:38.832 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:27:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:27:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:27:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:27:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 5368, 2022-07-20T02:27:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:27:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:27:38.835 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:27:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 5368 2022-07-20T02:27:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:27:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:27:38.836 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:27:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:27:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3193302735 2022-07-20T02:27:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:27:38.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:27:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:27:38.838 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:27:38.839 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:27:38.839 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3193302735 2022-07-20T02:27:38.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:27:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.5368 172.21.15.159:0/3193302735", 2022-07-20T02:27:38.843 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/3193302735", 2022-07-20T02:27:38.844 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:27:38.844 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 18, 2022-07-20T02:27:38.845 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 45, 2022-07-20T02:27:38.845 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:27:38.845 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:27:38.845 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:27:38.846 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:27:38.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.865 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:38.879 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:38.892 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:27:38.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:27:38.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.894 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:38.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:27:38.953 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:27:38.954 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:27:38.971 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:27:38.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:27:38.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:27:38.975 DEBUG:teuthology.orchestra.run.smithi159:> import sys 2022-07-20T02:27:38.975 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:27:38.976 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:27:38.976 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.977 DEBUG:teuthology.orchestra.run.smithi159:> n = 1000 2022-07-20T02:27:38.977 DEBUG:teuthology.orchestra.run.smithi159:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T02:27:38.977 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.978 DEBUG:teuthology.orchestra.run.smithi159:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T02:27:38.978 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T02:27:38.980 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:27:38.981 DEBUG:teuthology.orchestra.run.smithi159:> for i in range(0, n): 2022-07-20T02:27:38.983 DEBUG:teuthology.orchestra.run.smithi159:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T02:27:38.984 DEBUG:teuthology.orchestra.run.smithi159:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T02:27:38.985 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:27:38.986 DEBUG:teuthology.orchestra.run.smithi159:> if True: 2022-07-20T02:27:38.987 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:27:38.988 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(f.fileno()) 2022-07-20T02:27:38.988 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:27:46.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:46.360 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:46.369 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:46.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:27:50.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:50.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.212+0000 7f80f3dd7700 1 Processor -- start 2022-07-20T02:27:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.212+0000 7f80f3dd7700 1 -- start start 2022-07-20T02:27:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80ec0b2ac0 0x7f80ec0ab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f80ec0aa4a0 0x7f80ec140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0b2ea0 con 0x7f80ec061030 2022-07-20T02:27:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec0abb30 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec062420 con 0x7f80ec0b2ac0 2022-07-20T02:27:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f80ec0aa4a0 0x7f80ec140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f80ec0aa4a0 0x7f80ec140450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37216/0 (socket says 172.21.15.26:37216) 2022-07-20T02:27:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 -- 172.21.15.26:0/274863136 learned_addr learned my addr 172.21.15.26:0/274863136 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f1372700 1 --1- 172.21.15.26:0/274863136 >> v1:172.21.15.26:6789/0 conn(0x7f80ec061030 0x7f80ec0a9d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36494/0 (socket says 172.21.15.26:36494) 2022-07-20T02:27:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f1b73700 1 --2- 172.21.15.26:0/274863136 >> v2:172.21.15.120:3300/0 conn(0x7f80ec0b2ac0 0x7f80ec0ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 -- 172.21.15.26:0/274863136 >> v2:172.21.15.120:3300/0 conn(0x7f80ec0b2ac0 msgr2=0x7f80ec0ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 --2- 172.21.15.26:0/274863136 >> v2:172.21.15.120:3300/0 conn(0x7f80ec0b2ac0 0x7f80ec0ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 -- 172.21.15.26:0/274863136 >> v1:172.21.15.26:6789/0 conn(0x7f80ec061030 legacy=0x7f80ec0a9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 -- 172.21.15.26:0/274863136 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0a8fd0 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f2374700 1 --2- 172.21.15.26:0/274863136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec140450 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f80e40032f0 tx=0x7f80e4009290).ready entity=mon.0 client_cookie=41274b2d70d50b6f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f0b71700 1 -- 172.21.15.26:0/274863136 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e400b870 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f0b71700 1 -- 172.21.15.26:0/274863136 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e400b9d0 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f0b71700 1 -- 172.21.15.26:0/274863136 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e400bb40 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- 172.21.15.26:0/274863136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec0aa4a0 msgr2=0x7f80ec140450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/274863136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec140450 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f80e40032f0 tx=0x7f80e4009290).stop 2022-07-20T02:27:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- 172.21.15.26:0/274863136 shutdown_connections 2022-07-20T02:27:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/274863136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec140450 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/274863136 >> v2:172.21.15.120:3300/0 conn(0x7f80ec0b2ac0 0x7f80ec0ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- 172.21.15.26:0/274863136 >> 172.21.15.26:0/274863136 conn(0x7f80ec0b60a0 msgr2=0x7f80ec0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- 172.21.15.26:0/274863136 shutdown_connections 2022-07-20T02:27:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.216+0000 7f80f3dd7700 1 -- 172.21.15.26:0/274863136 wait complete. 2022-07-20T02:27:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 Processor -- start 2022-07-20T02:27:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 -- start start 2022-07-20T02:27:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec14aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80ec0b2ac0 0x7f80ec14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec0a8ad0 con 0x7f80ec061030 2022-07-20T02:27:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec064660 con 0x7f80ec0aa4a0 2022-07-20T02:27:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f3dd7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec14b4a0 con 0x7f80ec0b2ac0 2022-07-20T02:27:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37220/0 (socket says 172.21.15.26:37220) 2022-07-20T02:27:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1372700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec14aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 -- 172.21.15.26:0/2081618550 learned_addr learned my addr 172.21.15.26:0/2081618550 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f2374700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80ec0b2ac0 0x7f80ec14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 -- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80ec0b2ac0 msgr2=0x7f80ec14afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80ec0b2ac0 0x7f80ec14afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 -- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 msgr2=0x7f80ec14aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec14aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1b73700 1 -- 172.21.15.26:0/2081618550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e4009530 con 0x7f80ec061030 2022-07-20T02:27:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.220+0000 7f80f1372700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec14aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:27:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80f1b73700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f80ec0650f0 tx=0x7f80e8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e8010780 con 0x7f80ec061030 2022-07-20T02:27:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e80108e0 con 0x7f80ec061030 2022-07-20T02:27:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e8010c40 con 0x7f80ec061030 2022-07-20T02:27:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec137aa0 con 0x7f80ec061030 2022-07-20T02:27:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec066730 con 0x7f80ec061030 2022-07-20T02:27:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ec0665a0 con 0x7f80ec061030 2022-07-20T02:27:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f80e801c460 con 0x7f80ec061030 2022-07-20T02:27:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 0x7f80cc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f80e806c4c0 con 0x7f80ec061030 2022-07-20T02:27:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.224+0000 7f80f1372700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 0x7f80cc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.228+0000 7f80f1372700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 0x7f80cc055e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f80dc00d000 tx=0x7f80dc007b40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.232+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80e8043110 con 0x7f80ec061030 2022-07-20T02:27:50.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80ec0a9650 con 0x7f80ec061030 2022-07-20T02:27:50.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80daffd700 1 -- 172.21.15.26:0/2081618550 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f80e8046730 con 0x7f80ec061030 2022-07-20T02:27:50.395 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:50.395 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:50.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 msgr2=0x7f80cc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 0x7f80cc055e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f80dc00d000 tx=0x7f80dc007b40).stop 2022-07-20T02:27:50.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 msgr2=0x7f80ec136ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.392+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f80ec0650f0 tx=0x7f80e8006df0).stop 2022-07-20T02:27:50.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 shutdown_connections 2022-07-20T02:27:50.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f80cc053950 0x7f80cc055e10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80ec0b2ac0 0x7f80ec14afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80ec0aa4a0 0x7f80ec14aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 --2- 172.21.15.26:0/2081618550 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80ec061030 0x7f80ec136ca0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 >> 172.21.15.26:0/2081618550 conn(0x7f80ec0b60a0 msgr2=0x7f80ec0ad6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:50.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 shutdown_connections 2022-07-20T02:27:50.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.396+0000 7f80f3dd7700 1 -- 172.21.15.26:0/2081618550 wait complete. 2022-07-20T02:27:50.455 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:50.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config set mds_recall_max_caps 20 2022-07-20T02:27:50.567 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T02:27:50.568 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T02:27:50.569 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.664+0000 7fd35759e700 1 Processor -- start 2022-07-20T02:27:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.664+0000 7fd35759e700 1 -- start start 2022-07-20T02:27:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3580add30 0x7fd3580b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd3580af6d0 0x7fd3580afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580aa1e0 con 0x7fd358060950 2022-07-20T02:27:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580ae110 con 0x7fd3580af6d0 2022-07-20T02:27:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580a9f40 con 0x7fd3580add30 2022-07-20T02:27:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd3580af6d0 0x7fd3580afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd3580af6d0 0x7fd3580afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37230/0 (socket says 172.21.15.26:37230) 2022-07-20T02:27:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 -- 172.21.15.26:0/1108156348 learned_addr learned my addr 172.21.15.26:0/1108156348 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35659c700 1 --2- 172.21.15.26:0/1108156348 >> v2:172.21.15.120:3300/0 conn(0x7fd3580add30 0x7fd3580b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 -- 172.21.15.26:0/1108156348 >> v2:172.21.15.120:3300/0 conn(0x7fd3580add30 msgr2=0x7fd3580b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:50.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 --2- 172.21.15.26:0/1108156348 >> v2:172.21.15.120:3300/0 conn(0x7fd3580add30 0x7fd3580b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 -- 172.21.15.26:0/1108156348 >> v1:172.21.15.120:6789/0 conn(0x7fd358060950 legacy=0x7fd358060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 -- 172.21.15.26:0/1108156348 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580b1110 con 0x7fd3580af6d0 2022-07-20T02:27:50.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd356d9d700 1 --2- 172.21.15.26:0/1108156348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd3580afb30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fd348002f70 tx=0x7fd348006f40).ready entity=mon.0 client_cookie=ecfac633d3edac40 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35559a700 1 -- 172.21.15.26:0/1108156348 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34800da80 con 0x7fd3580af6d0 2022-07-20T02:27:50.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35559a700 1 -- 172.21.15.26:0/1108156348 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348000e90 con 0x7fd3580af6d0 2022-07-20T02:27:50.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35559a700 1 -- 172.21.15.26:0/1108156348 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34800de10 con 0x7fd3580af6d0 2022-07-20T02:27:50.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- 172.21.15.26:0/1108156348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 msgr2=0x7fd3580afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 --2- 172.21.15.26:0/1108156348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd3580afb30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fd348002f70 tx=0x7fd348006f40).stop 2022-07-20T02:27:50.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- 172.21.15.26:0/1108156348 shutdown_connections 2022-07-20T02:27:50.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 --2- 172.21.15.26:0/1108156348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd3580afb30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 --2- 172.21.15.26:0/1108156348 >> v2:172.21.15.120:3300/0 conn(0x7fd3580add30 0x7fd3580b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- 172.21.15.26:0/1108156348 >> 172.21.15.26:0/1108156348 conn(0x7fd3580b3050 msgr2=0x7fd3580b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:50.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- 172.21.15.26:0/1108156348 shutdown_connections 2022-07-20T02:27:50.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- 172.21.15.26:0/1108156348 wait complete. 2022-07-20T02:27:50.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 Processor -- start 2022-07-20T02:27:50.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.668+0000 7fd35759e700 1 -- start start 2022-07-20T02:27:50.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358060950 0x7fd3581402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3580add30 0x7fd358140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358141f00 con 0x7fd3580af6d0 2022-07-20T02:27:50.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358065b30 con 0x7fd358060950 2022-07-20T02:27:50.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd358156630 con 0x7fd3580add30 2022-07-20T02:27:50.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35659c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358060950 0x7fd3581402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37236/0 (socket says 172.21.15.26:37236) 2022-07-20T02:27:50.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 -- 172.21.15.26:0/83604834 learned_addr learned my addr 172.21.15.26:0/83604834 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:50.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd355d9b700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3580add30 0x7fd358140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 -- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3580add30 msgr2=0x7fd358140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3580add30 0x7fd358140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 -- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358060950 msgr2=0x7fd3581402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358060950 0x7fd3581402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 -- 172.21.15.26:0/83604834 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34800b460 con 0x7fd3580af6d0 2022-07-20T02:27:50.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd356d9d700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd3580a9770 tx=0x7fd3580ae860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34801d070 con 0x7fd3580af6d0 2022-07-20T02:27:50.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348010df0 con 0x7fd3580af6d0 2022-07-20T02:27:50.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3581568d0 con 0x7fd3580af6d0 2022-07-20T02:27:50.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3480185f0 con 0x7fd3580af6d0 2022-07-20T02:27:50.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3581408e0 con 0x7fd3580af6d0 2022-07-20T02:27:50.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.672+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd358141580 con 0x7fd3580af6d0 2022-07-20T02:27:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.676+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd348018750 con 0x7fd3580af6d0 2022-07-20T02:27:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.676+0000 7fd3477fe700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 0x7fd334055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.676+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd34806d450 con 0x7fd3580af6d0 2022-07-20T02:27:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.676+0000 7fd35659c700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 0x7fd334055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.676+0000 7fd35659c700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 0x7fd334055f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fd35000b000 tx=0x7fd3500091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.680+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd348044210 con 0x7fd3580af6d0 2022-07-20T02:27:50.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.840+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3580b13b0 con 0x7fd3580af6d0 2022-07-20T02:27:50.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.840+0000 7fd3477fe700 1 -- 172.21.15.26:0/83604834 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7fd348047830 con 0x7fd3580af6d0 2022-07-20T02:27:50.847 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:50.847 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 msgr2=0x7fd334055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 0x7fd334055f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fd35000b000 tx=0x7fd3500091d0).stop 2022-07-20T02:27:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 msgr2=0x7fd358138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd3580a9770 tx=0x7fd3580ae860).stop 2022-07-20T02:27:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 shutdown_connections 2022-07-20T02:27:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd3580af6d0 0x7fd358138800 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd334053aa0 0x7fd334055f60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3580add30 0x7fd358140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 --2- 172.21.15.26:0/83604834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358060950 0x7fd3581402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 >> 172.21.15.26:0/83604834 conn(0x7fd3580b3050 msgr2=0x7fd358060dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:50.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 shutdown_connections 2022-07-20T02:27:50.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:50.844+0000 7fd35759e700 1 -- 172.21.15.26:0/83604834 wait complete. 2022-07-20T02:27:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:50.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T02:27:51.017 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T02:27:51.018 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T02:27:51.018 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:27:51.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:27:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 Processor -- start 2022-07-20T02:27:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 -- start start 2022-07-20T02:27:51.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f381474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6f380a7fc0 0x7f6f381479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f380a8c40 0x7f6f380aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f38147eb0 con 0x7f6f380a8c40 2022-07-20T02:27:51.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380b62d0 con 0x7f6f380a7fc0 2022-07-20T02:27:51.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3f0ab700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3814aac0 con 0x7f6f380b5ef0 2022-07-20T02:27:51.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f380a8c40 0x7f6f380aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6f380a8c40 0x7f6f380aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37244/0 (socket says 172.21.15.26:37244) 2022-07-20T02:27:51.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 -- 172.21.15.26:0/98951152 learned_addr learned my addr 172.21.15.26:0/98951152 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3d8a8700 1 --2- 172.21.15.26:0/98951152 >> v2:172.21.15.120:3300/0 conn(0x7f6f380a7fc0 0x7f6f381479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e0a9700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f381474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 -- 172.21.15.26:0/98951152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 msgr2=0x7f6f381474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f381474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 -- 172.21.15.26:0/98951152 >> v2:172.21.15.120:3300/0 conn(0x7f6f380a7fc0 msgr2=0x7f6f381479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 --2- 172.21.15.26:0/98951152 >> v2:172.21.15.120:3300/0 conn(0x7f6f380a7fc0 0x7f6f381479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 -- 172.21.15.26:0/98951152 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3814ad60 con 0x7f6f380a8c40 2022-07-20T02:27:51.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3e8aa700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a8c40 0x7f6f380aee40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6f300032f0 tx=0x7f6f30009290).ready entity=mon.0 client_cookie=73f20a8ddf5f7639 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3d0a7700 1 -- 172.21.15.26:0/98951152 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f3000b870 con 0x7f6f380a8c40 2022-07-20T02:27:51.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3d0a7700 1 -- 172.21.15.26:0/98951152 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3000b9d0 con 0x7f6f380a8c40 2022-07-20T02:27:51.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.144+0000 7f6f3d0a7700 1 -- 172.21.15.26:0/98951152 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f3000bb40 con 0x7f6f380a8c40 2022-07-20T02:27:51.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/98951152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a8c40 msgr2=0x7f6f380aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a8c40 0x7f6f380aee40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6f300032f0 tx=0x7f6f30009290).stop 2022-07-20T02:27:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/98951152 shutdown_connections 2022-07-20T02:27:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a8c40 0x7f6f380aee40 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/98951152 >> v2:172.21.15.120:3300/0 conn(0x7f6f380a7fc0 0x7f6f381479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/98951152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f381474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/98951152 >> 172.21.15.26:0/98951152 conn(0x7f6f38060310 msgr2=0x7f6f380b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/98951152 shutdown_connections 2022-07-20T02:27:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/98951152 wait complete. 2022-07-20T02:27:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 Processor -- start 2022-07-20T02:27:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- start start 2022-07-20T02:27:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 0x7f6f38062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f380a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380abc90 con 0x7f6f380a7fc0 2022-07-20T02:27:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380626f0 con 0x7f6f380a8c40 2022-07-20T02:27:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.148+0000 7f6f3f0ab700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380aaed0 con 0x7f6f380b5ef0 2022-07-20T02:27:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37246/0 (socket says 172.21.15.26:37246) 2022-07-20T02:27:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 -- 172.21.15.26:0/2632719298 learned_addr learned my addr 172.21.15.26:0/2632719298 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3d8a8700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 0x7f6f38062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e8aa700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f380a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 -- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 msgr2=0x7f6f380a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f380a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 -- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 msgr2=0x7f6f38062b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 0x7f6f38062b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 -- 172.21.15.26:0/2632719298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009530 con 0x7f6f380a7fc0 2022-07-20T02:27:51.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3d8a8700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 0x7f6f38062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:51.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e8aa700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f380a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:51.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3e0a9700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6f280030b0 tx=0x7f6f28008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f28011780 con 0x7f6f380a7fc0 2022-07-20T02:27:51.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f280118e0 con 0x7f6f380a7fc0 2022-07-20T02:27:51.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f28011c40 con 0x7f6f380a7fc0 2022-07-20T02:27:51.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380abec0 con 0x7f6f380a7fc0 2022-07-20T02:27:51.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f38065170 con 0x7f6f380a7fc0 2022-07-20T02:27:51.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38062860 con 0x7f6f380a7fc0 2022-07-20T02:27:51.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.152+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6f28012420 con 0x7f6f380a7fc0 2022-07-20T02:27:51.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.156+0000 7f6f277fe700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 0x7f6f18055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.156+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6f2806bf40 con 0x7f6f380a7fc0 2022-07-20T02:27:51.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.156+0000 7f6f3d8a8700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 0x7f6f18055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.156+0000 7f6f3d8a8700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 0x7f6f18055ea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6f3400b000 tx=0x7f6f34008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.160+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f28042c20 con 0x7f6f380a7fc0 2022-07-20T02:27:51.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.324+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f3814b000 con 0x7f6f380a7fc0 2022-07-20T02:27:51.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.324+0000 7f6f277fe700 1 -- 172.21.15.26:0/2632719298 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f6f28046240 con 0x7f6f380a7fc0 2022-07-20T02:27:51.345 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:27:51.345 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:27:51.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 msgr2=0x7f6f18055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 0x7f6f18055ea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6f3400b000 tx=0x7f6f34008040).stop 2022-07-20T02:27:51.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 msgr2=0x7f6f380b54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6f280030b0 tx=0x7f6f28008040).stop 2022-07-20T02:27:51.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 shutdown_connections 2022-07-20T02:27:51.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6f180539e0 0x7f6f18055ea0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6f380b5ef0 0x7f6f380a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f380a8c40 0x7f6f38062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 --2- 172.21.15.26:0/2632719298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f380a7fc0 0x7f6f380b54f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 >> 172.21.15.26:0/2632719298 conn(0x7f6f38060310 msgr2=0x7f6f38147180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 shutdown_connections 2022-07-20T02:27:51.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.328+0000 7f6f3f0ab700 1 -- 172.21.15.26:0/2632719298 wait complete. 2022-07-20T02:27:51.367 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:27:51.369 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c cache drop 2022-07-20T02:27:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 Processor -- start 2022-07-20T02:27:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 -- start start 2022-07-20T02:27:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcddc0add30 0x7fcddc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc060950 0x7fcddc060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc0aa1e0 con 0x7fcddc061280 2022-07-20T02:27:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0ae110 con 0x7fcddc0add30 2022-07-20T02:27:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde10af700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0a9f40 con 0x7fcddc060950 2022-07-20T02:27:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde08ae700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcddc061280 0x7fcddc143450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36538/0 (socket says 172.21.15.26:36538) 2022-07-20T02:27:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcde08ae700 1 -- 172.21.15.26:0/1547769030 learned_addr learned my addr 172.21.15.26:0/1547769030 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:27:51.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1547769030 >> v2:172.21.15.120:3300/0 conn(0x7fcddc0add30 0x7fcddc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.480+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc060950 0x7fcddc060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddaffd700 1 -- 172.21.15.26:0/1547769030 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466785060 0 0) 0x7fcddc0aa1e0 con 0x7fcddc061280 2022-07-20T02:27:51.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddaffd700 1 -- 172.21.15.26:0/1547769030 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdbc003260 con 0x7fcddc061280 2022-07-20T02:27:51.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddbfff700 1 -- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc060950 msgr2=0x7fcddc060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc060950 0x7fcddc060d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddbfff700 1 -- 172.21.15.26:0/1547769030 >> v1:172.21.15.26:6789/0 conn(0x7fcddc061280 legacy=0x7fcddc143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddbfff700 1 -- 172.21.15.26:0/1547769030 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0b1110 con 0x7fcddc0add30 2022-07-20T02:27:51.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc0add30 0x7fcddc0b2ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fcdcc002f70 tx=0x7fcdcc006e80).ready entity=mon.1 client_cookie=9ad70ec5cebc019b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddaffd700 1 -- 172.21.15.26:0/1547769030 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdcc00b900 con 0x7fcddc0add30 2022-07-20T02:27:51.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddaffd700 1 -- 172.21.15.26:0/1547769030 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc000e90 con 0x7fcddc0add30 2022-07-20T02:27:51.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcddaffd700 1 -- 172.21.15.26:0/1547769030 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdcc00bc50 con 0x7fcddc0add30 2022-07-20T02:27:51.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 -- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc0add30 msgr2=0x7fcddc0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc0add30 0x7fcddc0b2ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fcdcc002f70 tx=0x7fcdcc006e80).stop 2022-07-20T02:27:51.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 -- 172.21.15.26:0/1547769030 shutdown_connections 2022-07-20T02:27:51.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc060950 0x7fcddc060d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 --2- 172.21.15.26:0/1547769030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc0add30 0x7fcddc0b2ab0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 -- 172.21.15.26:0/1547769030 >> 172.21.15.26:0/1547769030 conn(0x7fcddc0b3050 msgr2=0x7fcddc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 -- 172.21.15.26:0/1547769030 shutdown_connections 2022-07-20T02:27:51.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 -- 172.21.15.26:0/1547769030 wait complete. 2022-07-20T02:27:51.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.484+0000 7fcde10af700 1 Processor -- start 2022-07-20T02:27:51.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- start start 2022-07-20T02:27:51.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc060950 0x7fcddc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc061280 0x7fcddc1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc155800 con 0x7fcddc0add30 2022-07-20T02:27:51.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc155aa0 con 0x7fcddc060950 2022-07-20T02:27:51.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc155d40 con 0x7fcddc061280 2022-07-20T02:27:51.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37264/0 (socket says 172.21.15.26:37264) 2022-07-20T02:27:51.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 -- 172.21.15.26:0/563852370 learned_addr learned my addr 172.21.15.26:0/563852370 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcddbfff700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc060950 0x7fcddc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc061280 0x7fcddc1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 -- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc061280 msgr2=0x7fcddc1520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc061280 0x7fcddc1520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 -- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc060950 msgr2=0x7fcddc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc060950 0x7fcddc0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 -- 172.21.15.26:0/563852370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc00d410 con 0x7fcddc0add30 2022-07-20T02:27:51.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde08ae700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fcdd4001110 tx=0x7fcdd4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcdd97fa700 1 -- 172.21.15.26:0/563852370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdd40070d0 con 0x7fcddc0add30 2022-07-20T02:27:51.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc155fe0 con 0x7fcddc0add30 2022-07-20T02:27:51.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcdd97fa700 1 -- 172.21.15.26:0/563852370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd4007230 con 0x7fcddc0add30 2022-07-20T02:27:51.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcdd97fa700 1 -- 172.21.15.26:0/563852370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdd4005610 con 0x7fcddc0add30 2022-07-20T02:27:51.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.488+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc0ab8c0 con 0x7fcddc0add30 2022-07-20T02:27:51.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcdd97fa700 1 -- 172.21.15.26:0/563852370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcdd4010070 con 0x7fcddc0add30 2022-07-20T02:27:51.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcdd97fa700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 0x7fcdc0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcdd97fa700 1 -- 172.21.15.26:0/563852370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcdd406b7a0 con 0x7fcddc0add30 2022-07-20T02:27:51.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcddbfff700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 0x7fcdc0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcddbfff700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 0x7fcdc0055e40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fcdcc01a000 tx=0x7fcddc0b3db0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcdec568740 1 Processor -- start 2022-07-20T02:27:51.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.492+0000 7fcdec568740 1 -- start start 2022-07-20T02:27:51.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x15d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x1510010 con 0x15d3410 2022-07-20T02:27:51.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x14f0250 con 0x15d1330 2022-07-20T02:27:51.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x159e170 con 0x15d8bc0 2022-07-20T02:27:51.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37270/0 (socket says 172.21.15.26:37270) 2022-07-20T02:27:51.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3363916631 learned_addr learned my addr 172.21.15.26:0/3363916631 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddbfff700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcde08ae700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x15d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 msgr2=0x15d2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 msgr2=0x15d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x15d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3363916631 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1548030 con 0x15d3410 2022-07-20T02:27:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcde08ae700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x15d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddbfff700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x15a0240 tx=0x7fcdc40091d0).ready entity=mon.0 client_cookie=78046814f28a986c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcde10af700 1 -- 172.21.15.26:0/3363916631 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc4011cf0 con 0x15d3410 2022-07-20T02:27:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcde10af700 1 -- 172.21.15.26:0/3363916631 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4011e50 con 0x15d3410 2022-07-20T02:27:51.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcde10af700 1 -- 172.21.15.26:0/3363916631 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc4014440 con 0x15d3410 2022-07-20T02:27:51.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- 172.21.15.26:0/3363916631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 msgr2=0x15d3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x15a0240 tx=0x7fcdc40091d0).stop 2022-07-20T02:27:51.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- 172.21.15.26:0/3363916631 shutdown_connections 2022-07-20T02:27:51.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d3870 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x15d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 --2- 172.21.15.26:0/3363916631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- 172.21.15.26:0/3363916631 >> 172.21.15.26:0/3363916631 conn(0x159d540 msgr2=0x15d8150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- 172.21.15.26:0/3363916631 shutdown_connections 2022-07-20T02:27:51.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.496+0000 7fcdec568740 1 -- 172.21.15.26:0/3363916631 wait complete. 2022-07-20T02:27:51.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- start start 2022-07-20T02:27:51.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 Processor -- start 2022-07-20T02:27:51.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d3410 0x15cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d8bc0 0x159aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x159c7f0 con 0x15d1330 2022-07-20T02:27:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x159a280 con 0x15d8bc0 2022-07-20T02:27:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x159a520 con 0x15d3410 2022-07-20T02:27:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 10 client.0 ll_register_callbacks cb 0x1557610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:27:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37274/0 (socket says 172.21.15.26:37274) 2022-07-20T02:27:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 -- 172.21.15.26:0/1678396795 learned_addr learned my addr 172.21.15.26:0/1678396795 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcde08ae700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d8bc0 0x159aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d3410 0x15cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 -- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d3410 msgr2=0x15cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d3410 0x15cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 -- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d8bc0 msgr2=0x159aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d8bc0 0x159aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 -- 172.21.15.26:0/1678396795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc4009970 con 0x15d1330 2022-07-20T02:27:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddbfff700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fcdcc025000 tx=0x7fcdcc01f000).ready entity=mon.0 client_cookie=2573089db0a9bfbd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:27:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdcc02b070 con 0x15d1330 2022-07-20T02:27:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1505fc0 con 0x15d1330 2022-07-20T02:27:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcdcc0289d0 con 0x15d1330 2022-07-20T02:27:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdcc01f070 con 0x15d1330 2022-07-20T02:27:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 10 client.5433 fetch_fsmap learned FSMap version 18 2022-07-20T02:27:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x14e29b0 con 0x15d1330 2022-07-20T02:27:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1505fc0 con 0x15d1330 2022-07-20T02:27:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 18) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fcdcc02e020 con 0x15d1330 2022-07-20T02:27:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 10 client.5433 fetch_fsmap finished waiting for FSMap version 18 2022-07-20T02:27:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 10 client.5433 resolve_mds: resolved ID 'c' to GID 5241 2022-07-20T02:27:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 20 client.5433 populate_metadata read hostname 'smithi026' 2022-07-20T02:27:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x159b450 0x15e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 4 client.5433 mds_command: new command op to 5241 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T02:27:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 --> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x14fe040 con 0x159b450 2022-07-20T02:27:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.500+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x159b450 0x15e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x159b450 0x15e6b20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=20e9b9781d78a9bf server_cookie=9b26cec7b50a8881 in_seq=0 out_seq=0 2022-07-20T02:27:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdbaffd700 10 client.5433 ms_handle_connect on v2:172.21.15.26:6836/3286368975 2022-07-20T02:27:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdbaffd700 1 -- 172.21.15.26:0/1678396795 <== mds.0 v2:172.21.15.26:6836/3286368975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x14fe040 con 0x159b450 2022-07-20T02:27:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdbaffd700 10 client.5433 handle_command_reply: tid=0 2022-07-20T02:27:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 client.5433 shutdown 2022-07-20T02:27:51.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 msgr2=0x15cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fcdcc025000 tx=0x7fcdcc01f000).stop 2022-07-20T02:27:51.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 shutdown_connections 2022-07-20T02:27:51.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d8bc0 0x159aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d3410 0x15cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x159b450 0x15e6b20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 --2- 172.21.15.26:0/1678396795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15cce90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdbaffd700 0 client.5433 ms_handle_reset on v2:172.21.15.26:6836/3286368975 2022-07-20T02:27:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 >> 172.21.15.26:0/1678396795 conn(0x159d540 msgr2=0x159e300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 shutdown_connections 2022-07-20T02:27:51.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 1 -- 172.21.15.26:0/1678396795 wait complete. 2022-07-20T02:27:51.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.504+0000 7fcdec568740 20 client.5433 trim_cache size 0 max 16384 2022-07-20T02:27:51.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.516+0000 7fcdec568740 1 Processor -- start 2022-07-20T02:27:51.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.516+0000 7fcdec568740 1 -- start start 2022-07-20T02:27:51.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x159f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x153cb70 con 0x15d3410 2022-07-20T02:27:51.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x14fe040 con 0x15d1330 2022-07-20T02:27:51.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x15a0080 con 0x15d8bc0 2022-07-20T02:27:51.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37284/0 (socket says 172.21.15.26:37284) 2022-07-20T02:27:51.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3271592997 learned_addr learned my addr 172.21.15.26:0/3271592997 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddbfff700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x159f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcde08ae700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 msgr2=0x15d71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 msgr2=0x159f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x159f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 -- 172.21.15.26:0/3271592997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x14ff5b0 con 0x15d3410 2022-07-20T02:27:51.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcde08ae700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:27:51.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fcdc402c000 tx=0x7fcdc401a000).ready entity=mon.0 client_cookie=8500345e8cddd2c9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdd17fa700 1 -- 172.21.15.26:0/3271592997 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc4006ea0 con 0x15d3410 2022-07-20T02:27:51.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdd17fa700 1 -- 172.21.15.26:0/3271592997 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4009ed0 con 0x15d3410 2022-07-20T02:27:51.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdd17fa700 1 -- 172.21.15.26:0/3271592997 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc4026040 con 0x15d3410 2022-07-20T02:27:51.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- 172.21.15.26:0/3271592997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 msgr2=0x15d6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:27:51.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fcdc402c000 tx=0x7fcdc401a000).stop 2022-07-20T02:27:51.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- 172.21.15.26:0/3271592997 shutdown_connections 2022-07-20T02:27:51.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d3410 0x15d6cc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x15d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 --2- 172.21.15.26:0/3271592997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d1330 0x159f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- 172.21.15.26:0/3271592997 >> 172.21.15.26:0/3271592997 conn(0x159d540 msgr2=0x159e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:27:51.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- 172.21.15.26:0/3271592997 shutdown_connections 2022-07-20T02:27:51.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.520+0000 7fcdec568740 1 -- 172.21.15.26:0/3271592997 wait complete. 2022-07-20T02:27:51.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- start start 2022-07-20T02:27:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 Processor -- start 2022-07-20T02:27:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d3410 0x1637150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x1637640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x1636d60 con 0x15d1330 2022-07-20T02:27:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x15a2870 con 0x15d3410 2022-07-20T02:27:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x15a0c80 con 0x15d8bc0 2022-07-20T02:27:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 10 client.0 ll_register_callbacks cb 0x159faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:27:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37288/0 (socket says 172.21.15.26:37288) 2022-07-20T02:27:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 -- 172.21.15.26:0/547865048 learned_addr learned my addr 172.21.15.26:0/547865048 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:27:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcde08ae700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x1637640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d3410 0x1637150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 -- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 msgr2=0x1637640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x1637640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 -- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d3410 msgr2=0x1637150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:27:51.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d3410 0x1637150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:27:51.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 -- 172.21.15.26:0/547865048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1548030 con 0x15d1330 2022-07-20T02:27:51.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcde08ae700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x1637640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:27:51.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcddbfff700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fcdcc026000 tx=0x7fcdcc02b000).ready entity=mon.0 client_cookie=32944afb013cd4bc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:27:51.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdbbfff700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:27:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdcc01f040 con 0x15d1330 2022-07-20T02:27:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcdcc03adc0 con 0x15d1330 2022-07-20T02:27:51.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x15d2ae0 con 0x15d1330 2022-07-20T02:27:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdcc01ab80 con 0x15d1330 2022-07-20T02:27:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 10 client.5439 fetch_fsmap learned FSMap version 18 2022-07-20T02:27:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.524+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x1504b90 con 0x15d1330 2022-07-20T02:27:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x15d2ae0 con 0x15d1330 2022-07-20T02:27:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 18) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fcdcc02e030 con 0x15d1330 2022-07-20T02:27:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 10 client.5439 fetch_fsmap finished waiting for FSMap version 18 2022-07-20T02:27:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 10 client.5439 resolve_mds: resolved ID 'c' to GID 5241 2022-07-20T02:27:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 20 client.5439 populate_metadata read hostname 'smithi026' 2022-07-20T02:27:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x15d9680 0x163a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:27:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 4 client.5439 mds_command: new command op to 5241 tid=0[{"prefix": "cache drop"}] 2022-07-20T02:27:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 --> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x7fcdc4011e50 con 0x15d9680 2022-07-20T02:27:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x15d9680 0x163a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:27:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcdbbfff700 10 client.5439 ms_handle_connect on v2:172.21.15.26:6836/3286368975 2022-07-20T02:27:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:27:51.528+0000 7fcddb7fe700 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x15d9680 0x163a2b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=452a6c57b3b56556 server_cookie=2f8a42967e0af711 in_seq=0 out_seq=0 2022-07-20T02:28:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdbbfff700 1 -- 172.21.15.26:0/547865048 <== mds.0 v2:172.21.15.26:6836/3286368975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+380 (crc 0 0 0) 0x7fcdc4011e50 con 0x15d9680 2022-07-20T02:28:02.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdbbfff700 10 client.5439 handle_command_reply: tid=0 2022-07-20T02:28:02.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 client.5439 shutdown 2022-07-20T02:28:02.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 msgr2=0x15d9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:02.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fcdcc026000 tx=0x7fcdcc02b000).stop 2022-07-20T02:28:02.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 shutdown_connections 2022-07-20T02:28:02.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x15d8bc0 0x1637640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x15d3410 0x1637150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:6836/3286368975,v1:172.21.15.26:6837/3286368975] conn(0x15d9680 0x163a2b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 --2- 172.21.15.26:0/547865048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x15d1330 0x15d9190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 >> 172.21.15.26:0/547865048 conn(0x159d540 msgr2=0x15ccbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:02.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdbbfff700 0 client.5439 ms_handle_reset on v2:172.21.15.26:6836/3286368975 2022-07-20T02:28:02.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 shutdown_connections 2022-07-20T02:28:02.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 1 -- 172.21.15.26:0/547865048 wait complete. 2022-07-20T02:28:02.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcdec568740 20 client.5439 trim_cache size 0 max 16384 2022-07-20T02:28:02.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:28:02.747 INFO:teuthology.orchestra.run.smithi026.stdout: "client_recall": { 2022-07-20T02:28:02.747 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:28:02.748 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "(0) Success", 2022-07-20T02:28:02.748 INFO:teuthology.orchestra.run.smithi026.stdout: "recalled": 220 2022-07-20T02:28:02.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:02.750 INFO:teuthology.orchestra.run.smithi026.stdout: "flush_journal": { 2022-07-20T02:28:02.750 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:28:02.751 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "" 2022-07-20T02:28:02.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:02.752 INFO:teuthology.orchestra.run.smithi026.stdout: "trim_cache": { 2022-07-20T02:28:02.752 INFO:teuthology.orchestra.run.smithi026.stdout: "trimmed": 0 2022-07-20T02:28:02.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:02.753 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2022-07-20T02:28:02.754 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": { 2022-07-20T02:28:02.754 INFO:teuthology.orchestra.run.smithi026.stdout: "items": 19985, 2022-07-20T02:28:02.755 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes": 3099360 2022-07-20T02:28:02.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:02.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:02.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 11.196986011 2022-07-20T02:28:02.757 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:28:02.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 msgr2=0x7fcdc0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:02.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 0x7fcdc0055e40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fcdcc01a000 tx=0x7fcddc0b3db0).stop 2022-07-20T02:28:02.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 msgr2=0x7fcddc1525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:02.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fcdd4001110 tx=0x7fcdd4007af0).stop 2022-07-20T02:28:02.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 shutdown_connections 2022-07-20T02:28:02.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcddc0add30 0x7fcddc1525a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdc0053980 0x7fcdc0055e40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcddc061280 0x7fcddc1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 --2- 172.21.15.26:0/563852370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcddc060950 0x7fcddc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:02.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 >> 172.21.15.26:0/563852370 conn(0x7fcddc0b3050 msgr2=0x7fcddc060190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:02.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.724+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 shutdown_connections 2022-07-20T02:28:02.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:02.728+0000 7fcde10af700 1 -- 172.21.15.26:0/563852370 wait complete. 2022-07-20T02:28:02.770 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T02:28:02.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:02.772 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:02.774 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:28:02.830 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T02:28:02.831 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T02:28:02.832 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:28:02.857 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:28:02.858 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:02.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:02.909 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:02.909 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:02.938 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:28:02.939 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:28:02.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:28:02.942 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:02.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:28:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 Processor -- start 2022-07-20T02:28:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 -- start start 2022-07-20T02:28:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7bf80952f0 0x7f7bf8060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7bf805f840 0x7f7bf8066f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf81485b0 con 0x7f7bf80b30c0 2022-07-20T02:28:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf8095a50 con 0x7f7bf80952f0 2022-07-20T02:28:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfefb4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf814b730 con 0x7f7bf805f840 2022-07-20T02:28:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7bf80952f0 0x7f7bf8060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7bf80952f0 0x7f7bf8060570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37300/0 (socket says 172.21.15.26:37300) 2022-07-20T02:28:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/2509938622 learned_addr learned my addr 172.21.15.26:0/2509938622 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:03.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfe7b3700 1 --2- 172.21.15.26:0/2509938622 >> v2:172.21.15.120:3300/0 conn(0x7f7bf805f840 0x7f7bf8066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/2509938622 >> v2:172.21.15.120:3300/0 conn(0x7f7bf805f840 msgr2=0x7f7bf8066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 --2- 172.21.15.26:0/2509938622 >> v2:172.21.15.120:3300/0 conn(0x7f7bf805f840 0x7f7bf8066f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/2509938622 >> v1:172.21.15.120:6789/0 conn(0x7f7bf80b30c0 legacy=0x7f7bf8147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/2509938622 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf814b9d0 con 0x7f7bf80952f0 2022-07-20T02:28:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfd7b1700 1 --2- 172.21.15.26:0/2509938622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80952f0 0x7f7bf8060570 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f7bf8147680 tx=0x7f7be8009290).ready entity=mon.0 client_cookie=aecdd814dd15f591 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfcfb0700 1 -- 172.21.15.26:0/2509938622 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be800b860 con 0x7f7bf80952f0 2022-07-20T02:28:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfcfb0700 1 -- 172.21.15.26:0/2509938622 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be800b9c0 con 0x7f7bf80952f0 2022-07-20T02:28:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.092+0000 7f7bfcfb0700 1 -- 172.21.15.26:0/2509938622 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be800bb30 con 0x7f7bf80952f0 2022-07-20T02:28:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- 172.21.15.26:0/2509938622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80952f0 msgr2=0x7f7bf8060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/2509938622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80952f0 0x7f7bf8060570 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f7bf8147680 tx=0x7f7be8009290).stop 2022-07-20T02:28:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- 172.21.15.26:0/2509938622 shutdown_connections 2022-07-20T02:28:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/2509938622 >> v2:172.21.15.120:3300/0 conn(0x7f7bf805f840 0x7f7bf8066f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/2509938622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80952f0 0x7f7bf8060570 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- 172.21.15.26:0/2509938622 >> 172.21.15.26:0/2509938622 conn(0x7f7bf80b67f0 msgr2=0x7f7bf80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- 172.21.15.26:0/2509938622 shutdown_connections 2022-07-20T02:28:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- 172.21.15.26:0/2509938622 wait complete. 2022-07-20T02:28:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 Processor -- start 2022-07-20T02:28:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- start start 2022-07-20T02:28:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bf805f840 0x7f7bf80646b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 0x7f7bf8146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf809a6b0 con 0x7f7bf80b30c0 2022-07-20T02:28:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8099ad0 con 0x7f7bf80952f0 2022-07-20T02:28:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfefb4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf809a210 con 0x7f7bf805f840 2022-07-20T02:28:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfe7b3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 0x7f7bf8146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:56996/0 (socket says 172.21.15.26:56996) 2022-07-20T02:28:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/3409330448 learned_addr learned my addr 172.21.15.26:0/3409330448 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfdfb2700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bf805f840 0x7f7bf80646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bf805f840 msgr2=0x7f7bf80646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bf805f840 0x7f7bf80646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 msgr2=0x7f7bf8146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 0x7f7bf8146f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 -- 172.21.15.26:0/3409330448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be8009530 con 0x7f7bf80952f0 2022-07-20T02:28:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfe7b3700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 0x7f7bf8146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:28:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bfd7b1700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7bf8098e10 tx=0x7f7be8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.096+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be80032f0 con 0x7f7bf80952f0 2022-07-20T02:28:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8021440 con 0x7f7bf80952f0 2022-07-20T02:28:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf81473f0 con 0x7f7bf80952f0 2022-07-20T02:28:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be80217a0 con 0x7f7bf80952f0 2022-07-20T02:28:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf809bae0 con 0x7f7bf80952f0 2022-07-20T02:28:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf8099930 con 0x7f7bf80952f0 2022-07-20T02:28:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7be8027020 con 0x7f7bf80952f0 2022-07-20T02:28:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bf6ffd700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 0x7f7bd8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7be806dec0 con 0x7f7bf80952f0 2022-07-20T02:28:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bfdfb2700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 0x7f7bd8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.100+0000 7f7bfdfb2700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 0x7f7bd8055ea0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7bec00d000 tx=0x7f7bec007b70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.112+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7be8044cc0 con 0x7f7bf80952f0 2022-07-20T02:28:03.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.236+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bf8099930 con 0x7f7bf80952f0 2022-07-20T02:28:03.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bf6ffd700 1 -- 172.21.15.26:0/3409330448 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f7bf8099930 con 0x7f7bf80952f0 2022-07-20T02:28:03.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:28:03.246 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:27:33.936082+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:27:30.671084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T02:27:31.927380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:28:03.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 msgr2=0x7f7bd8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 0x7f7bd8055ea0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7bec00d000 tx=0x7f7bec007b70).stop 2022-07-20T02:28:03.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 msgr2=0x7f7bf8146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7bf8098e10 tx=0x7f7be8016000).stop 2022-07-20T02:28:03.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 shutdown_connections 2022-07-20T02:28:03.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7bf80b30c0 0x7f7bf8146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7bd80539e0 0x7f7bd8055ea0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bf80952f0 0x7f7bf8146a10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 --2- 172.21.15.26:0/3409330448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bf805f840 0x7f7bf80646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 >> 172.21.15.26:0/3409330448 conn(0x7f7bf80b67f0 msgr2=0x7f7bf80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 shutdown_connections 2022-07-20T02:28:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.240+0000 7f7bfefb4700 1 -- 172.21.15.26:0/3409330448 wait complete. 2022-07-20T02:28:03.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:27:30.671084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:27:31.927380+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:28:03.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:28:03.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.364+0000 7f3aea2bb700 1 Processor -- start 2022-07-20T02:28:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.364+0000 7f3aea2bb700 1 -- start start 2022-07-20T02:28:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40b5d50 0x7f3ae4144450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3ae40ac150 0x7f3ae4144940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae405d370 con 0x7f3ae40afe40 2022-07-20T02:28:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae40ab770 con 0x7f3ae40ac150 2022-07-20T02:28:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40aba10 con 0x7f3ae40b5d50 2022-07-20T02:28:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3ae40ac150 0x7f3ae4144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3ae40ac150 0x7f3ae4144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37312/0 (socket says 172.21.15.26:37312) 2022-07-20T02:28:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/4023667876 learned_addr learned my addr 172.21.15.26:0/4023667876 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:03.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae92b9700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40b5d50 0x7f3ae4144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/4023667876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40b5d50 msgr2=0x7f3ae4144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40b5d50 0x7f3ae4144450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/4023667876 >> v1:172.21.15.120:6789/0 conn(0x7f3ae40afe40 legacy=0x7f3ae40aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/4023667876 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae40abcb0 con 0x7f3ae40ac150 2022-07-20T02:28:03.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3ae8ab8700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40ac150 0x7f3ae4144940 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3ae40ac530 tx=0x7f3ad8006eb0).ready entity=mon.0 client_cookie=1f89d2c7a4b3a7d1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3acffff700 1 -- 172.21.15.26:0/4023667876 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ad800d570 con 0x7f3ae40ac150 2022-07-20T02:28:03.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3acffff700 1 -- 172.21.15.26:0/4023667876 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8000e90 con 0x7f3ae40ac150 2022-07-20T02:28:03.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3acffff700 1 -- 172.21.15.26:0/4023667876 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ad800d8c0 con 0x7f3ae40ac150 2022-07-20T02:28:03.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- 172.21.15.26:0/4023667876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40ac150 msgr2=0x7f3ae4144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40ac150 0x7f3ae4144940 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3ae40ac530 tx=0x7f3ad8006eb0).stop 2022-07-20T02:28:03.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- 172.21.15.26:0/4023667876 shutdown_connections 2022-07-20T02:28:03.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40ac150 0x7f3ae4144940 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/4023667876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40b5d50 0x7f3ae4144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- 172.21.15.26:0/4023667876 >> 172.21.15.26:0/4023667876 conn(0x7f3ae4060310 msgr2=0x7f3ae40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- 172.21.15.26:0/4023667876 shutdown_connections 2022-07-20T02:28:03.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- 172.21.15.26:0/4023667876 wait complete. 2022-07-20T02:28:03.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 Processor -- start 2022-07-20T02:28:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.368+0000 7f3aea2bb700 1 -- start start 2022-07-20T02:28:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40ac150 0x7f3ae40b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ae40b5d50 0x7f3ae413da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae413e500 con 0x7f3ae40afe40 2022-07-20T02:28:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae413e7a0 con 0x7f3ae40b5d50 2022-07-20T02:28:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae413ea40 con 0x7f3ae40ac150 2022-07-20T02:28:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37320/0 (socket says 172.21.15.26:37320) 2022-07-20T02:28:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/359496971 learned_addr learned my addr 172.21.15.26:0/359496971 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae92b9700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40ac150 0x7f3ae40b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae9aba700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ae40b5d50 0x7f3ae413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40ac150 msgr2=0x7f3ae40b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40ac150 0x7f3ae40b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ae40b5d50 msgr2=0x7f3ae413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ae40b5d50 0x7f3ae413da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 -- 172.21.15.26:0/359496971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad800b1e0 con 0x7f3ae40afe40 2022-07-20T02:28:03.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ae8ab8700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3ae40ac530 tx=0x7f3ad8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ad8011760 con 0x7f3ae40afe40 2022-07-20T02:28:03.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad80118c0 con 0x7f3ae40afe40 2022-07-20T02:28:03.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae413d560 con 0x7f3ae40afe40 2022-07-20T02:28:03.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ad8011c20 con 0x7f3ae40afe40 2022-07-20T02:28:03.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.372+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae40b2440 con 0x7f3ae40afe40 2022-07-20T02:28:03.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae405cff0 con 0x7f3ae40afe40 2022-07-20T02:28:03.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ad8025030 con 0x7f3ae40afe40 2022-07-20T02:28:03.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3ad6ffd700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 0x7f3ac0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3ad806d1c0 con 0x7f3ae40afe40 2022-07-20T02:28:03.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3ae92b9700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 0x7f3ac0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.376+0000 7f3ae92b9700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 0x7f3ac0055e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3ae000c000 tx=0x7f3ae0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.384+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ad8043f80 con 0x7f3ae40afe40 2022-07-20T02:28:03.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.508+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3ae40574e0 con 0x7f3ae40afe40 2022-07-20T02:28:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3ad6ffd700 1 -- 172.21.15.26:0/359496971 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v45) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3ad8023020 con 0x7f3ae40afe40 2022-07-20T02:28:03.515 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:28:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 msgr2=0x7f3ac0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 0x7f3ac0055e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3ae000c000 tx=0x7f3ae0008040).stop 2022-07-20T02:28:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 msgr2=0x7f3ae4153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3ae40ac530 tx=0x7f3ad8015000).stop 2022-07-20T02:28:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 shutdown_connections 2022-07-20T02:28:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ae40afe40 0x7f3ae4153740 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ac0053950 0x7f3ac0055e10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ae40b5d50 0x7f3ae413da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 --2- 172.21.15.26:0/359496971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ae40ac150 0x7f3ae40b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 >> 172.21.15.26:0/359496971 conn(0x7f3ae4060310 msgr2=0x7f3ae4140670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 shutdown_connections 2022-07-20T02:28:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.512+0000 7f3aea2bb700 1 -- 172.21.15.26:0/359496971 wait complete. 2022-07-20T02:28:03.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:28:03.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 Processor -- start 2022-07-20T02:28:03.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 -- start start 2022-07-20T02:28:03.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec740ac300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7405d4d0 con 0x7fec740afe40 2022-07-20T02:28:03.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740a9ff0 con 0x7fec7405c720 2022-07-20T02:28:03.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79835700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec74067e60 con 0x7fec740b1200 2022-07-20T02:28:03.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec73fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fec740afe40 0x7fec740abbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36606/0 (socket says 172.21.15.26:36606) 2022-07-20T02:28:03.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec73fff700 1 -- 172.21.15.26:0/3810687954 learned_addr learned my addr 172.21.15.26:0/3810687954 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:03.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79034700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec740ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397282882 0 0) 0x7fec7405d4d0 con 0x7fec740afe40 2022-07-20T02:28:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54003260 con 0x7fec740afe40 2022-07-20T02:28:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355934978 0 0) 0x7fec54003260 con 0x7fec740afe40 2022-07-20T02:28:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 msgr2=0x7fec740ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec740ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 >> v1:172.21.15.120:6789/0 conn(0x7fec7405c720 legacy=0x7fec74147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec74067bc0 con 0x7fec740afe40 2022-07-20T02:28:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fec5c0019e0 con 0x7fec740afe40 2022-07-20T02:28:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 >> v1:172.21.15.26:6789/0 conn(0x7fec740afe40 legacy=0x7fec740abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec54003780 0x7fec54003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec54004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec54004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74067e60 con 0x7fec54003780 2022-07-20T02:28:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5400b750 con 0x7fec540049f0 2022-07-20T02:28:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5400b8c0 con 0x7fec740b1200 2022-07-20T02:28:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec78833700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec54003780 0x7fec54003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec79034700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec54004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.656+0000 7fec73fff700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec54004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 -- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 msgr2=0x7fec54004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740b1200 0x7fec54004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 -- 172.21.15.26:0/3810687954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec54003780 msgr2=0x7fec54003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec54003780 0x7fec54003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 -- 172.21.15.26:0/3810687954 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54003260 con 0x7fec540049f0 2022-07-20T02:28:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec78833700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec54003780 0x7fec54003b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:28:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec54004e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec6c00c000 tx=0x7fec6c006df0).ready entity=mon.1 client_cookie=8db35fb63fd62980 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec6c00e830 con 0x7fec540049f0 2022-07-20T02:28:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c00e990 con 0x7fec540049f0 2022-07-20T02:28:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec737fe700 1 -- 172.21.15.26:0/3810687954 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec6c00ecf0 con 0x7fec540049f0 2022-07-20T02:28:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 msgr2=0x7fec54004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 --2- 172.21.15.26:0/3810687954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec54004e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec6c00c000 tx=0x7fec6c006df0).stop 2022-07-20T02:28:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79034700 1 -- 172.21.15.26:0/3810687954 reap_dead start 2022-07-20T02:28:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- 172.21.15.26:0/3810687954 shutdown_connections 2022-07-20T02:28:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- 172.21.15.26:0/3810687954 >> 172.21.15.26:0/3810687954 conn(0x7fec74060310 msgr2=0x7fec740b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- 172.21.15.26:0/3810687954 shutdown_connections 2022-07-20T02:28:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- 172.21.15.26:0/3810687954 wait complete. 2022-07-20T02:28:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 Processor -- start 2022-07-20T02:28:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.660+0000 7fec79835700 1 -- start start 2022-07-20T02:28:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec74062db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740ac5f0 0x7fec740aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74063ac0 con 0x7fec74066db0 2022-07-20T02:28:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74067670 con 0x7fec540049f0 2022-07-20T02:28:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740661b0 con 0x7fec740ac5f0 2022-07-20T02:28:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37342/0 (socket says 172.21.15.26:37342) 2022-07-20T02:28:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 -- 172.21.15.26:0/1634970437 learned_addr learned my addr 172.21.15.26:0/1634970437 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79034700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740ac5f0 0x7fec740aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec78833700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec74062db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 -- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740ac5f0 msgr2=0x7fec740aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740ac5f0 0x7fec740aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 -- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 msgr2=0x7fec74062db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec74062db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 -- 172.21.15.26:0/1634970437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740673d0 con 0x7fec74066db0 2022-07-20T02:28:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec78833700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec74062db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:28:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec73fff700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fec5c00d000 tx=0x7fec74067d50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec5c00fa60 con 0x7fec74066db0 2022-07-20T02:28:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74065a90 con 0x7fec74066db0 2022-07-20T02:28:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c00fbc0 con 0x7fec74066db0 2022-07-20T02:28:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74063e90 con 0x7fec74066db0 2022-07-20T02:28:03.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec5c0078d0 con 0x7fec74066db0 2022-07-20T02:28:03.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec740aa250 con 0x7fec74066db0 2022-07-20T02:28:03.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.664+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec5c01b030 con 0x7fec74066db0 2022-07-20T02:28:03.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.668+0000 7fec71ffb700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 0x7fec58055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:03.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.668+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fec5c06bf10 con 0x7fec74066db0 2022-07-20T02:28:03.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.668+0000 7fec78833700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 0x7fec58055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:03.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.668+0000 7fec78833700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 0x7fec58055f30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fec64009000 tx=0x7fec64009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:03.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.672+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec5c042ce0 con 0x7fec74066db0 2022-07-20T02:28:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.796+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fec740574e0 con 0x7fec74066db0 2022-07-20T02:28:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.796+0000 7fec71ffb700 1 -- 172.21.15.26:0/1634970437 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v45) v1 ==== 92+0+10 (secure 0 0 0) 0x7fec5c019030 con 0x7fec74066db0 2022-07-20T02:28:03.841 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:28:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 msgr2=0x7fec58055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 0x7fec58055f30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fec64009000 tx=0x7fec64009040).stop 2022-07-20T02:28:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 msgr2=0x7fec740ac100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fec5c00d000 tx=0x7fec74067d50).stop 2022-07-20T02:28:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 shutdown_connections 2022-07-20T02:28:03.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fec74066db0 0x7fec740ac100 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fec58053a70 0x7fec58055f30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec740ac5f0 0x7fec740aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 --2- 172.21.15.26:0/1634970437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec540049f0 0x7fec74062db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:03.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 >> 172.21.15.26:0/1634970437 conn(0x7fec74060310 msgr2=0x7fec74149550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:03.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 shutdown_connections 2022-07-20T02:28:03.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.800+0000 7fec79835700 1 -- 172.21.15.26:0/1634970437 wait complete. 2022-07-20T02:28:03.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:28:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 Processor -- start 2022-07-20T02:28:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- start start 2022-07-20T02:28:04.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f21981404c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21981409b0 0x7f2198140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21980b2f10 con 0x7f21980b2b30 2022-07-20T02:28:04.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21980ad650 con 0x7f21981409b0 2022-07-20T02:28:04.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21980a9230 con 0x7f21980b10f0 2022-07-20T02:28:04.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21981409b0 0x7f2198140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f218ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f21981404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21981409b0 0x7f2198140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37350/0 (socket says 172.21.15.26:37350) 2022-07-20T02:28:04.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 -- 172.21.15.26:0/306820485 learned_addr learned my addr 172.21.15.26:0/306820485 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:04.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f2196ffd700 1 --1- 172.21.15.26:0/306820485 >> v1:172.21.15.120:6789/0 conn(0x7f21980b2b30 0x7f21981416e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:50838/0 (socket says 172.21.15.26:50838) 2022-07-20T02:28:04.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 -- 172.21.15.26:0/306820485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 msgr2=0x7f21981404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:04.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 --2- 172.21.15.26:0/306820485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f21981404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 -- 172.21.15.26:0/306820485 >> v1:172.21.15.120:6789/0 conn(0x7f21980b2b30 legacy=0x7f21981416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 -- 172.21.15.26:0/306820485 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198060fb0 con 0x7f21981409b0 2022-07-20T02:28:04.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21977fe700 1 --2- 172.21.15.26:0/306820485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f2198140dd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f21900032f0 tx=0x7f21900090f0).ready entity=mon.0 client_cookie=46a233b96b5e7e1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21967fc700 1 -- 172.21.15.26:0/306820485 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f219000cc90 con 0x7f21981409b0 2022-07-20T02:28:04.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21967fc700 1 -- 172.21.15.26:0/306820485 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219000cdf0 con 0x7f21981409b0 2022-07-20T02:28:04.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f21967fc700 1 -- 172.21.15.26:0/306820485 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f219000f650 con 0x7f21981409b0 2022-07-20T02:28:04.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- 172.21.15.26:0/306820485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 msgr2=0x7f2198140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 --2- 172.21.15.26:0/306820485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f2198140dd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f21900032f0 tx=0x7f21900090f0).stop 2022-07-20T02:28:04.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- 172.21.15.26:0/306820485 shutdown_connections 2022-07-20T02:28:04.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 --2- 172.21.15.26:0/306820485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f2198140dd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 --2- 172.21.15.26:0/306820485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f21981404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- 172.21.15.26:0/306820485 >> 172.21.15.26:0/306820485 conn(0x7f21980b6110 msgr2=0x7f21980b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- 172.21.15.26:0/306820485 shutdown_connections 2022-07-20T02:28:04.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.980+0000 7f219d450700 1 -- 172.21.15.26:0/306820485 wait complete. 2022-07-20T02:28:04.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 Processor -- start 2022-07-20T02:28:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 -- start start 2022-07-20T02:28:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f219813f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 0x7f21980ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219814e100 con 0x7f21981409b0 2022-07-20T02:28:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21980afe70 con 0x7f21980b2b30 2022-07-20T02:28:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f219d450700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21981404b0 con 0x7f21980b10f0 2022-07-20T02:28:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37356/0 (socket says 172.21.15.26:37356) 2022-07-20T02:28:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 -- 172.21.15.26:0/3487542351 learned_addr learned my addr 172.21.15.26:0/3487542351 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f2196ffd700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f219813f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f218ffff700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 0x7f21980ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 -- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 msgr2=0x7f219813f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f219813f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 -- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 msgr2=0x7f21980ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 0x7f21980ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f21977fe700 1 -- 172.21.15.26:0/3487542351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219000b5e0 con 0x7f21981409b0 2022-07-20T02:28:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f2196ffd700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f219813f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:28:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.984+0000 7f218ffff700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 0x7f21980ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:28:04.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f21977fe700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f21900032f0 tx=0x7f2190016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21900197f0 con 0x7f21981409b0 2022-07-20T02:28:04.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2190019950 con 0x7f21981409b0 2022-07-20T02:28:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219814e2f0 con 0x7f21981409b0 2022-07-20T02:28:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2190019cb0 con 0x7f21981409b0 2022-07-20T02:28:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21980aa410 con 0x7f21981409b0 2022-07-20T02:28:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219814f080 con 0x7f21981409b0 2022-07-20T02:28:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f219000e3c0 con 0x7f21981409b0 2022-07-20T02:28:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 0x7f217c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f219006ded0 con 0x7f21981409b0 2022-07-20T02:28:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.988+0000 7f2196ffd700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 0x7f217c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.992+0000 7f2196ffd700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 0x7f217c055f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f218800b000 tx=0x7f2188000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:03.996+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2190044b50 con 0x7f21981409b0 2022-07-20T02:28:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.124+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21980574e0 con 0x7f21981409b0 2022-07-20T02:28:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.124+0000 7f2194ff9700 1 -- 172.21.15.26:0/3487542351 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2190048170 con 0x7f21981409b0 2022-07-20T02:28:04.136 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:28:04.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 msgr2=0x7f217c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 0x7f217c055f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f218800b000 tx=0x7f2188000bc0).stop 2022-07-20T02:28:04.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 msgr2=0x7f219814e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f21900032f0 tx=0x7f2190016000).stop 2022-07-20T02:28:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 shutdown_connections 2022-07-20T02:28:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21981409b0 0x7f219814e9f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f217c053a70 0x7f217c055f30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21980b2b30 0x7f21980ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 --2- 172.21.15.26:0/3487542351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21980b10f0 0x7f219813f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 >> 172.21.15.26:0/3487542351 conn(0x7f21980b6110 msgr2=0x7f2198145a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 shutdown_connections 2022-07-20T02:28:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.128+0000 7f219d450700 1 -- 172.21.15.26:0/3487542351 wait complete. 2022-07-20T02:28:04.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:28:04.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.332+0000 7fae3eef8700 1 Processor -- start 2022-07-20T02:28:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.332+0000 7fae3eef8700 1 -- start start 2022-07-20T02:28:04.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 0x7fae38098eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae380b35b0 con 0x7fae3805fef0 2022-07-20T02:28:04.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae380b1f00 con 0x7fae38097d80 2022-07-20T02:28:04.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae38098370 con 0x7fae380b1b20 2022-07-20T02:28:04.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3e6f7700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fae3805fef0 0x7fae3809b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36642/0 (socket says 172.21.15.26:36642) 2022-07-20T02:28:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3e6f7700 1 -- 172.21.15.26:0/1734424590 learned_addr learned my addr 172.21.15.26:0/1734424590 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3def6700 1 --2- 172.21.15.26:0/1734424590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 0x7fae38098eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955100130 0 0) 0x7fae380b35b0 con 0x7fae3805fef0 2022-07-20T02:28:04.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae20003260 con 0x7fae3805fef0 2022-07-20T02:28:04.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882261553 0 0) 0x7fae380b1f00 con 0x7fae38097d80 2022-07-20T02:28:04.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae380b35b0 con 0x7fae38097d80 2022-07-20T02:28:04.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 166297172 0 0) 0x7fae20003260 con 0x7fae3805fef0 2022-07-20T02:28:04.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3def6700 1 -- 172.21.15.26:0/1734424590 >> v1:172.21.15.120:6789/0 conn(0x7fae38097d80 legacy=0x7fae380b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3def6700 1 -- 172.21.15.26:0/1734424590 >> v1:172.21.15.26:6789/0 conn(0x7fae3805fef0 legacy=0x7fae3809b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3def6700 1 -- 172.21.15.26:0/1734424590 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38098610 con 0x7fae380b1b20 2022-07-20T02:28:04.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3def6700 1 --2- 172.21.15.26:0/1734424590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 0x7fae38098eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fae30001490 tx=0x7fae34006df0).ready entity=mon.2 client_cookie=d075a860772f73b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae300019e0 con 0x7fae380b1b20 2022-07-20T02:28:04.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34000e90 con 0x7fae380b1b20 2022-07-20T02:28:04.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3cef4700 1 -- 172.21.15.26:0/1734424590 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae34012460 con 0x7fae380b1b20 2022-07-20T02:28:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 -- 172.21.15.26:0/1734424590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 msgr2=0x7fae38098eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.336+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1734424590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 0x7fae38098eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fae30001490 tx=0x7fae34006df0).stop 2022-07-20T02:28:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 -- 172.21.15.26:0/1734424590 shutdown_connections 2022-07-20T02:28:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1734424590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae380b1b20 0x7fae38098eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 -- 172.21.15.26:0/1734424590 >> 172.21.15.26:0/1734424590 conn(0x7fae38063030 msgr2=0x7fae38063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 -- 172.21.15.26:0/1734424590 shutdown_connections 2022-07-20T02:28:04.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 -- 172.21.15.26:0/1734424590 wait complete. 2022-07-20T02:28:04.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 Processor -- start 2022-07-20T02:28:04.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.340+0000 7fae3eef8700 1 -- start start 2022-07-20T02:28:04.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b1b20 0x7fae38152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae381473f0 0x7fae38146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38152850 con 0x7fae3805fef0 2022-07-20T02:28:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae380ad420 con 0x7fae380b1b20 2022-07-20T02:28:04.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae380ad6c0 con 0x7fae381473f0 2022-07-20T02:28:04.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37366/0 (socket says 172.21.15.26:37366) 2022-07-20T02:28:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 -- 172.21.15.26:0/1848341659 learned_addr learned my addr 172.21.15.26:0/1848341659 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3e6f7700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae381473f0 0x7fae38146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3d6f5700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b1b20 0x7fae38152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 -- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae381473f0 msgr2=0x7fae38146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae381473f0 0x7fae38146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 -- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b1b20 msgr2=0x7fae38152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b1b20 0x7fae38152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 -- 172.21.15.26:0/1848341659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3400b180 con 0x7fae3805fef0 2022-07-20T02:28:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3def6700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fae30001490 tx=0x7fae34014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3400d920 con 0x7fae3805fef0 2022-07-20T02:28:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3400da80 con 0x7fae3805fef0 2022-07-20T02:28:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3400dde0 con 0x7fae3805fef0 2022-07-20T02:28:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae380ad960 con 0x7fae3805fef0 2022-07-20T02:28:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae38147030 con 0x7fae3805fef0 2022-07-20T02:28:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.344+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae380ac9c0 con 0x7fae3805fef0 2022-07-20T02:28:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.348+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae3401c070 con 0x7fae3805fef0 2022-07-20T02:28:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.348+0000 7fae2affd700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 0x7fae18055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.348+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fae3406c9a0 con 0x7fae3805fef0 2022-07-20T02:28:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.348+0000 7fae3d6f5700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 0x7fae18055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.348+0000 7fae3d6f5700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 0x7fae18055f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fae2c00d000 tx=0x7fae2c007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.356+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae380ac9c0 con 0x7fae3805fef0 2022-07-20T02:28:04.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae380ac9c0 con 0x7fae3805fef0 2022-07-20T02:28:04.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae2affd700 1 -- 172.21.15.26:0/1848341659 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+10 (secure 0 0 0) 0x7fae380ac9c0 con 0x7fae3805fef0 2022-07-20T02:28:04.493 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:28:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 msgr2=0x7fae18055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 0x7fae18055f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fae2c00d000 tx=0x7fae2c007ad0).stop 2022-07-20T02:28:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 msgr2=0x7fae38145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fae30001490 tx=0x7fae34014000).stop 2022-07-20T02:28:04.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 shutdown_connections 2022-07-20T02:28:04.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae18053a70 0x7fae18055f30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae381473f0 0x7fae38146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae380b1b20 0x7fae38152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 --2- 172.21.15.26:0/1848341659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae3805fef0 0x7fae38145b80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 >> 172.21.15.26:0/1848341659 conn(0x7fae38063030 msgr2=0x7fae380b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 shutdown_connections 2022-07-20T02:28:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.476+0000 7fae3eef8700 1 -- 172.21.15.26:0/1848341659 wait complete. 2022-07-20T02:28:04.534 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:28:04.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:28:04.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 Processor -- start 2022-07-20T02:28:04.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 -- start start 2022-07-20T02:28:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4f18147be0 0x7f4f180aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f180af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f180b62d0 con 0x7f4f180b5ef0 2022-07-20T02:28:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1814aac0 con 0x7f4f18147be0 2022-07-20T02:28:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1814ad60 con 0x7f4f180a7fc0 2022-07-20T02:28:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4f180b5ef0 0x7f4f181474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36656/0 (socket says 172.21.15.26:36656) 2022-07-20T02:28:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1659c700 1 -- 172.21.15.26:0/3904515993 learned_addr learned my addr 172.21.15.26:0/3904515993 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f16d9d700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f180af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 --2- 172.21.15.26:0/3904515993 >> v2:172.21.15.120:3300/0 conn(0x7f4f18147be0 0x7f4f180aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1559a700 1 -- 172.21.15.26:0/3904515993 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538994440 0 0) 0x7f4f180b62d0 con 0x7f4f180b5ef0 2022-07-20T02:28:04.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1559a700 1 -- 172.21.15.26:0/3904515993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4efc003260 con 0x7f4f180b5ef0 2022-07-20T02:28:04.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 -- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 msgr2=0x7f4f180af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:04.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f180af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 -- 172.21.15.26:0/3904515993 >> v1:172.21.15.26:6789/0 conn(0x7f4f180b5ef0 legacy=0x7f4f181474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 -- 172.21.15.26:0/3904515993 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1814b000 con 0x7f4f18147be0 2022-07-20T02:28:04.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f15d9b700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147be0 0x7f4f180aee40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4f08002f70 tx=0x7f4f08006df0).ready entity=mon.1 client_cookie=106a04b5ff0ed119 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1559a700 1 -- 172.21.15.26:0/3904515993 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0800d4a0 con 0x7f4f18147be0 2022-07-20T02:28:04.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1559a700 1 -- 172.21.15.26:0/3904515993 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08000e90 con 0x7f4f18147be0 2022-07-20T02:28:04.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.644+0000 7f4f1559a700 1 -- 172.21.15.26:0/3904515993 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0800d7f0 con 0x7f4f18147be0 2022-07-20T02:28:04.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147be0 msgr2=0x7f4f180aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147be0 0x7f4f180aee40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4f08002f70 tx=0x7f4f08006df0).stop 2022-07-20T02:28:04.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- 172.21.15.26:0/3904515993 shutdown_connections 2022-07-20T02:28:04.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f180af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3904515993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147be0 0x7f4f180aee40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- 172.21.15.26:0/3904515993 >> 172.21.15.26:0/3904515993 conn(0x7f4f18060310 msgr2=0x7f4f180b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- 172.21.15.26:0/3904515993 shutdown_connections 2022-07-20T02:28:04.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- 172.21.15.26:0/3904515993 wait complete. 2022-07-20T02:28:04.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 Processor -- start 2022-07-20T02:28:04.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- start start 2022-07-20T02:28:04.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f1814e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f180b5ef0 0x7f4f18146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180ab430 con 0x7f4f18147be0 2022-07-20T02:28:04.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180ab7b0 con 0x7f4f180b5ef0 2022-07-20T02:28:04.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180aba50 con 0x7f4f180a7fc0 2022-07-20T02:28:04.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37384/0 (socket says 172.21.15.26:37384) 2022-07-20T02:28:04.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 -- 172.21.15.26:0/3521033813 learned_addr learned my addr 172.21.15.26:0/3521033813 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:04.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f1659c700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f1814e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f15d9b700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f180b5ef0 0x7f4f18146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 -- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 msgr2=0x7f4f1814e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f1814e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 -- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f180b5ef0 msgr2=0x7f4f18146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f180b5ef0 0x7f4f18146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.648+0000 7f4f16d9d700 1 -- 172.21.15.26:0/3521033813 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0800b180 con 0x7f4f18147be0 2022-07-20T02:28:04.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f16d9d700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4f10006e00 tx=0x7f4f100091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000c640 con 0x7f4f18147be0 2022-07-20T02:28:04.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000c7a0 con 0x7f4f18147be0 2022-07-20T02:28:04.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f180abcf0 con 0x7f4f18147be0 2022-07-20T02:28:04.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000cb00 con 0x7f4f18147be0 2022-07-20T02:28:04.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f180aa160 con 0x7f4f18147be0 2022-07-20T02:28:04.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1814b2a0 con 0x7f4f18147be0 2022-07-20T02:28:04.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f10022070 con 0x7f4f18147be0 2022-07-20T02:28:04.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 0x7f4ef4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:04.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4f1006c700 con 0x7f4f18147be0 2022-07-20T02:28:04.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.652+0000 7f4f1659c700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 0x7f4ef4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:04.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.656+0000 7f4f1659c700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 0x7f4ef4055f30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f4f0c00d000 tx=0x7f4f0c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:04.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.660+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f100434c0 con 0x7f4f18147be0 2022-07-20T02:28:04.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f180574e0 con 0x7f4f18147be0 2022-07-20T02:28:04.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f077fe700 1 -- 172.21.15.26:0/3521033813 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f4f10046ae0 con 0x7f4f18147be0 2022-07-20T02:28:04.827 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:28:04.827 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:28:04.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 msgr2=0x7f4ef4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 0x7f4ef4055f30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f4f0c00d000 tx=0x7f4f0c009040).stop 2022-07-20T02:28:04.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 msgr2=0x7f4f18147410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:04.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4f10006e00 tx=0x7f4f100091d0).stop 2022-07-20T02:28:04.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 shutdown_connections 2022-07-20T02:28:04.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18147be0 0x7f4f18147410 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ef4053a70 0x7f4ef4055f30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f180b5ef0 0x7f4f18146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 --2- 172.21.15.26:0/3521033813 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180a7fc0 0x7f4f1814e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:04.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 >> 172.21.15.26:0/3521033813 conn(0x7f4f18060310 msgr2=0x7f4f180b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:04.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 shutdown_connections 2022-07-20T02:28:04.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:04.824+0000 7f4f1759e700 1 -- 172.21.15.26:0/3521033813 wait complete. 2022-07-20T02:28:04.837 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:28:05.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:27:32.932112+0000', 'modified': '2022-07-20T02:27:33.936408+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'c', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3286368975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3286368975}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3286368975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:28:05.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:28:05.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:28:05.323 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":32.365066466000002,"mdsmap_epoch":18,"osdmap_epoch":45,"osdmap_epoch_barrier":44,"uptime":37.593066774} 2022-07-20T02:28:05.324 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 45, "osdmap_epoch_barrier": 44, "rank_uptime": 32.365066466, "state": "up:active", "uptime": 37.593066774, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:28:05.325 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:28:05.331 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:28:05.332 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:28:05.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:05.334 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:05.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:28:05.384 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:28:05.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:28:05.407 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:28:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:28:05.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:05.412 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:28:05.463 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:28:05.464 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:05.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:28:05.532 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:28:05.533 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:28:05.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:28:05.540+0000 7f209968f080 -1 init, newargv = 0x55d59b681d60 newargc=9 2022-07-20T02:28:05.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22483]: starting ceph client 2022-07-20T02:28:05.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22483]: starting fuse 2022-07-20T02:28:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:28:05.605 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:28:05.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:05.630 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:28:05.692 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:28:05.693 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:28:05.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:05.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:28:05.696 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:28:05.697 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:28:05.698 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:28:05.699 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:28:05.699 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:28:05.707 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:28:05.708 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:28:05.711 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:28:05.712 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:28:05.713 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:28:05.713 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:28:05.715 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:28:05.715 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:28:05.716 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:28:05.718 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:28:05.722 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:28:05.723 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:28:05.724 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:28:05.725 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:28:05.726 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:28:05.727 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:28:05.728 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:28:05.803 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22483.asok 2022-07-20T02:28:05.804 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22483.asok 2022-07-20T02:28:05.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:05.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22483.asok status 2022-07-20T02:28:05.973 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:28:05.973 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:28:05.974 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:28:05.974 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:28:05.975 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:28:05.975 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:28:05.976 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:28:05.977 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22483", 2022-07-20T02:28:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:28:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:28:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:28:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:28:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 5484, 2022-07-20T02:28:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:28:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:28:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:28:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 5484 2022-07-20T02:28:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:28:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:28:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:28:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:28:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3251866390 2022-07-20T02:28:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:28:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:28:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:28:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:28:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:28:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3251866390 2022-07-20T02:28:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:28:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.5484 172.21.15.159:0/3251866390", 2022-07-20T02:28:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/3251866390", 2022-07-20T02:28:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:28:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 18, 2022-07-20T02:28:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 45, 2022-07-20T02:28:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:28:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:28:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:28:06.037 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:28:06.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:06.038 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:06.047 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:06.096 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:28:06.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:28:06.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:06.097 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:16.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:16.492 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:16.526 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:16.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:46.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:46.589 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:46.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:46.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:28:52.718 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:28:52.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:28:52.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:28:52.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:28:52.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:28:52.813 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:28:52.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:52.815 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:52.816 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:52.856 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:28:52.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:28:52.857 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:28:52.858 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:28:52.859 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:52.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22483]: fuse finished with error 0 and tester_r 0 2022-07-20T02:28:52.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:52.921 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:52.921 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:52.971 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:28:52.971 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:28:53.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:28:53.008 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:28:53.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:28:53.026 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:28:53.027 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:28:53.035 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:28:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:28:53.039 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:28:53.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:28:53.041 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:28:53.052 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:28:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:28:53.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:28:53.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 Processor -- start 2022-07-20T02:28:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- start start 2022-07-20T02:28:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21880b2920 0x7f2188147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2188067190 0x7f2188148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880b1bc0 con 0x7f2188061050 2022-07-20T02:28:53.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218814b040 con 0x7f21880b2920 2022-07-20T02:28:53.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21880a8860 con 0x7f2188067190 2022-07-20T02:28:53.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21880b2920 0x7f2188147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21880b2920 0x7f2188147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37388/0 (socket says 172.21.15.26:37388) 2022-07-20T02:28:53.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 -- 172.21.15.26:0/913574699 learned_addr learned my addr 172.21.15.26:0/913574699 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:53.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e9bf700 1 --2- 172.21.15.26:0/913574699 >> v2:172.21.15.120:3300/0 conn(0x7f2188067190 0x7f2188148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 -- 172.21.15.26:0/913574699 >> v2:172.21.15.120:3300/0 conn(0x7f2188067190 msgr2=0x7f2188148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 --2- 172.21.15.26:0/913574699 >> v2:172.21.15.120:3300/0 conn(0x7f2188067190 0x7f2188148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 -- 172.21.15.26:0/913574699 >> v1:172.21.15.26:6789/0 conn(0x7f2188061050 legacy=0x7f2188066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 -- 172.21.15.26:0/913574699 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188147700 con 0x7f21880b2920 2022-07-20T02:28:53.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218e1be700 1 --2- 172.21.15.26:0/913574699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f2188147d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2184002f70 tx=0x7f2184006eb0).ready entity=mon.0 client_cookie=68bcf3aef752b144 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218d1bc700 1 -- 172.21.15.26:0/913574699 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218400d610 con 0x7f21880b2920 2022-07-20T02:28:53.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218d1bc700 1 -- 172.21.15.26:0/913574699 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2184000e90 con 0x7f21880b2920 2022-07-20T02:28:53.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f218d1bc700 1 -- 172.21.15.26:0/913574699 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218400d9a0 con 0x7f21880b2920 2022-07-20T02:28:53.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- 172.21.15.26:0/913574699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 msgr2=0x7f2188147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 --2- 172.21.15.26:0/913574699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f2188147d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2184002f70 tx=0x7f2184006eb0).stop 2022-07-20T02:28:53.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- 172.21.15.26:0/913574699 shutdown_connections 2022-07-20T02:28:53.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 --2- 172.21.15.26:0/913574699 >> v2:172.21.15.120:3300/0 conn(0x7f2188067190 0x7f2188148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 --2- 172.21.15.26:0/913574699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f2188147d90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- 172.21.15.26:0/913574699 >> 172.21.15.26:0/913574699 conn(0x7f21880b60a0 msgr2=0x7f21880b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- 172.21.15.26:0/913574699 shutdown_connections 2022-07-20T02:28:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.172+0000 7f2190422700 1 -- 172.21.15.26:0/913574699 wait complete. 2022-07-20T02:28:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 Processor -- start 2022-07-20T02:28:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- start start 2022-07-20T02:28:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 0x7f21880b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2188067190 0x7f21880632a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880651b0 con 0x7f21880b2920 2022-07-20T02:28:53.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2188065450 con 0x7f2188061050 2022-07-20T02:28:53.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218815a060 con 0x7f2188067190 2022-07-20T02:28:53.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37398/0 (socket says 172.21.15.26:37398) 2022-07-20T02:28:53.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 -- 172.21.15.26:0/208612389 learned_addr learned my addr 172.21.15.26:0/208612389 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:53.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e1be700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 0x7f21880b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218d9bd700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2188067190 0x7f21880632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 -- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2188067190 msgr2=0x7f21880632a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2188067190 0x7f21880632a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 -- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 msgr2=0x7f21880b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 0x7f21880b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 -- 172.21.15.26:0/208612389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218400b1e0 con 0x7f21880b2920 2022-07-20T02:28:53.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e1be700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 0x7f21880b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:28:53.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f218e9bf700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f218000a000 tx=0x7f2180008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2180013620 con 0x7f21880b2920 2022-07-20T02:28:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180013780 con 0x7f21880b2920 2022-07-20T02:28:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2188063790 con 0x7f21880b2920 2022-07-20T02:28:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2180013ae0 con 0x7f21880b2920 2022-07-20T02:28:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.176+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21881569f0 con 0x7f21880b2920 2022-07-20T02:28:53.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218815a220 con 0x7f21880b2920 2022-07-20T02:28:53.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2180013c40 con 0x7f21880b2920 2022-07-20T02:28:53.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f21777fe700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 0x7f2168055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f218006bff0 con 0x7f21880b2920 2022-07-20T02:28:53.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f218e1be700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 0x7f2168055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.180+0000 7f218e1be700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 0x7f2168055f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f218401a000 tx=0x7f2184015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.188+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2180042cb0 con 0x7f21880b2920 2022-07-20T02:28:53.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.316+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f218814ada0 con 0x7f21880b2920 2022-07-20T02:28:53.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.320+0000 7f21777fe700 1 -- 172.21.15.26:0/208612389 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v45) v1 ==== 81+0+34085 (secure 0 0 0) 0x7f21800462d0 con 0x7f21880b2920 2022-07-20T02:28:53.324 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:28:53.325 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:28:53.325 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 45, 2022-07-20T02:28:53.326 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:28:53.326 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:28:53.327 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:27:33.936082+0000", 2022-07-20T02:28:53.327 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:28:53.328 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:28:53.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:28:53.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:28:53.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:28:53.330 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:28:53.331 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:28:53.331 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:28:53.332 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:28:53.334 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:28:53.334 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:28:53.335 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:28:53.335 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:28:53.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:28:53.336 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:28:53.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 8, 2022-07-20T02:28:53.337 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:28:53.338 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:28:53.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:28:53.339 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:28:53.339 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:28:53.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:28:53.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:28:53.341 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:28:53.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:28:53.343 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:28:53.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:28:53.345 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:28:53.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:28:53.346 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:28:53.347 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:28:53.347 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:28:53.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:28:53.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:28:53.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:28:53.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:28:53.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:28:53.350 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:28:53.350 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:28:53.351 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:28:53.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:28:53.352 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:28:53.352 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:28:53.353 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:28:53.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.355 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:28:53.356 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:28:53.356 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:28:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:28:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:28:53.358 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:28:53.358 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:28:53.358 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:28:53.371 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:28:53.371 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:28:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:28:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:28:53.372 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:28:53.373 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:28:53.373 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:28:53.376 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:28:53.376 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:28:53.377 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:28:53.377 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:28:53.378 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:28:53.379 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:28:53.379 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:28:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:28:53.381 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:28:53.382 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:28:53.382 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:28:53.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:28:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:28:53.384 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:28:53.384 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:28:53.385 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:28:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:28:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:28:53.388 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:28:53.388 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:28:53.389 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:28:53.390 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:28:53.390 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:28:53.391 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:28:53.392 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:28:53.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.393 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:28:53.393 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:28:53.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.395 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:28:53.395 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:28:53.396 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:28:53.397 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:28:53.398 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:28:53.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:28:53.399 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:28:53.400 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:28:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:28:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:28:53.402 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:28:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:28:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:28:53.404 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:28:53.404 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:28:53.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:28:53.416 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:28:53.416 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:28:53.417 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:28:53.432 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:28:53.433 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:28:53.447 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:28:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:28:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:28:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:28:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:28:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:28:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:28:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:28:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:28:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:28:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:28:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:28:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:28:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:28:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:28:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:28:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:28:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:28:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:28:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:28:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:28:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:28:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:28:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:28:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:28:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:28:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:28:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:28:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:28:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:28:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:28:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:28:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:28:53.456 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:28:53.457 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:28:53.457 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:28:53.458 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:28:53.459 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:28:53.459 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:28:53.459 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:28:53.459 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:28:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:28:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:28:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 7, 2022-07-20T02:28:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:28:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:27:30.671084+0000", 2022-07-20T02:28:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:28:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:28:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:28:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:28:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:28:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:28:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:28:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:28:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:28:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:28:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:28:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:28:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:28:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:28:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:28:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:28:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:28:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:28:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:28:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:28:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "44", 2022-07-20T02:28:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:28:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:28:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:28:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:28:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:28:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:28:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:28:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:28:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:28:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:28:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:28:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:28:53.470 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:28:53.470 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:28:53.470 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:28:53.470 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:28:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:28:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:28:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:28:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:28:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:28:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:28:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:28:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:28:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:28:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:28:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:28:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:28:53.474 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:28:53.474 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:28:53.474 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:28:53.474 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:28:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:28:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:28:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:28:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:28:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:28:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:28:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:28:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:28:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:28:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:28:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:28:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:28:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.479 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 8, 2022-07-20T02:28:53.479 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:28:53.479 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:27:31.927380+0000", 2022-07-20T02:28:53.479 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:28:53.480 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:28:53.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:28:53.480 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:28:53.480 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:28:53.481 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:28:53.481 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:28:53.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:28:53.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:28:53.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:28:53.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:28:53.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:28:53.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:28:53.482 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:28:53.483 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:28:53.483 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:28:53.483 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:28:53.483 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:28:53.483 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:28:53.484 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:28:53.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.484 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "45", 2022-07-20T02:28:53.484 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:28:53.485 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:28:53.485 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:28:53.485 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:28:53.485 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:28:53.485 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:28:53.486 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:28:53.486 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:28:53.486 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:28:53.486 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:28:53.487 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:28:53.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:28:53.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:28:53.487 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:28:53.487 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:28:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:28:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:28:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:28:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:28:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:28:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:28:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:28:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:28:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:28:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:28:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:28:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:28:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:28:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:28:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:28:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:28:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:28:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:28:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:28:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:28:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:28:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:28:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:28:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:28:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:28:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:28:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:28:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:28:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:28:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:28:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:28:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:28:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.506 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:28:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:28:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:28:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:28:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:28:53.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:28:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:28:53.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:28:53.526 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:28:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:28:53.527 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:28:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.528 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.530 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:28:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:28:53.531 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.532 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.533 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:28:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:28:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:28:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:28:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:28:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:28:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:28:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:28:53.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:28:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:28:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:28:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:28:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:28:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.548 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:28:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:28:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.550 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.551 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:28:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:28:53.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:28:53.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.557 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:28:53.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:28:53.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.564 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:28:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:28:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:28:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:28:53.575 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:28:53.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:28:53.587 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:28:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:28:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:28:53.588 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.589 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.591 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:28:53.593 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:28:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.594 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.595 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.596 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.598 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:28:53.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:28:53.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.606 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:28:53.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:28:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:28:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:28:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:28:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:28:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:28:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:28:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:28:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:28:53.616 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:28:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:28:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:28:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.620 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:28:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:28:53.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:28:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:28:53.626 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.627 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:28:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:28:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:28:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:28:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:28:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:28:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:28:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:28:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:28:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:28:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:28:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.638 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.639 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:28:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:28:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:28:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:28:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:28:53.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:28:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:28:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:28:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:28:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:28:53.653 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:28:53.654 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:28:53.654 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:28:53.654 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.654 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:28:53.656 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:28:53.656 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.656 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.657 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.657 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.659 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:28:53.659 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.660 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:28:53.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:28:53.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.680 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:28:53.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:28:53.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.686 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:28:53.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:28:53.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.691 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:28:53.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:28:53.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:28:53.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:28:53.699 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:28:53.699 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:28:53.700 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:28:53.701 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.701 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.702 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.704 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:28:53.704 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:28:53.704 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:28:53.705 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:28:53.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:28:53.706 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:28:53.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:28:53.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:28:53.707 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:28:53.707 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 42, 2022-07-20T02:28:53.708 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:28:53.708 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:28:53.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:28:53.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:28:53.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:28:53.714 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.715 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:28:53.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:28:53.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:28:53.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.723 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:28:53.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:28:53.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:28:53.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.729 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:28:53.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:28:53.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:28:53.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:28:53.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:28:53.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:28:53.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:28:53.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:28:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:28:53.748 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:28:53.748 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:28:53.749 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:28:53.749 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:28:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:28:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:28:53.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.752 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:28:53.753 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:28:53.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.754 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:28:53.755 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.755 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.756 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.756 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.757 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.757 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:28:53.758 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:28:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.761 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:28:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:28:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:28:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:28:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.773 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.774 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:28:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:28:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.782 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.782 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.783 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.783 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:28:53.783 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.784 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:28:53.784 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.784 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.784 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.785 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.785 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.785 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:28:53.785 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:28:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:28:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:28:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:28:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:28:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:28:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:28:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:28:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:28:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:28:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:28:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:28:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:28:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:28:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:28:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:28:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:28:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:28:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:28:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:28:53.795 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:28:53.795 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:28:53.795 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:28:53.795 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:28:53.796 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:28:53.796 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:28:53.796 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:28:53.796 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:28:53.797 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:28:53.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.797 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:28:53.798 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:28:53.799 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:28:53.799 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:28:53.799 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:28:53.799 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:28:53.800 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:28:53.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:28:53.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:28:53.800 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:28:53.800 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:28:53.801 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:28:53.801 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:28:53.801 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:28:53.801 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:28:53.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 msgr2=0x7f2168055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 0x7f2168055f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f218401a000 tx=0x7f2184015000).stop 2022-07-20T02:28:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 msgr2=0x7f218815a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f218000a000 tx=0x7f2180008040).stop 2022-07-20T02:28:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 shutdown_connections 2022-07-20T02:28:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21880b2920 0x7f218815a500 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2168053a70 0x7f2168055f30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2188067190 0x7f21880632a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 --2- 172.21.15.26:0/208612389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2188061050 0x7f21880b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 >> 172.21.15.26:0/208612389 conn(0x7f21880b60a0 msgr2=0x7f2188145d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 shutdown_connections 2022-07-20T02:28:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.324+0000 7f2190422700 1 -- 172.21.15.26:0/208612389 wait complete. 2022-07-20T02:28:53.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:28:53.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.916+0000 7f4ee15c5700 1 Processor -- start 2022-07-20T02:28:53.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.916+0000 7f4ee15c5700 1 -- start start 2022-07-20T02:28:53.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.916+0000 7f4ee15c5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4edc0b5ef0 0x7f4edc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4edc0b62d0 con 0x7f4edc0a7fc0 2022-07-20T02:28:53.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4edc14aac0 con 0x7f4edc0a8c40 2022-07-20T02:28:53.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4edc14ad60 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edb7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4edc0a7fc0 0x7f4edc0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36686/0 (socket says 172.21.15.26:36686) 2022-07-20T02:28:53.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edb7fe700 1 -- 172.21.15.26:0/2737855500 learned_addr learned my addr 172.21.15.26:0/2737855500 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:53.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edbfff700 1 --2- 172.21.15.26:0/2737855500 >> v2:172.21.15.120:3300/0 conn(0x7f4edc0b5ef0 0x7f4edc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edaffd700 1 -- 172.21.15.26:0/2737855500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319874475 0 0) 0x7f4edc0b62d0 con 0x7f4edc0a7fc0 2022-07-20T02:28:53.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edaffd700 1 -- 172.21.15.26:0/2737855500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ebc003260 con 0x7f4edc0a7fc0 2022-07-20T02:28:53.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edbfff700 1 -- 172.21.15.26:0/2737855500 >> v1:172.21.15.120:6789/0 conn(0x7f4edc0a8c40 legacy=0x7f4edc151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edbfff700 1 -- 172.21.15.26:0/2737855500 >> v1:172.21.15.26:6789/0 conn(0x7f4edc0a7fc0 legacy=0x7f4edc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edbfff700 1 -- 172.21.15.26:0/2737855500 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc14b000 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edbfff700 1 --2- 172.21.15.26:0/2737855500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc1474d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4ed40032f0 tx=0x7f4ed40091d0).ready entity=mon.1 client_cookie=3f4d8af6fa8b70 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edaffd700 1 -- 172.21.15.26:0/2737855500 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ed400b710 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edaffd700 1 -- 172.21.15.26:0/2737855500 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed400b870 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4edaffd700 1 -- 172.21.15.26:0/2737855500 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ed400b9e0 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- 172.21.15.26:0/2737855500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0b5ef0 msgr2=0x7f4edc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/2737855500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc1474d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4ed40032f0 tx=0x7f4ed40091d0).stop 2022-07-20T02:28:53.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- 172.21.15.26:0/2737855500 shutdown_connections 2022-07-20T02:28:53.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/2737855500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc1474d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- 172.21.15.26:0/2737855500 >> 172.21.15.26:0/2737855500 conn(0x7f4edc060310 msgr2=0x7f4edc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:53.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- 172.21.15.26:0/2737855500 shutdown_connections 2022-07-20T02:28:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.920+0000 7f4ee15c5700 1 -- 172.21.15.26:0/2737855500 wait complete. 2022-07-20T02:28:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 Processor -- start 2022-07-20T02:28:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 -- start start 2022-07-20T02:28:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 0x7f4edc065b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0a8c40 0x7f4edc067540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc067f20 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc0aafc0 con 0x7f4edc0a8c40 2022-07-20T02:28:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee15c5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc1573c0 con 0x7f4edc0a7fc0 2022-07-20T02:28:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee0dc4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4edbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 0x7f4edc065b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4edb7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0a8c40 0x7f4edc067540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee0dc4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37412/0 (socket says 172.21.15.26:37412) 2022-07-20T02:28:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4edbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 0x7f4edc065b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:58924/0 (socket says 172.21.15.26:58924) 2022-07-20T02:28:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.924+0000 7f4ee0dc4700 1 -- 172.21.15.26:0/1872311739 learned_addr learned my addr 172.21.15.26:0/1872311739 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:53.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 -- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 msgr2=0x7f4edc065b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 0x7f4edc065b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 -- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0a8c40 msgr2=0x7f4edc067540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:53.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0a8c40 0x7f4edc067540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:53.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 -- 172.21.15.26:0/1872311739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed4009470 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee0dc4700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4edc06b790 tx=0x7f4ecc007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ecc012400 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc012560 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4edc0abe60 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ecc0128c0 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4edc157660 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4edc157200 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ecc019030 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 0x7f4ec0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:53.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4ecc06b2d0 con 0x7f4edc0b5ef0 2022-07-20T02:28:53.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.928+0000 7f4edbfff700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 0x7f4ec0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:53.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.932+0000 7f4edbfff700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 0x7f4ec0055f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4ed401b000 tx=0x7f4ed4016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:53.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:53.940+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ecc042090 con 0x7f4edc0b5ef0 2022-07-20T02:28:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.104+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4edc062fc0 con 0x7f4edc0b5ef0 2022-07-20T02:28:54.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.108+0000 7f4ed97fa700 1 -- 172.21.15.26:0/1872311739 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2494 (secure 0 0 0) 0x7f4ecc0456b0 con 0x7f4edc0b5ef0 2022-07-20T02:28:54.110 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:28:54.111 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:27:33.936408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3286368975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3286368975},{"type":"v1","addr":"172.21.15.26:6837","nonce":3286368975}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:28:54.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 msgr2=0x7f4ec0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 0x7f4ec0055f30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4ed401b000 tx=0x7f4ed4016000).stop 2022-07-20T02:28:54.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 msgr2=0x7f4edc067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4edc06b790 tx=0x7f4ecc007bc0).stop 2022-07-20T02:28:54.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 shutdown_connections 2022-07-20T02:28:54.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4edc0b5ef0 0x7f4edc067a30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4ec0053a70 0x7f4ec0055f30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4edc0a8c40 0x7f4edc067540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 --2- 172.21.15.26:0/1872311739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4edc0a7fc0 0x7f4edc065b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 >> 172.21.15.26:0/1872311739 conn(0x7f4edc060310 msgr2=0x7f4edc149e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:54.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 shutdown_connections 2022-07-20T02:28:54.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.112+0000 7f4ee15c5700 1 -- 172.21.15.26:0/1872311739 wait complete. 2022-07-20T02:28:54.131 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 18 2022-07-20T02:28:54.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:28:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 Processor -- start 2022-07-20T02:28:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 -- start start 2022-07-20T02:28:54.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efbf00ac8e0 0x7efbf00b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:54.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0140190 con 0x7efbf00b5d50 2022-07-20T02:28:54.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0140430 con 0x7efbf00b1900 2022-07-20T02:28:54.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.264+0000 7efbf754c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf00af740 con 0x7efbf00ac8e0 2022-07-20T02:28:54.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf6d4b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efbf00ac8e0 0x7efbf00b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:54.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf654a700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efbf00b5d50 0x7efbf013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36698/0 (socket says 172.21.15.26:36698) 2022-07-20T02:28:54.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf654a700 1 -- 172.21.15.26:0/72668806 learned_addr learned my addr 172.21.15.26:0/72668806 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:54.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf6d4b700 1 -- 172.21.15.26:0/72668806 >> v1:172.21.15.120:6789/0 conn(0x7efbf00b1900 legacy=0x7efbf00afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf6d4b700 1 -- 172.21.15.26:0/72668806 >> v1:172.21.15.26:6789/0 conn(0x7efbf00b5d50 legacy=0x7efbf013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf5548700 1 -- 172.21.15.26:0/72668806 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134788683 0 0) 0x7efbf0140430 con 0x7efbf00b1900 2022-07-20T02:28:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf6d4b700 1 -- 172.21.15.26:0/72668806 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf01406d0 con 0x7efbf00ac8e0 2022-07-20T02:28:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf6d4b700 1 --2- 172.21.15.26:0/72668806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf00b0550 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7efbe80032f0 tx=0x7efbe800b470).ready entity=mon.0 client_cookie=d941e40a04c0783b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf5548700 1 -- 172.21.15.26:0/72668806 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbe800de20 con 0x7efbf00ac8e0 2022-07-20T02:28:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf5548700 1 -- 172.21.15.26:0/72668806 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8010470 con 0x7efbf00ac8e0 2022-07-20T02:28:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf5548700 1 -- 172.21.15.26:0/72668806 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbe80107d0 con 0x7efbf00ac8e0 2022-07-20T02:28:54.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 -- 172.21.15.26:0/72668806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 msgr2=0x7efbf00b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 --2- 172.21.15.26:0/72668806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf00b0550 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7efbe80032f0 tx=0x7efbe800b470).stop 2022-07-20T02:28:54.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 -- 172.21.15.26:0/72668806 shutdown_connections 2022-07-20T02:28:54.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 --2- 172.21.15.26:0/72668806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf00b0550 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 -- 172.21.15.26:0/72668806 >> 172.21.15.26:0/72668806 conn(0x7efbf0060310 msgr2=0x7efbf00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:54.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 -- 172.21.15.26:0/72668806 shutdown_connections 2022-07-20T02:28:54.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.268+0000 7efbf754c700 1 -- 172.21.15.26:0/72668806 wait complete. 2022-07-20T02:28:54.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 Processor -- start 2022-07-20T02:28:54.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 -- start start 2022-07-20T02:28:54.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:54.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbf00b1900 0x7efbf013d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:54.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbf00b5d50 0x7efbf009ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:54.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00669f0 con 0x7efbf00ac8e0 2022-07-20T02:28:54.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf014ec40 con 0x7efbf00b5d50 2022-07-20T02:28:54.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf754c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf0066240 con 0x7efbf00b1900 2022-07-20T02:28:54.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:54.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37422/0 (socket says 172.21.15.26:37422) 2022-07-20T02:28:54.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 -- 172.21.15.26:0/2276220400 learned_addr learned my addr 172.21.15.26:0/2276220400 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:54.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf6d4b700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbf00b5d50 0x7efbf009ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:54.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf5d49700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbf00b1900 0x7efbf013d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:54.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 -- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbf00b1900 msgr2=0x7efbf013d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbf00b1900 0x7efbf013d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 -- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbf00b5d50 msgr2=0x7efbf009ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:54.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbf00b5d50 0x7efbf009ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:54.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.272+0000 7efbf654a700 1 -- 172.21.15.26:0/2276220400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe800b710 con 0x7efbf00ac8e0 2022-07-20T02:28:54.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf654a700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7efbf00a9770 tx=0x7efbec007c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:54.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbec007110 con 0x7efbf00ac8e0 2022-07-20T02:28:54.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec007270 con 0x7efbf00ac8e0 2022-07-20T02:28:54.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbec005030 con 0x7efbf00ac8e0 2022-07-20T02:28:54.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0067260 con 0x7efbf00ac8e0 2022-07-20T02:28:54.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf0066fc0 con 0x7efbf00ac8e0 2022-07-20T02:28:54.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf0140970 con 0x7efbf00ac8e0 2022-07-20T02:28:54.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efbec01b070 con 0x7efbf00ac8e0 2022-07-20T02:28:54.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 0x7efbd0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:54.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6420+0+0 (secure 0 0 0) 0x7efbec06b450 con 0x7efbf00ac8e0 2022-07-20T02:28:54.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf5d49700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 0x7efbd0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:54.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.276+0000 7efbf5d49700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 0x7efbd0055ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efbe400d000 tx=0x7efbe4007a90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:54.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.284+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efbec042170 con 0x7efbf00ac8e0 2022-07-20T02:28:54.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:54.444+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efbf00574e0 con 0x7efbf00ac8e0 2022-07-20T02:28:55.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:28:55.368+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:28:55.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.384+0000 7efbe37fe700 1 -- 172.21.15.26:0/2276220400 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) v1 ==== 161+0+0 (secure 0 0 0) 0x7efbec045790 con 0x7efbf00ac8e0 2022-07-20T02:28:55.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 msgr2=0x7efbd0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 0x7efbd0055ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efbe400d000 tx=0x7efbe4007a90).stop 2022-07-20T02:28:55.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 msgr2=0x7efbf013d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7efbf00a9770 tx=0x7efbec007c60).stop 2022-07-20T02:28:55.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 shutdown_connections 2022-07-20T02:28:55.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbd0053b30 0x7efbd0055ff0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbf00b5d50 0x7efbf009ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbf00b1900 0x7efbf013d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 --2- 172.21.15.26:0/2276220400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbf00ac8e0 0x7efbf013d2f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 >> 172.21.15.26:0/2276220400 conn(0x7efbf0060310 msgr2=0x7efbf013f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 shutdown_connections 2022-07-20T02:28:55.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.388+0000 7efbf754c700 1 -- 172.21.15.26:0/2276220400 wait complete. 2022-07-20T02:28:55.395 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:28:55.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:28:55.423 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:28:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 Processor -- start 2022-07-20T02:28:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 -- start start 2022-07-20T02:28:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212c0ae370 con 0x7f212c0b14b0 2022-07-20T02:28:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212c062050 con 0x7f212c0614b0 2022-07-20T02:28:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2131182700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c0b28b0 con 0x7f212c0adf90 2022-07-20T02:28:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212a59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f212c0b14b0 0x7f212c060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36730/0 (socket says 172.21.15.26:36730) 2022-07-20T02:28:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212a59c700 1 -- 172.21.15.26:0/2429564216 learned_addr learned my addr 172.21.15.26:0/2429564216 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212ad9d700 1 --2- 172.21.15.26:0/2429564216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692445635 0 0) 0x7f212c0ae370 con 0x7f212c0b14b0 2022-07-20T02:28:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c003260 con 0x7f212c0b14b0 2022-07-20T02:28:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212ad9d700 1 -- 172.21.15.26:0/2429564216 >> v1:172.21.15.120:6789/0 conn(0x7f212c0614b0 legacy=0x7f212c061940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212ad9d700 1 -- 172.21.15.26:0/2429564216 >> v1:172.21.15.26:6789/0 conn(0x7f212c0b14b0 legacy=0x7f212c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599230542 0 0) 0x7f210c003260 con 0x7f212c0b14b0 2022-07-20T02:28:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212ad9d700 1 -- 172.21.15.26:0/2429564216 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c0b0610 con 0x7f212c0adf90 2022-07-20T02:28:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f212ad9d700 1 --2- 172.21.15.26:0/2429564216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c0b2c60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2124001b80 tx=0x7f2114006df0).ready entity=mon.2 client_cookie=9ab36c71885dd789 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21240019e0 con 0x7f212c0adf90 2022-07-20T02:28:55.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114000e90 con 0x7f212c0adf90 2022-07-20T02:28:55.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.508+0000 7f2129d9b700 1 -- 172.21.15.26:0/2429564216 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f211400d4a0 con 0x7f212c0adf90 2022-07-20T02:28:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- 172.21.15.26:0/2429564216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 msgr2=0x7f212c0b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 --2- 172.21.15.26:0/2429564216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c0b2c60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2124001b80 tx=0x7f2114006df0).stop 2022-07-20T02:28:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- 172.21.15.26:0/2429564216 shutdown_connections 2022-07-20T02:28:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 --2- 172.21.15.26:0/2429564216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c0b2c60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- 172.21.15.26:0/2429564216 >> 172.21.15.26:0/2429564216 conn(0x7f212c0b66e0 msgr2=0x7f212c0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- 172.21.15.26:0/2429564216 shutdown_connections 2022-07-20T02:28:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- 172.21.15.26:0/2429564216 wait complete. 2022-07-20T02:28:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 Processor -- start 2022-07-20T02:28:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- start start 2022-07-20T02:28:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f212c0b14b0 0x7f212c143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c1524b0 con 0x7f212c0614b0 2022-07-20T02:28:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c066800 con 0x7f212c0b14b0 2022-07-20T02:28:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f2131182700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c066aa0 con 0x7f212c0adf90 2022-07-20T02:28:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f212ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f212ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37454/0 (socket says 172.21.15.26:37454) 2022-07-20T02:28:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f212b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f212c0b14b0 0x7f212c143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f212ad9d700 1 -- 172.21.15.26:0/1434841376 learned_addr learned my addr 172.21.15.26:0/1434841376 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.512+0000 7f212a59c700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 -- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 msgr2=0x7f212c143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 -- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f212c0b14b0 msgr2=0x7f212c143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f212c0b14b0 0x7f212c143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 -- 172.21.15.26:0/1434841376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211400b180 con 0x7f212c0614b0 2022-07-20T02:28:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f212ad9d700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f2124001b80 tx=0x7f2114014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2114016030 con 0x7f212c0614b0 2022-07-20T02:28:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114013bf0 con 0x7f212c0614b0 2022-07-20T02:28:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c066d40 con 0x7f212c0614b0 2022-07-20T02:28:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f211400d4a0 con 0x7f212c0614b0 2022-07-20T02:28:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c143f60 con 0x7f212c0614b0 2022-07-20T02:28:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c0574e0 con 0x7f212c0614b0 2022-07-20T02:28:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.516+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2114024020 con 0x7f212c0614b0 2022-07-20T02:28:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.520+0000 7f2123fff700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 0x7f2110055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.520+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f2114017070 con 0x7f212c0614b0 2022-07-20T02:28:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.520+0000 7f212a59c700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 0x7f2110055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.520+0000 7f212a59c700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 0x7f2110055e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f212400d000 tx=0x7f2124000f40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.524+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2114043d10 con 0x7f212c0614b0 2022-07-20T02:28:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f212c066ed0 con 0x7f212c0614b0 2022-07-20T02:28:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2123fff700 1 -- 172.21.15.26:0/1434841376 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2133 (secure 0 0 0) 0x7f2114047330 con 0x7f212c0614b0 2022-07-20T02:28:55.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:28:55.694 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:27:32.932112+0000","modified":"2022-07-20T02:28:55.371125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T02:28:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 msgr2=0x7f2110055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 0x7f2110055e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f212400d000 tx=0x7f2124000f40).stop 2022-07-20T02:28:55.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 msgr2=0x7f212c1491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f2124001b80 tx=0x7f2114014000).stop 2022-07-20T02:28:55.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 shutdown_connections 2022-07-20T02:28:55.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2110053950 0x7f2110055e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f212c0b14b0 0x7f212c143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f212c0adf90 0x7f212c143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 --2- 172.21.15.26:0/1434841376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f212c0614b0 0x7f212c1491a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 >> 172.21.15.26:0/1434841376 conn(0x7f212c0b66e0 msgr2=0x7f212c144fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:55.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 shutdown_connections 2022-07-20T02:28:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.688+0000 7f2131182700 1 -- 172.21.15.26:0/1434841376 wait complete. 2022-07-20T02:28:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 19 2022-07-20T02:28:55.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:28:55.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 Processor -- start 2022-07-20T02:28:55.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 -- start start 2022-07-20T02:28:55.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 0x7f7308143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7308147000 0x7f73080b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73081474e0 con 0x7f7308060f00 2022-07-20T02:28:55.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730805d3b0 con 0x7f7308147000 2022-07-20T02:28:55.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730f05c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73081431e0 con 0x7f730805cfd0 2022-07-20T02:28:55.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730d859700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7308147000 0x7f73080b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730d859700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7308147000 0x7f73080b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37462/0 (socket says 172.21.15.26:37462) 2022-07-20T02:28:55.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.868+0000 7f730d859700 1 -- 172.21.15.26:0/630900370 learned_addr learned my addr 172.21.15.26:0/630900370 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:55.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730e05a700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 0x7f7308143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d859700 1 -- 172.21.15.26:0/630900370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 msgr2=0x7f7308143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:55.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d859700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 0x7f7308143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d859700 1 -- 172.21.15.26:0/630900370 >> v1:172.21.15.120:6789/0 conn(0x7f7308060f00 legacy=0x7f730813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d058700 1 -- 172.21.15.26:0/630900370 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760622383 0 0) 0x7f73081474e0 con 0x7f7308060f00 2022-07-20T02:28:55.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d859700 1 -- 172.21.15.26:0/630900370 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308143480 con 0x7f7308147000 2022-07-20T02:28:55.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730e05a700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 0x7f7308143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:28:55.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d859700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308147000 0x7f73080b4730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f73040032f0 tx=0x7f7304009290).ready entity=mon.0 client_cookie=3397b1f389e9c31b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d058700 1 -- 172.21.15.26:0/630900370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730400b8a0 con 0x7f7308147000 2022-07-20T02:28:55.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d058700 1 -- 172.21.15.26:0/630900370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400ba00 con 0x7f7308147000 2022-07-20T02:28:55.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730d058700 1 -- 172.21.15.26:0/630900370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730400bb70 con 0x7f7308147000 2022-07-20T02:28:55.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- 172.21.15.26:0/630900370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308147000 msgr2=0x7f73080b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308147000 0x7f73080b4730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f73040032f0 tx=0x7f7304009290).stop 2022-07-20T02:28:55.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- 172.21.15.26:0/630900370 shutdown_connections 2022-07-20T02:28:55.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308147000 0x7f73080b4730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 --2- 172.21.15.26:0/630900370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f730805cfd0 0x7f7308143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- 172.21.15.26:0/630900370 >> 172.21.15.26:0/630900370 conn(0x7f73080620c0 msgr2=0x7f730805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:55.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- 172.21.15.26:0/630900370 shutdown_connections 2022-07-20T02:28:55.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- 172.21.15.26:0/630900370 wait complete. 2022-07-20T02:28:55.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 Processor -- start 2022-07-20T02:28:55.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.872+0000 7f730f05c700 1 -- start start 2022-07-20T02:28:55.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7308147000 0x7f7308151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 0x7f73081524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080ad690 con 0x7f7308060f00 2022-07-20T02:28:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73081416b0 con 0x7f7308147000 2022-07-20T02:28:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73080adc50 con 0x7f7308142050 2022-07-20T02:28:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37468/0 (socket says 172.21.15.26:37468) 2022-07-20T02:28:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 -- 172.21.15.26:0/1650000096 learned_addr learned my addr 172.21.15.26:0/1650000096 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730d859700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7308147000 0x7f7308151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e85b700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 0x7f73081524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 -- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 msgr2=0x7f73081524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 0x7f73081524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 -- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7308147000 msgr2=0x7f7308151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7308147000 0x7f7308151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 -- 172.21.15.26:0/1650000096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7304009530 con 0x7f7308060f00 2022-07-20T02:28:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e85b700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 0x7f73081524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:28:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730e05a700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f72f8006940 tx=0x7f72f8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f8011780 con 0x7f7308060f00 2022-07-20T02:28:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f80118e0 con 0x7f7308060f00 2022-07-20T02:28:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f8011c40 con 0x7f7308060f00 2022-07-20T02:28:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73080ad8c0 con 0x7f7308060f00 2022-07-20T02:28:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7308141bb0 con 0x7f7308060f00 2022-07-20T02:28:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.876+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7308141470 con 0x7f7308060f00 2022-07-20T02:28:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.880+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f72f801c460 con 0x7f7308060f00 2022-07-20T02:28:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.880+0000 7f72ff7fe700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 0x7f72e8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.880+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f72f806c380 con 0x7f7308060f00 2022-07-20T02:28:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.880+0000 7f730d859700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 0x7f72e8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.880+0000 7f730d859700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 0x7f72e8055ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7304019000 tx=0x7f7304016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:55.884+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72f8043090 con 0x7f7308060f00 2022-07-20T02:28:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.048+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7308143720 con 0x7f7308060f00 2022-07-20T02:28:56.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.392+0000 7f72ff7fe700 1 -- 172.21.15.26:0/1650000096 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f72f80466b0 con 0x7f7308060f00 2022-07-20T02:28:56.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 msgr2=0x7f72e8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 0x7f72e8055ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7304019000 tx=0x7f7304016000).stop 2022-07-20T02:28:56.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 msgr2=0x7f7308142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f72f8006940 tx=0x7f72f8008040).stop 2022-07-20T02:28:56.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 shutdown_connections 2022-07-20T02:28:56.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e80539e0 0x7f72e8055ea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7308142050 0x7f73081524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7308147000 0x7f7308151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 --2- 172.21.15.26:0/1650000096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7308060f00 0x7f7308142930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 >> 172.21.15.26:0/1650000096 conn(0x7f730805cfd0 msgr2=0x7f73080b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:56.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 shutdown_connections 2022-07-20T02:28:56.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.400+0000 7f730f05c700 1 -- 172.21.15.26:0/1650000096 wait complete. 2022-07-20T02:28:56.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:28:56.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.532+0000 7f0e4cae4700 1 Processor -- start 2022-07-20T02:28:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.532+0000 7f0e4cae4700 1 -- start start 2022-07-20T02:28:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.532+0000 7f0e4cae4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e4805fa50 0x7f0e48067060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e48067550 con 0x7f0e480aeae0 2022-07-20T02:28:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e480aeec0 con 0x7f0e480602c0 2022-07-20T02:28:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480b2ba0 con 0x7f0e4805fa50 2022-07-20T02:28:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0e480aeae0 0x7f0e48066950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36758/0 (socket says 172.21.15.26:36758) 2022-07-20T02:28:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4659c700 1 -- 172.21.15.26:0/1858383347 learned_addr learned my addr 172.21.15.26:0/1858383347 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e46d9d700 1 --2- 172.21.15.26:0/1858383347 >> v2:172.21.15.120:3300/0 conn(0x7f0e4805fa50 0x7f0e48067060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4559a700 1 -- 172.21.15.26:0/1858383347 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498090905 0 0) 0x7f0e38001110 con 0x7f0e480aeae0 2022-07-20T02:28:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4559a700 1 -- 172.21.15.26:0/1858383347 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e2c003260 con 0x7f0e480aeae0 2022-07-20T02:28:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e46d9d700 1 -- 172.21.15.26:0/1858383347 >> v1:172.21.15.120:6789/0 conn(0x7f0e480602c0 legacy=0x7f0e480af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e46d9d700 1 -- 172.21.15.26:0/1858383347 >> v1:172.21.15.26:6789/0 conn(0x7f0e480aeae0 legacy=0x7f0e48066950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e46d9d700 1 -- 172.21.15.26:0/1858383347 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480af210 con 0x7f0e4805fa50 2022-07-20T02:28:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e46d9d700 1 --2- 172.21.15.26:0/1858383347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e4805fa50 0x7f0e48067060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f0e400032f0 tx=0x7f0e400091d0).ready entity=mon.1 client_cookie=78ba45f4b7a65be7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4559a700 1 -- 172.21.15.26:0/1858383347 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4000b6c0 con 0x7f0e4805fa50 2022-07-20T02:28:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4559a700 1 -- 172.21.15.26:0/1858383347 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000b820 con 0x7f0e4805fa50 2022-07-20T02:28:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4559a700 1 -- 172.21.15.26:0/1858383347 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4000b990 con 0x7f0e4805fa50 2022-07-20T02:28:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- 172.21.15.26:0/1858383347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e4805fa50 msgr2=0x7f0e48067060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/1858383347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e4805fa50 0x7f0e48067060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f0e400032f0 tx=0x7f0e400091d0).stop 2022-07-20T02:28:56.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- 172.21.15.26:0/1858383347 shutdown_connections 2022-07-20T02:28:56.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/1858383347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e4805fa50 0x7f0e48067060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- 172.21.15.26:0/1858383347 >> 172.21.15.26:0/1858383347 conn(0x7f0e480b66e0 msgr2=0x7f0e480b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- 172.21.15.26:0/1858383347 shutdown_connections 2022-07-20T02:28:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.536+0000 7f0e4cae4700 1 -- 172.21.15.26:0/1858383347 wait complete. 2022-07-20T02:28:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 Processor -- start 2022-07-20T02:28:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 -- start start 2022-07-20T02:28:56.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:56.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e480602c0 0x7f0e480a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:56.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e480aeae0 0x7f0e480a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:56.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480a3590 con 0x7f0e4805fa50 2022-07-20T02:28:56.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e48146820 con 0x7f0e480aeae0 2022-07-20T02:28:56.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4cae4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e48146ac0 con 0x7f0e480602c0 2022-07-20T02:28:56.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:56.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37484/0 (socket says 172.21.15.26:37484) 2022-07-20T02:28:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e4659c700 1 -- 172.21.15.26:0/2444238626 learned_addr learned my addr 172.21.15.26:0/2444238626 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e46d9d700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e480aeae0 0x7f0e480a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.540+0000 7f0e45d9b700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e480602c0 0x7f0e480a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 -- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e480602c0 msgr2=0x7f0e480a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e480602c0 0x7f0e480a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 -- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e480aeae0 msgr2=0x7f0e480a40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:56.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e480aeae0 0x7f0e480a40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:56.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 -- 172.21.15.26:0/2444238626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e40009470 con 0x7f0e4805fa50 2022-07-20T02:28:56.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4659c700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f0e380012a0 tx=0x7f0e38007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:56.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e380070d0 con 0x7f0e4805fa50 2022-07-20T02:28:56.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e38007230 con 0x7f0e4805fa50 2022-07-20T02:28:56.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e48146d60 con 0x7f0e4805fa50 2022-07-20T02:28:56.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e38005610 con 0x7f0e4805fa50 2022-07-20T02:28:56.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e480a6380 con 0x7f0e4805fa50 2022-07-20T02:28:56.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e48155d00 con 0x7f0e4805fa50 2022-07-20T02:28:56.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.544+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e3801a060 con 0x7f0e4805fa50 2022-07-20T02:28:56.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.548+0000 7f0e377fe700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 0x7f0e24055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:56.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.548+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f0e3806b650 con 0x7f0e4805fa50 2022-07-20T02:28:56.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.548+0000 7f0e45d9b700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 0x7f0e24055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:56.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.548+0000 7f0e45d9b700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 0x7f0e24055f30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0e3c00d000 tx=0x7f0e3c000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:56.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.552+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e380423c0 con 0x7f0e4805fa50 2022-07-20T02:28:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:56.676+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e48067550 con 0x7f0e4805fa50 2022-07-20T02:28:57.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.400+0000 7f0e377fe700 1 -- 172.21.15.26:0/2444238626 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) v1 ==== 185+0+0 (secure 0 0 0) 0x7f0e48067550 con 0x7f0e4805fa50 2022-07-20T02:28:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.404+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 msgr2=0x7f0e24055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.404+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 0x7f0e24055f30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0e3c00d000 tx=0x7f0e3c000bc0).stop 2022-07-20T02:28:57.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.404+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 msgr2=0x7f0e48065890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.404+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f0e380012a0 tx=0x7f0e38007a00).stop 2022-07-20T02:28:57.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 shutdown_connections 2022-07-20T02:28:57.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e24053a70 0x7f0e24055f30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e480aeae0 0x7f0e480a40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e480602c0 0x7f0e480a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 --2- 172.21.15.26:0/2444238626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e4805fa50 0x7f0e48065890 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 >> 172.21.15.26:0/2444238626 conn(0x7f0e480b66e0 msgr2=0x7f0e480b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:57.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 shutdown_connections 2022-07-20T02:28:57.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.408+0000 7f0e4cae4700 1 -- 172.21.15.26:0/2444238626 wait complete. 2022-07-20T02:28:57.426 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:28:57.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:28:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 Processor -- start 2022-07-20T02:28:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 -- start start 2022-07-20T02:28:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21fc0add30 0x7f21fc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0af6d0 0x7f21fc0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0aa1e0 con 0x7f21fc060950 2022-07-20T02:28:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc0ae110 con 0x7f21fc0add30 2022-07-20T02:28:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.548+0000 7f2203081700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0a9f40 con 0x7f21fc0af6d0 2022-07-20T02:28:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21fc0add30 0x7f21fc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220187e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f21fc060950 0x7f21fc060dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36774/0 (socket says 172.21.15.26:36774) 2022-07-20T02:28:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220187e700 1 -- 172.21.15.26:0/2609575628 learned_addr learned my addr 172.21.15.26:0/2609575628 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:57.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2202880700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0af6d0 0x7f21fc0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220107d700 1 -- 172.21.15.26:0/2609575628 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357487083 0 0) 0x7f21fc0aa1e0 con 0x7f21fc060950 2022-07-20T02:28:57.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220107d700 1 -- 172.21.15.26:0/2609575628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e4003260 con 0x7f21fc060950 2022-07-20T02:28:57.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 -- 172.21.15.26:0/2609575628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0af6d0 msgr2=0x7f21fc0afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:57.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0af6d0 0x7f21fc0afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 -- 172.21.15.26:0/2609575628 >> v1:172.21.15.26:6789/0 conn(0x7f21fc060950 legacy=0x7f21fc060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 -- 172.21.15.26:0/2609575628 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0b1110 con 0x7f21fc0add30 2022-07-20T02:28:57.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220207f700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0add30 0x7f21fc0b2ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f21f8002f70 tx=0x7f21f8006df0).ready entity=mon.0 client_cookie=f3a5dae44cf7ff9c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:57.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220107d700 1 -- 172.21.15.26:0/2609575628 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f800d4a0 con 0x7f21fc0add30 2022-07-20T02:28:57.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220107d700 1 -- 172.21.15.26:0/2609575628 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8000e90 con 0x7f21fc0add30 2022-07-20T02:28:57.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f220107d700 1 -- 172.21.15.26:0/2609575628 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f800d7f0 con 0x7f21fc0add30 2022-07-20T02:28:57.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 -- 172.21.15.26:0/2609575628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0add30 msgr2=0x7f21fc0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0add30 0x7f21fc0b2ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f21f8002f70 tx=0x7f21f8006df0).stop 2022-07-20T02:28:57.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 -- 172.21.15.26:0/2609575628 shutdown_connections 2022-07-20T02:28:57.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0af6d0 0x7f21fc0afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 --2- 172.21.15.26:0/2609575628 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0add30 0x7f21fc0b2ab0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 -- 172.21.15.26:0/2609575628 >> 172.21.15.26:0/2609575628 conn(0x7f21fc0b3050 msgr2=0x7f21fc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:57.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 -- 172.21.15.26:0/2609575628 shutdown_connections 2022-07-20T02:28:57.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 -- 172.21.15.26:0/2609575628 wait complete. 2022-07-20T02:28:57.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.552+0000 7f2203081700 1 Processor -- start 2022-07-20T02:28:57.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- start start 2022-07-20T02:28:57.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21fc060950 0x7f21fc05fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0add30 0x7f21fc13e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc145d30 con 0x7f21fc0af6d0 2022-07-20T02:28:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc13dcc0 con 0x7f21fc060950 2022-07-20T02:28:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0b3bc0 con 0x7f21fc0add30 2022-07-20T02:28:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37502/0 (socket says 172.21.15.26:37502) 2022-07-20T02:28:57.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 -- 172.21.15.26:0/1928459466 learned_addr learned my addr 172.21.15.26:0/1928459466 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:57.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f220207f700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21fc060950 0x7f21fc05fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f220187e700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0add30 0x7f21fc13e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 -- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0add30 msgr2=0x7f21fc13e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0add30 0x7f21fc13e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 -- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21fc060950 msgr2=0x7f21fc05fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:57.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21fc060950 0x7f21fc05fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:57.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 -- 172.21.15.26:0/1928459466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f800b180 con 0x7f21fc0af6d0 2022-07-20T02:28:57.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2202880700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f21f4006e00 tx=0x7f21f40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:57.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f4012600 con 0x7f21fc0af6d0 2022-07-20T02:28:57.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f4012760 con 0x7f21fc0af6d0 2022-07-20T02:28:57.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc0ad6f0 con 0x7f21fc0af6d0 2022-07-20T02:28:57.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f4012ac0 con 0x7f21fc0af6d0 2022-07-20T02:28:57.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.556+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc146310 con 0x7f21fc0af6d0 2022-07-20T02:28:57.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f21f4022070 con 0x7f21fc0af6d0 2022-07-20T02:28:57.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc0b13b0 con 0x7f21fc0af6d0 2022-07-20T02:28:57.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f21eb7fe700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 0x7f21dc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:57.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f21f406c550 con 0x7f21fc0af6d0 2022-07-20T02:28:57.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f220207f700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 0x7f21dc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:57.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.560+0000 7f220207f700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 0x7f21dc055e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f21f801a000 tx=0x7f21f8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:57.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.564+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21f4043480 con 0x7f21fc0af6d0 2022-07-20T02:28:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:57.692+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f21fc0574e0 con 0x7f21fc0af6d0 2022-07-20T02:28:58.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.412+0000 7f21eb7fe700 1 -- 172.21.15.26:0/1928459466 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) v1 ==== 173+0+0 (secure 0 0 0) 0x7f21f40160d0 con 0x7f21fc0af6d0 2022-07-20T02:28:58.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 msgr2=0x7f21dc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 0x7f21dc055e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f21f801a000 tx=0x7f21f8014000).stop 2022-07-20T02:28:58.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 msgr2=0x7f21fc13eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f21f4006e00 tx=0x7f21f40091d0).stop 2022-07-20T02:28:58.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 shutdown_connections 2022-07-20T02:28:58.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21fc0af6d0 0x7f21fc13eb10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21dc053950 0x7f21dc055e10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21fc0add30 0x7f21fc13e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 --2- 172.21.15.26:0/1928459466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21fc060950 0x7f21fc05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 >> 172.21.15.26:0/1928459466 conn(0x7f21fc0b3050 msgr2=0x7f21fc1439b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:58.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 shutdown_connections 2022-07-20T02:28:58.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.420+0000 7f2203081700 1 -- 172.21.15.26:0/1928459466 wait complete. 2022-07-20T02:28:58.424 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:28:58.442 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:28:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 Processor -- start 2022-07-20T02:28:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 -- start start 2022-07-20T02:28:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f499c0621f0 0x7f499c067020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0b0f50 con 0x7f499c060250 2022-07-20T02:28:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499c060db0 con 0x7f499c0621f0 2022-07-20T02:28:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.548+0000 7f49a3577700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c13e7f0 con 0x7f499c0aff40 2022-07-20T02:28:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f499c0621f0 0x7f499c067020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a2d76700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f499c060250 0x7f499c0606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51002/0 (socket says 172.21.15.26:51002) 2022-07-20T02:28:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a2d76700 1 -- 172.21.15.26:0/39177067 learned_addr learned my addr 172.21.15.26:0/39177067 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a2575700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 -- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 msgr2=0x7f499c066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:28:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 --2- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c066b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 -- 172.21.15.26:0/39177067 >> v1:172.21.15.120:6789/0 conn(0x7f499c060250 legacy=0x7f499c0606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 -- 172.21.15.26:0/39177067 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c05f8e0 con 0x7f499c0621f0 2022-07-20T02:28:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1d74700 1 --2- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0621f0 0x7f499c067020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998006df0).ready entity=mon.1 client_cookie=a796988e019b7c24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1573700 1 -- 172.21.15.26:0/39177067 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499800d570 con 0x7f499c0621f0 2022-07-20T02:28:58.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1573700 1 -- 172.21.15.26:0/39177067 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4998000e90 con 0x7f499c0621f0 2022-07-20T02:28:58.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a1573700 1 -- 172.21.15.26:0/39177067 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499800d8c0 con 0x7f499c0621f0 2022-07-20T02:28:58.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 -- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0621f0 msgr2=0x7f499c067020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 --2- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0621f0 0x7f499c067020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998006df0).stop 2022-07-20T02:28:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 -- 172.21.15.26:0/39177067 shutdown_connections 2022-07-20T02:28:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 --2- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0621f0 0x7f499c067020 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 --2- 172.21.15.26:0/39177067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c066b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 -- 172.21.15.26:0/39177067 >> 172.21.15.26:0/39177067 conn(0x7f499c0b3050 msgr2=0x7f499c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 -- 172.21.15.26:0/39177067 shutdown_connections 2022-07-20T02:28:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.552+0000 7f49a3577700 1 -- 172.21.15.26:0/39177067 wait complete. 2022-07-20T02:28:58.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 Processor -- start 2022-07-20T02:28:58.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- start start 2022-07-20T02:28:58.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c060250 0x7f499c140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c13a460 con 0x7f499c0621f0 2022-07-20T02:28:58.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0662d0 con 0x7f499c060250 2022-07-20T02:28:58.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c066570 con 0x7f499c0aff40 2022-07-20T02:28:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37514/0 (socket says 172.21.15.26:37514) 2022-07-20T02:28:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 -- 172.21.15.26:0/1192207316 learned_addr learned my addr 172.21.15.26:0/1192207316 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a2d76700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a2575700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c060250 0x7f499c140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 -- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 msgr2=0x7f499c154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 -- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c060250 msgr2=0x7f499c140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c060250 0x7f499c140710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 -- 172.21.15.26:0/1192207316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499800b180 con 0x7f499c0621f0 2022-07-20T02:28:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a2d76700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:28:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a1d74700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499800dea0 con 0x7f499c0621f0 2022-07-20T02:28:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4998013de0 con 0x7f499c0621f0 2022-07-20T02:28:58.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c153ad0 con 0x7f499c0621f0 2022-07-20T02:28:58.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499801e5f0 con 0x7f499c0621f0 2022-07-20T02:28:58.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.556+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c0aa790 con 0x7f499c0621f0 2022-07-20T02:28:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499c0aaa70 con 0x7f499c0621f0 2022-07-20T02:28:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4998025030 con 0x7f499c0621f0 2022-07-20T02:28:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f498f7fe700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 0x7f497c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f499806cb00 con 0x7f499c0621f0 2022-07-20T02:28:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f49a2575700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 0x7f497c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.560+0000 7f49a2575700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 0x7f497c055ea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f499000b000 tx=0x7f4990006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.564+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f499806ee60 con 0x7f499c0621f0 2022-07-20T02:28:58.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.692+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f499c066730 con 0x7f499c0621f0 2022-07-20T02:28:58.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.700+0000 7f498f7fe700 1 -- 172.21.15.26:0/1192207316 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4998023020 con 0x7f499c0621f0 2022-07-20T02:28:58.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 msgr2=0x7f497c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 0x7f497c055ea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f499000b000 tx=0x7f4990006df0).stop 2022-07-20T02:28:58.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 msgr2=0x7f499c153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998014000).stop 2022-07-20T02:28:58.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 shutdown_connections 2022-07-20T02:28:58.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0621f0 0x7f499c153f70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c0539e0 0x7f497c055ea0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c0aff40 0x7f499c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 --2- 172.21.15.26:0/1192207316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c060250 0x7f499c140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 >> 172.21.15.26:0/1192207316 conn(0x7f499c0b3050 msgr2=0x7f499c1412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:58.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 shutdown_connections 2022-07-20T02:28:58.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.704+0000 7f49a3577700 1 -- 172.21.15.26:0/1192207316 wait complete. 2022-07-20T02:28:58.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-20T02:28:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 Processor -- start 2022-07-20T02:28:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 -- start start 2022-07-20T02:28:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43b8147be0 0x7f43b80aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b80b62d0 con 0x7f43b80b5ef0 2022-07-20T02:28:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b814aac0 con 0x7f43b80a7fc0 2022-07-20T02:28:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b814ad60 con 0x7f43b8147be0 2022-07-20T02:28:58.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f43b80b5ef0 0x7f43b81474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36802/0 (socket says 172.21.15.26:36802) 2022-07-20T02:28:58.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b659c700 1 -- 172.21.15.26:0/3345363427 learned_addr learned my addr 172.21.15.26:0/3345363427 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:58.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.820+0000 7f43b5d9b700 1 --2- 172.21.15.26:0/3345363427 >> v2:172.21.15.120:3300/0 conn(0x7f43b8147be0 0x7f43b80aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b559a700 1 -- 172.21.15.26:0/3345363427 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980275018 0 0) 0x7f43b80b62d0 con 0x7f43b80b5ef0 2022-07-20T02:28:58.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b559a700 1 -- 172.21.15.26:0/3345363427 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f439c003260 con 0x7f43b80b5ef0 2022-07-20T02:28:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b5d9b700 1 -- 172.21.15.26:0/3345363427 >> v1:172.21.15.120:6789/0 conn(0x7f43b80a7fc0 legacy=0x7f43b80af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b5d9b700 1 -- 172.21.15.26:0/3345363427 >> v1:172.21.15.26:6789/0 conn(0x7f43b80b5ef0 legacy=0x7f43b81474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b5d9b700 1 -- 172.21.15.26:0/3345363427 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b814b000 con 0x7f43b8147be0 2022-07-20T02:28:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b5d9b700 1 --2- 172.21.15.26:0/3345363427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b8147be0 0x7f43b80aee40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f43a8002f70 tx=0x7f43a8006df0).ready entity=mon.1 client_cookie=95bfe1ddfa0e4952 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b559a700 1 -- 172.21.15.26:0/3345363427 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43a800d610 con 0x7f43b8147be0 2022-07-20T02:28:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b559a700 1 -- 172.21.15.26:0/3345363427 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8000e90 con 0x7f43b8147be0 2022-07-20T02:28:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b559a700 1 -- 172.21.15.26:0/3345363427 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43a800d960 con 0x7f43b8147be0 2022-07-20T02:28:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- 172.21.15.26:0/3345363427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b8147be0 msgr2=0x7f43b80aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 --2- 172.21.15.26:0/3345363427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b8147be0 0x7f43b80aee40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f43a8002f70 tx=0x7f43a8006df0).stop 2022-07-20T02:28:58.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- 172.21.15.26:0/3345363427 shutdown_connections 2022-07-20T02:28:58.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 --2- 172.21.15.26:0/3345363427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b8147be0 0x7f43b80aee40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- 172.21.15.26:0/3345363427 >> 172.21.15.26:0/3345363427 conn(0x7f43b8060310 msgr2=0x7f43b80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:58.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- 172.21.15.26:0/3345363427 shutdown_connections 2022-07-20T02:28:58.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- 172.21.15.26:0/3345363427 wait complete. 2022-07-20T02:28:58.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 Processor -- start 2022-07-20T02:28:58.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.824+0000 7f43b759e700 1 -- start start 2022-07-20T02:28:58.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b80b5ef0 0x7f43b80b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43b8147be0 0x7f43b8159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b8065660 con 0x7f43b80a7fc0 2022-07-20T02:28:58.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b80aaaf0 con 0x7f43b80b5ef0 2022-07-20T02:28:58.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b80ab6c0 con 0x7f43b8147be0 2022-07-20T02:28:58.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37526/0 (socket says 172.21.15.26:37526) 2022-07-20T02:28:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 -- 172.21.15.26:0/3878692579 learned_addr learned my addr 172.21.15.26:0/3878692579 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:58.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b5d9b700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b80b5ef0 0x7f43b80b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b6d9d700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43b8147be0 0x7f43b8159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 -- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43b8147be0 msgr2=0x7f43b8159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43b8147be0 0x7f43b8159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 -- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b80b5ef0 msgr2=0x7f43b80b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b80b5ef0 0x7f43b80b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:58.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 -- 172.21.15.26:0/3878692579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a800b180 con 0x7f43b80a7fc0 2022-07-20T02:28:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b659c700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f43b80b52b0 tx=0x7f43ac007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43ac0070d0 con 0x7f43b80a7fc0 2022-07-20T02:28:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b80aad50 con 0x7f43b80a7fc0 2022-07-20T02:28:58.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ac007230 con 0x7f43b80a7fc0 2022-07-20T02:28:58.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43ac005610 con 0x7f43b80a7fc0 2022-07-20T02:28:58.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b80ab380 con 0x7f43b80a7fc0 2022-07-20T02:28:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.828+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b814b2a0 con 0x7f43b80a7fc0 2022-07-20T02:28:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.832+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43ac010070 con 0x7f43b80a7fc0 2022-07-20T02:28:58.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.832+0000 7f43a77fe700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 0x7f4394055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:58.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.832+0000 7f43b5d9b700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 0x7f4394055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:58.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.832+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f43ac06bbd0 con 0x7f43b80a7fc0 2022-07-20T02:28:58.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.832+0000 7f43b5d9b700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 0x7f4394055ff0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f43a801a000 tx=0x7f43a8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:58.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:58.836+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43ac042b60 con 0x7f43b80a7fc0 2022-07-20T02:28:59.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.000+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7f43b80574e0 con 0x7f43b80a7fc0 2022-07-20T02:28:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.412+0000 7f43a77fe700 1 -- 172.21.15.26:0/3878692579 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v161) v1 ==== 144+0+0 (secure 0 0 0) 0x7f43ac007c60 con 0x7f43b80a7fc0 2022-07-20T02:28:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 msgr2=0x7f4394055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 0x7f4394055ff0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f43a801a000 tx=0x7f43a8014000).stop 2022-07-20T02:28:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 msgr2=0x7f43b80abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f43b80b52b0 tx=0x7f43ac007b40).stop 2022-07-20T02:28:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 shutdown_connections 2022-07-20T02:28:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4394053b30 0x7f4394055ff0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43b8147be0 0x7f43b8159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b80b5ef0 0x7f43b80b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 --2- 172.21.15.26:0/3878692579 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43b80a7fc0 0x7f43b80abd70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 >> 172.21.15.26:0/3878692579 conn(0x7f43b8060310 msgr2=0x7f43b8149dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 shutdown_connections 2022-07-20T02:28:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.416+0000 7f43b759e700 1 -- 172.21.15.26:0/3878692579 wait complete. 2022-07-20T02:28:59.434 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T02:28:59.435 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T02:28:59.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-20T02:28:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 Processor -- start 2022-07-20T02:28:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 -- start start 2022-07-20T02:28:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1910061860 0x7f1910067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191005ff70 con 0x7f19100b3260 2022-07-20T02:28:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100606e0 con 0x7f19100980d0 2022-07-20T02:28:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1916a2d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19100b3640 con 0x7f1910061860 2022-07-20T02:28:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1915a2b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f19100b3260 0x7f19100979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36814/0 (socket says 172.21.15.26:36814) 2022-07-20T02:28:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1915a2b700 1 -- 172.21.15.26:0/1364384308 learned_addr learned my addr 172.21.15.26:0/1364384308 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:28:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f191522a700 1 --2- 172.21.15.26:0/1364384308 >> v2:172.21.15.26:3300/0 conn(0x7f1910061860 0x7f1910067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1914a29700 1 -- 172.21.15.26:0/1364384308 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488600184 0 0) 0x7f191005ff70 con 0x7f19100b3260 2022-07-20T02:28:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1914a29700 1 -- 172.21.15.26:0/1364384308 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8003260 con 0x7f19100b3260 2022-07-20T02:28:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f191522a700 1 -- 172.21.15.26:0/1364384308 >> v1:172.21.15.120:6789/0 conn(0x7f19100980d0 legacy=0x7f1910142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f191522a700 1 -- 172.21.15.26:0/1364384308 >> v1:172.21.15.26:6789/0 conn(0x7f19100b3260 legacy=0x7f19100979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f191522a700 1 -- 172.21.15.26:0/1364384308 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100601a0 con 0x7f1910061860 2022-07-20T02:28:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f191522a700 1 --2- 172.21.15.26:0/1364384308 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f1910067900 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1900002f70 tx=0x7f1900006e80).ready entity=mon.0 client_cookie=106490a2b31548f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1914a29700 1 -- 172.21.15.26:0/1364384308 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1900014070 con 0x7f1910061860 2022-07-20T02:28:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1914a29700 1 -- 172.21.15.26:0/1364384308 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1900000e90 con 0x7f1910061860 2022-07-20T02:28:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.544+0000 7f1914a29700 1 -- 172.21.15.26:0/1364384308 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19000107f0 con 0x7f1910061860 2022-07-20T02:28:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- 172.21.15.26:0/1364384308 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 msgr2=0x7f1910067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 --2- 172.21.15.26:0/1364384308 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f1910067900 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1900002f70 tx=0x7f1900006e80).stop 2022-07-20T02:28:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- 172.21.15.26:0/1364384308 shutdown_connections 2022-07-20T02:28:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 --2- 172.21.15.26:0/1364384308 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f1910067900 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- 172.21.15.26:0/1364384308 >> 172.21.15.26:0/1364384308 conn(0x7f19100b67f0 msgr2=0x7f19100b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:28:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- 172.21.15.26:0/1364384308 shutdown_connections 2022-07-20T02:28:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- 172.21.15.26:0/1364384308 wait complete. 2022-07-20T02:28:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 Processor -- start 2022-07-20T02:28:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- start start 2022-07-20T02:28:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19100980d0 0x7f19101544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19100b3260 0x7f19100b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191013c0c0 con 0x7f1910061860 2022-07-20T02:28:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1910153dd0 con 0x7f19100b3260 2022-07-20T02:28:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1916a2d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1910154030 con 0x7f19100980d0 2022-07-20T02:28:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.548+0000 7f1915a2b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37540/0 (socket says 172.21.15.26:37540) 2022-07-20T02:28:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 -- 172.21.15.26:0/2433049170 learned_addr learned my addr 172.21.15.26:0/2433049170 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:28:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f191522a700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19100980d0 0x7f19101544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f191622c700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19100b3260 0x7f19100b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 -- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19100980d0 msgr2=0x7f19101544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19100980d0 0x7f19101544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 -- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19100b3260 msgr2=0x7f19100b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:28:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19100b3260 0x7f19100b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:28:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 -- 172.21.15.26:0/2433049170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190000c310 con 0x7f1910061860 2022-07-20T02:28:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1915a2b700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f190c001110 tx=0x7f190c007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190c0070d0 con 0x7f1910061860 2022-07-20T02:28:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c007230 con 0x7f1910061860 2022-07-20T02:28:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191013c2f0 con 0x7f1910061860 2022-07-20T02:28:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190c005610 con 0x7f1910061860 2022-07-20T02:28:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1910062f50 con 0x7f1910061860 2022-07-20T02:28:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.552+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1910060440 con 0x7f1910061860 2022-07-20T02:28:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.556+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f190c010070 con 0x7f1910061860 2022-07-20T02:28:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.556+0000 7f18feffd700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 0x7f18f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:28:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.556+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f190c06b560 con 0x7f1910061860 2022-07-20T02:28:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.556+0000 7f191522a700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 0x7f18f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:28:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.556+0000 7f191522a700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 0x7f18f0055fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f190001c000 tx=0x7f1910060330).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:28:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.564+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f190c0424f0 con 0x7f1910061860 2022-07-20T02:28:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:28:59.728+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f19100574e0 con 0x7f1910061860 2022-07-20T02:29:00.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.420+0000 7f18feffd700 1 -- 172.21.15.26:0/2433049170 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v162) v1 ==== 155+0+0 (secure 0 0 0) 0x7f190c045b10 con 0x7f1910061860 2022-07-20T02:29:00.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 msgr2=0x7f18f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 0x7f18f0055fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f190001c000 tx=0x7f1910060330).stop 2022-07-20T02:29:00.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 msgr2=0x7f191009b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f190c001110 tx=0x7f190c007b80).stop 2022-07-20T02:29:00.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 shutdown_connections 2022-07-20T02:29:00.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18f0053b00 0x7f18f0055fc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19100b3260 0x7f19100b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19100980d0 0x7f19101544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 --2- 172.21.15.26:0/2433049170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1910061860 0x7f191009b330 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 >> 172.21.15.26:0/2433049170 conn(0x7f19100b67f0 msgr2=0x7f19100b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:00.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 shutdown_connections 2022-07-20T02:29:00.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.424+0000 7f1916a2d700 1 -- 172.21.15.26:0/2433049170 wait complete. 2022-07-20T02:29:00.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:29:00.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 Processor -- start 2022-07-20T02:29:00.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 -- start start 2022-07-20T02:29:00.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe52c05cf80 0x7fe52c143040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0ab530 con 0x7fe52c140320 2022-07-20T02:29:00.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c0b1ce0 con 0x7fe52c05cf80 2022-07-20T02:29:00.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.560+0000 7fe532bcb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c05cc90 con 0x7fe52c147020 2022-07-20T02:29:00.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe531bc9700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe52c140320 0x7fe52c05c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36828/0 (socket says 172.21.15.26:36828) 2022-07-20T02:29:00.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe52c05cf80 0x7fe52c143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe531bc9700 1 -- 172.21.15.26:0/2451325080 learned_addr learned my addr 172.21.15.26:0/2451325080 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:00.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5313c8700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe530bc7700 1 -- 172.21.15.26:0/2451325080 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828401417 0 0) 0x7fe528001110 con 0x7fe52c140320 2022-07-20T02:29:00.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe530bc7700 1 -- 172.21.15.26:0/2451325080 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514003260 con 0x7fe52c140320 2022-07-20T02:29:00.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 -- 172.21.15.26:0/2451325080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 msgr2=0x7fe52c140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:00.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c140700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 -- 172.21.15.26:0/2451325080 >> v1:172.21.15.26:6789/0 conn(0x7fe52c140320 legacy=0x7fe52c05c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 -- 172.21.15.26:0/2451325080 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0aab50 con 0x7fe52c05cf80 2022-07-20T02:29:00.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe5323ca700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c143040 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fe5240032f0 tx=0x7fe5240091d0).ready entity=mon.0 client_cookie=7228419419336e1f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:00.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe530bc7700 1 -- 172.21.15.26:0/2451325080 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52400b600 con 0x7fe52c05cf80 2022-07-20T02:29:00.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe530bc7700 1 -- 172.21.15.26:0/2451325080 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400b760 con 0x7fe52c05cf80 2022-07-20T02:29:00.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe530bc7700 1 -- 172.21.15.26:0/2451325080 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52400b8d0 con 0x7fe52c05cf80 2022-07-20T02:29:00.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- 172.21.15.26:0/2451325080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 msgr2=0x7fe52c143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c143040 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fe5240032f0 tx=0x7fe5240091d0).stop 2022-07-20T02:29:00.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- 172.21.15.26:0/2451325080 shutdown_connections 2022-07-20T02:29:00.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c143040 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 --2- 172.21.15.26:0/2451325080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c140700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- 172.21.15.26:0/2451325080 >> 172.21.15.26:0/2451325080 conn(0x7fe52c060310 msgr2=0x7fe52c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:00.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- 172.21.15.26:0/2451325080 shutdown_connections 2022-07-20T02:29:00.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- 172.21.15.26:0/2451325080 wait complete. 2022-07-20T02:29:00.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 Processor -- start 2022-07-20T02:29:00.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.564+0000 7fe532bcb700 1 -- start start 2022-07-20T02:29:00.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 0x7fe52c146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0a0740 con 0x7fe52c05cf80 2022-07-20T02:29:00.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c09ff60 con 0x7fe52c140320 2022-07-20T02:29:00.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0a0200 con 0x7fe52c147020 2022-07-20T02:29:00.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37554/0 (socket says 172.21.15.26:37554) 2022-07-20T02:29:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 -- 172.21.15.26:0/3876702408 learned_addr learned my addr 172.21.15.26:0/3876702408 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe5313c8700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 0x7fe52c146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe5323ca700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 -- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 msgr2=0x7fe52c0ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c0ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 -- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 msgr2=0x7fe52c146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 0x7fe52c146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 -- 172.21.15.26:0/3876702408 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009470 con 0x7fe52c05cf80 2022-07-20T02:29:00.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe5313c8700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 0x7fe52c146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:00.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe5323ca700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:00.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe531bc9700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe5280012a0 tx=0x7fe5280079a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5280070d0 con 0x7fe52c05cf80 2022-07-20T02:29:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528007230 con 0x7fe52c05cf80 2022-07-20T02:29:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c0a04a0 con 0x7fe52c05cf80 2022-07-20T02:29:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe528005610 con 0x7fe52c05cf80 2022-07-20T02:29:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c0b0ba0 con 0x7fe52c05cf80 2022-07-20T02:29:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.568+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c0af0b0 con 0x7fe52c05cf80 2022-07-20T02:29:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.572+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe528010070 con 0x7fe52c05cf80 2022-07-20T02:29:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.572+0000 7fe51affd700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 0x7fe50c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.572+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe52806b270 con 0x7fe52c05cf80 2022-07-20T02:29:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.572+0000 7fe5313c8700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 0x7fe50c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.572+0000 7fe5313c8700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 0x7fe50c055ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fe51c00b000 tx=0x7fe51c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.580+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe528042200 con 0x7fe52c05cf80 2022-07-20T02:29:00.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.712+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe52c0ab530 con 0x7fe52c05cf80 2022-07-20T02:29:00.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.720+0000 7fe51affd700 1 -- 172.21.15.26:0/3876702408 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe52c0ab530 con 0x7fe52c05cf80 2022-07-20T02:29:00.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 msgr2=0x7fe50c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 0x7fe50c055ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fe51c00b000 tx=0x7fe51c006df0).stop 2022-07-20T02:29:00.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 msgr2=0x7fe52c0af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:00.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe5280012a0 tx=0x7fe5280079a0).stop 2022-07-20T02:29:00.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 shutdown_connections 2022-07-20T02:29:00.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe50c0539e0 0x7fe50c055ea0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe52c147020 0x7fe52c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe52c140320 0x7fe52c146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 --2- 172.21.15.26:0/3876702408 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe52c05cf80 0x7fe52c0af9d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:00.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 >> 172.21.15.26:0/3876702408 conn(0x7fe52c060310 msgr2=0x7fe52c0ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:00.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 shutdown_connections 2022-07-20T02:29:00.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:00.724+0000 7fe532bcb700 1 -- 172.21.15.26:0/3876702408 wait complete. 2022-07-20T02:29:00.739 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:29:00.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:29:00.760 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:29:00.783 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:29:00.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:29:00.842 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:29:00.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:00.844 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:00.858 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:29:00.877 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:29:00.877 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:29:00.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:00.896 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:00.899 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:29:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:29:00.917 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:29:00.919 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:29:00.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:29:01.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 Processor -- start 2022-07-20T02:29:01.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- start start 2022-07-20T02:29:01.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f11cc05cfd0 0x7f11cc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11cc060f00 0x7f11cc13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc1474e0 con 0x7f11cc147000 2022-07-20T02:29:01.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc0b4e40 con 0x7f11cc060f00 2022-07-20T02:29:01.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc05d3b0 con 0x7f11cc05cfd0 2022-07-20T02:29:01.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11cc060f00 0x7f11cc13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11cbfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f11cc147000 0x7f11cc0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36846/0 (socket says 172.21.15.26:36846) 2022-07-20T02:29:01.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d0e7a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f11cc05cfd0 0x7f11cc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11cbfff700 1 -- 172.21.15.26:0/36972780 learned_addr learned my addr 172.21.15.26:0/36972780 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:01.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11cc060f00 0x7f11cc13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37564/0 (socket says 172.21.15.26:37564) 2022-07-20T02:29:01.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 -- 172.21.15.26:0/36972780 >> v2:172.21.15.120:3300/0 conn(0x7f11cc05cfd0 msgr2=0x7f11cc143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:01.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 --2- 172.21.15.26:0/36972780 >> v2:172.21.15.120:3300/0 conn(0x7f11cc05cfd0 0x7f11cc143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 -- 172.21.15.26:0/36972780 >> v1:172.21.15.26:6789/0 conn(0x7f11cc147000 legacy=0x7f11cc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 -- 172.21.15.26:0/36972780 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc1431e0 con 0x7f11cc060f00 2022-07-20T02:29:01.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d167b700 1 --2- 172.21.15.26:0/36972780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc060f00 0x7f11cc13c9b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f11c40032f0 tx=0x7f11c40091d0).ready entity=mon.0 client_cookie=705718260a8a2876 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11cb7fe700 1 -- 172.21.15.26:0/36972780 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c400b680 con 0x7f11cc060f00 2022-07-20T02:29:01.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11cb7fe700 1 -- 172.21.15.26:0/36972780 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c400b7e0 con 0x7f11cc060f00 2022-07-20T02:29:01.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11cb7fe700 1 -- 172.21.15.26:0/36972780 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c400b950 con 0x7f11cc060f00 2022-07-20T02:29:01.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- 172.21.15.26:0/36972780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc060f00 msgr2=0x7f11cc13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/36972780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc060f00 0x7f11cc13c9b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f11c40032f0 tx=0x7f11c40091d0).stop 2022-07-20T02:29:01.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- 172.21.15.26:0/36972780 shutdown_connections 2022-07-20T02:29:01.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/36972780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc060f00 0x7f11cc13c9b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/36972780 >> v2:172.21.15.120:3300/0 conn(0x7f11cc05cfd0 0x7f11cc143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.036+0000 7f11d1e7c700 1 -- 172.21.15.26:0/36972780 >> 172.21.15.26:0/36972780 conn(0x7f11cc0620c0 msgr2=0x7f11cc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:01.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- 172.21.15.26:0/36972780 shutdown_connections 2022-07-20T02:29:01.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- 172.21.15.26:0/36972780 wait complete. 2022-07-20T02:29:01.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 Processor -- start 2022-07-20T02:29:01.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- start start 2022-07-20T02:29:01.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11cc060f00 0x7f11cc155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11cc147000 0x7f11cc1418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0ae990 con 0x7f11cc0ad690 2022-07-20T02:29:01.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0ae2b0 con 0x7f11cc147000 2022-07-20T02:29:01.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d1e7c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0ae050 con 0x7f11cc060f00 2022-07-20T02:29:01.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d167b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d167b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37568/0 (socket says 172.21.15.26:37568) 2022-07-20T02:29:01.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d167b700 1 -- 172.21.15.26:0/1644696675 learned_addr learned my addr 172.21.15.26:0/1644696675 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:01.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11cbfff700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11cc147000 0x7f11cc1418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.040+0000 7f11d0e7a700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11cc060f00 0x7f11cc155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 -- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11cc060f00 msgr2=0x7f11cc155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11cc060f00 0x7f11cc155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 -- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11cc147000 msgr2=0x7f11cc1418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11cc147000 0x7f11cc1418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 -- 172.21.15.26:0/1644696675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4009470 con 0x7f11cc0ad690 2022-07-20T02:29:01.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d167b700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f11c40032f0 tx=0x7f11c4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c400b680 con 0x7f11cc0ad690 2022-07-20T02:29:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4021630 con 0x7f11cc0ad690 2022-07-20T02:29:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc0ae6f0 con 0x7f11cc0ad690 2022-07-20T02:29:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c4021990 con 0x7f11cc0ad690 2022-07-20T02:29:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc142d10 con 0x7f11cc0ad690 2022-07-20T02:29:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc0574e0 con 0x7f11cc0ad690 2022-07-20T02:29:01.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11c4027020 con 0x7f11cc0ad690 2022-07-20T02:29:01.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 0x7f11ac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f11c406d000 con 0x7f11cc0ad690 2022-07-20T02:29:01.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d0e7a700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 0x7f11ac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.044+0000 7f11d0e7a700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 0x7f11ac055f30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f11bc00b000 tx=0x7f11bc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.048+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11c40440a0 con 0x7f11cc0ad690 2022-07-20T02:29:01.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.172+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f11cc0683d0 con 0x7f11cc0ad690 2022-07-20T02:29:01.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.172+0000 7f11c9ffb700 1 -- 172.21.15.26:0/1644696675 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v48) v1 ==== 81+0+29267 (secure 0 0 0) 0x7f11c4015020 con 0x7f11cc0ad690 2022-07-20T02:29:01.176 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:01.176 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:29:01.176 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 48, 2022-07-20T02:29:01.176 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:29:01.177 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:29:01.177 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:28:58.396943+0000", 2022-07-20T02:29:01.177 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:29:01.177 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:29:01.178 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:29:01.178 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:29:01.178 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:29:01.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:29:01.178 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:29:01.179 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:29:01.179 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:29:01.179 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:29:01.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:29:01.180 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:29:01.180 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:29:01.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:29:01.180 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:29:01.180 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 8, 2022-07-20T02:29:01.181 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:29:01.181 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:29:01.181 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:29:01.182 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:29:01.182 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:29:01.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:29:01.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:29:01.183 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:29:01.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:29:01.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:29:01.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:29:01.184 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:29:01.184 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:29:01.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:29:01.185 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:29:01.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:29:01.186 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:29:01.186 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:29:01.186 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:29:01.187 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:29:01.187 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:29:01.187 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:29:01.187 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:29:01.192 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:29:01.192 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:29:01.192 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:29:01.193 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:29:01.194 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:29:01.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:29:01.195 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:29:01.195 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:29:01.195 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:29:01.196 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:29:01.196 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:29:01.196 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:29:01.196 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:29:01.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:29:01.197 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:29:01.197 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:29:01.197 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:29:01.197 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:29:01.198 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:29:01.198 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:29:01.198 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:29:01.199 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:29:01.199 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:29:01.199 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:29:01.199 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:29:01.199 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:29:01.200 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:29:01.200 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:29:01.200 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:29:01.200 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:29:01.201 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:29:01.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:29:01.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.201 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:29:01.202 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:29:01.202 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:29:01.202 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:29:01.202 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:29:01.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:29:01.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:29:01.204 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:29:01.205 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:29:01.205 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:29:01.205 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:29:01.205 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:29:01.206 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:29:01.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.206 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:29:01.206 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:29:01.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.207 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:29:01.208 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:29:01.208 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:29:01.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:29:01.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:29:01.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:29:01.209 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:29:01.209 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:29:01.209 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:29:01.209 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:29:01.210 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:29:01.210 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:29:01.210 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:29:01.211 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:29:01.211 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:29:01.211 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:29:01.211 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:29:01.211 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:29:01.212 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:29:01.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:29:01.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:29:01.212 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:29:01.213 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:29:01.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.213 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:29:01.213 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:29:01.214 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:29:01.214 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:29:01.214 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:29:01.214 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:29:01.215 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:29:01.215 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:29:01.215 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:29:01.215 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:29:01.215 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:29:01.216 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:29:01.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:29:01.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:29:01.217 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:29:01.217 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:29:01.217 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:29:01.217 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:29:01.218 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:29:01.218 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:29:01.219 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:29:01.219 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:29:01.219 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:29:01.220 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:29:01.220 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:29:01.220 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:29:01.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:29:01.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.222 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:29:01.222 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:29:01.222 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:29:01.223 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:29:01.223 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:29:01.223 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:29:01.224 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:29:01.224 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:29:01.224 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:29:01.224 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:29:01.225 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:29:01.225 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:29:01.225 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:29:01.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:29:01.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:29:01.226 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:29:01.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:29:01.227 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:29:01.227 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.228 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.228 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.228 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.228 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.229 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.229 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.229 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.229 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.230 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.230 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:29:01.233 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:29:01.235 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.237 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:29:01.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:29:01.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.267 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:29:01.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:29:01.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.273 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:29:01.275 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:29:01.279 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:29:01.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:29:01.281 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:29:01.281 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:29:01.282 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:29:01.282 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.283 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.283 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:29:01.285 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:29:01.286 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.287 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.287 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.288 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.288 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.289 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.289 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.290 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.290 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.291 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.291 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:29:01.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.293 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:29:01.293 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.294 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.294 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.295 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:29:01.295 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:29:01.296 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.297 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.298 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.298 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:29:01.300 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:29:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:29:01.303 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:29:01.304 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:29:01.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.305 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:29:01.305 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:29:01.306 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:29:01.306 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:29:01.306 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.306 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.306 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.308 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:29:01.308 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:29:01.309 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.309 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.309 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.309 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.309 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.310 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.310 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.310 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 31, 2022-07-20T02:29:01.311 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.311 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.311 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.311 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:29:01.312 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:29:01.314 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.320 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.320 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.321 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:29:01.322 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:29:01.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.323 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.324 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:29:01.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:29:01.326 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.327 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:29:01.328 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:29:01.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:29:01.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:29:01.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:29:01.331 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:29:01.331 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:29:01.331 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.331 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.333 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:29:01.333 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:29:01.333 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.333 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.333 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.334 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.334 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.334 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.334 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.335 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.335 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.335 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:29:01.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:29:01.338 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.339 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:29:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:29:01.346 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.348 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:29:01.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:29:01.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.354 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:29:01.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:29:01.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:29:01.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:29:01.361 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:29:01.361 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:29:01.361 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:29:01.361 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.362 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.362 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.363 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:29:01.363 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:29:01.363 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.367 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.367 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.368 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.368 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.368 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.369 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.369 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:29:01.370 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:29:01.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.373 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:29:01.374 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:29:01.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.376 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:29:01.378 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:29:01.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.380 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.381 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:29:01.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:29:01.382 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:29:01.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.383 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.383 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:29:01.384 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:29:01.384 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:29:01.384 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:29:01.384 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.384 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.385 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.386 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:29:01.386 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:29:01.386 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.386 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.387 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.387 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.387 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.387 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.388 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.388 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.388 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.388 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:29:01.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:29:01.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.392 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:29:01.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:29:01.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.394 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.394 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.395 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:29:01.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:29:01.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.398 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:29:01.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:29:01.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:29:01.401 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.401 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:29:01.402 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:29:01.402 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:29:01.402 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:29:01.403 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.403 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.403 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.404 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:29:01.404 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:29:01.404 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.405 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.405 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.405 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.405 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.405 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.406 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.406 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 43, 2022-07-20T02:29:01.406 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.406 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.407 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:29:01.408 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:29:01.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.410 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.410 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:29:01.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:29:01.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.413 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.414 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:29:01.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:29:01.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.417 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:29:01.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:29:01.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:29:01.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:29:01.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:29:01.421 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:29:01.421 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:29:01.421 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.421 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.421 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.422 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:29:01.423 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:29:01.423 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:29:01.423 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:29:01.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:29:01.424 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:29:01.424 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:29:01.424 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:29:01.424 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:29:01.424 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 42, 2022-07-20T02:29:01.425 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:29:01.425 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:29:01.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:29:01.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:29:01.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:29:01.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.434 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:29:01.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:29:01.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:29:01.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.441 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:29:01.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:29:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:29:01.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.446 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:29:01.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:29:01.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:29:01.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:29:01.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:29:01.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:29:01.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:29:01.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:29:01.457 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:29:01.458 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:29:01.458 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:29:01.459 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:29:01.459 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:29:01.460 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:29:01.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:01.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.461 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:29:01.461 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:29:01.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.462 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:29:01.463 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.463 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.464 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.464 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.465 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.465 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:29:01.465 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.467 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:29:01.467 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.468 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.468 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.468 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.469 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.469 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:29:01.469 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.471 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:29:01.471 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.472 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.472 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.473 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.474 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:29:01.474 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.476 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:29:01.476 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.476 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.477 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.477 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.478 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.478 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:29:01.478 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.480 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:29:01.480 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.480 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.481 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.481 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.482 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.482 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:29:01.483 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.484 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:29:01.484 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.485 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.485 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.486 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.486 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.487 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:29:01.487 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.488 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:29:01.489 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.489 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.490 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.491 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.491 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:29:01.492 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:01.493 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:29:01.493 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:29:01.494 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:29:01.495 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:29:01.496 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:29:01.496 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:29:01.496 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:29:01.497 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:29:01.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.498 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:29:01.498 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:29:01.499 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:29:01.499 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:29:01.500 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:29:01.500 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:29:01.500 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/3286368975": "2022-07-21T02:28:54.446182+0000", 2022-07-20T02:29:01.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/3286368975": "2022-07-21T02:28:54.446182+0000" 2022-07-20T02:29:01.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.502 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:29:01.502 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:29:01.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:29:01.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:29:01.504 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:29:01.504 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:29:01.505 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:29:01.505 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:29:01.506 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:29:01.506 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:29:01.507 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:29:01.507 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:29:01.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.508 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:29:01.509 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:29:01.509 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:29:01.509 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:29:01.510 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:29:01.510 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:29:01.511 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:29:01.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:01.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:01.512 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:29:01.513 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:29:01.513 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:29:01.513 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:29:01.520 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:29:01.520 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:29:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 msgr2=0x7f11ac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 0x7f11ac055f30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f11bc00b000 tx=0x7f11bc006df0).stop 2022-07-20T02:29:01.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 msgr2=0x7f11cc141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f11c40032f0 tx=0x7f11c4016000).stop 2022-07-20T02:29:01.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 shutdown_connections 2022-07-20T02:29:01.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11cc0ad690 0x7f11cc141dc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11ac053a70 0x7f11ac055f30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11cc147000 0x7f11cc1418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 --2- 172.21.15.26:0/1644696675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11cc060f00 0x7f11cc155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 >> 172.21.15.26:0/1644696675 conn(0x7f11cc05cfd0 msgr2=0x7f11cc13a660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:01.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 shutdown_connections 2022-07-20T02:29:01.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.176+0000 7f11d1e7c700 1 -- 172.21.15.26:0/1644696675 wait complete. 2022-07-20T02:29:01.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:01.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 Processor -- start 2022-07-20T02:29:01.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 -- start start 2022-07-20T02:29:01.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6bf80b7d80 0x7f6bf80aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf814b470 con 0x7f6bf80b4b80 2022-07-20T02:29:01.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf8064650 con 0x7f6bf8149ce0 2022-07-20T02:29:01.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfdfc2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf80b4ff0 con 0x7f6bf80b7d80 2022-07-20T02:29:01.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6bf80b4b80 0x7f6bf814ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36856/0 (socket says 172.21.15.26:36856) 2022-07-20T02:29:01.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf7fff700 1 -- 172.21.15.26:0/2713158949 learned_addr learned my addr 172.21.15.26:0/2713158949 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:01.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfcfc0700 1 --2- 172.21.15.26:0/2713158949 >> v2:172.21.15.120:3300/0 conn(0x7f6bf80b7d80 0x7f6bf80aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf77fe700 1 -- 172.21.15.26:0/2713158949 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708389843 0 0) 0x7f6bf814b470 con 0x7f6bf80b4b80 2022-07-20T02:29:01.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf77fe700 1 -- 172.21.15.26:0/2713158949 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be4003260 con 0x7f6bf80b4b80 2022-07-20T02:29:01.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/2713158949 >> v1:172.21.15.120:6789/0 conn(0x7f6bf8149ce0 legacy=0x7f6bf814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/2713158949 >> v1:172.21.15.26:6789/0 conn(0x7f6bf80b4b80 legacy=0x7f6bf814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/2713158949 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf80b41a0 con 0x7f6bf80b7d80 2022-07-20T02:29:01.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bfcfc0700 1 --2- 172.21.15.26:0/2713158949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf80b7d80 0x7f6bf80aaf00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6be80032f0 tx=0x7f6be80091d0).ready entity=mon.1 client_cookie=e86e14ed848b55b4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf77fe700 1 -- 172.21.15.26:0/2713158949 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be800b6c0 con 0x7f6bf80b7d80 2022-07-20T02:29:01.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf77fe700 1 -- 172.21.15.26:0/2713158949 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be800b820 con 0x7f6bf80b7d80 2022-07-20T02:29:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.652+0000 7f6bf77fe700 1 -- 172.21.15.26:0/2713158949 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be800b990 con 0x7f6bf80b7d80 2022-07-20T02:29:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/2713158949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf80b7d80 msgr2=0x7f6bf80aaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/2713158949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf80b7d80 0x7f6bf80aaf00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6be80032f0 tx=0x7f6be80091d0).stop 2022-07-20T02:29:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/2713158949 shutdown_connections 2022-07-20T02:29:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/2713158949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf80b7d80 0x7f6bf80aaf00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/2713158949 >> 172.21.15.26:0/2713158949 conn(0x7f6bf805f840 msgr2=0x7f6bf805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:01.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/2713158949 shutdown_connections 2022-07-20T02:29:01.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/2713158949 wait complete. 2022-07-20T02:29:01.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 Processor -- start 2022-07-20T02:29:01.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- start start 2022-07-20T02:29:01.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bf8149ce0 0x7f6bf80af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf809f6b0 0x7f6bf80af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf80b1270 con 0x7f6bf80b4b80 2022-07-20T02:29:01.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8156760 con 0x7f6bf809f6b0 2022-07-20T02:29:01.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfdfc2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8157720 con 0x7f6bf8149ce0 2022-07-20T02:29:01.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfcfc0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfcfc0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37582/0 (socket says 172.21.15.26:37582) 2022-07-20T02:29:01.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/3299062036 learned_addr learned my addr 172.21.15.26:0/3299062036 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:01.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bf7fff700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bf8149ce0 0x7f6bf80af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.656+0000 7f6bfd7c1700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf809f6b0 0x7f6bf80af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bf8149ce0 msgr2=0x7f6bf80af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bf8149ce0 0x7f6bf80af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf809f6b0 msgr2=0x7f6bf80af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf809f6b0 0x7f6bf80af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 -- 172.21.15.26:0/3299062036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8009470 con 0x7f6bf80b4b80 2022-07-20T02:29:01.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfcfc0700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f6be80032f0 tx=0x7f6be8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be800b6c0 con 0x7f6bf80b4b80 2022-07-20T02:29:01.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8021630 con 0x7f6bf80b4b80 2022-07-20T02:29:01.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf80ae1b0 con 0x7f6bf80b4b80 2022-07-20T02:29:01.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be8021990 con 0x7f6bf80b4b80 2022-07-20T02:29:01.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf80ae3e0 con 0x7f6bf80b4b80 2022-07-20T02:29:01.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.660+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf80aef90 con 0x7f6bf80b4b80 2022-07-20T02:29:01.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.664+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6be8028070 con 0x7f6bf80b4b80 2022-07-20T02:29:01.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.664+0000 7f6bf5ffb700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 0x7f6bd8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.664+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6be806cd50 con 0x7f6bf80b4b80 2022-07-20T02:29:01.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.664+0000 7f6bf7fff700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 0x7f6bd8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.664+0000 7f6bf7fff700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 0x7f6bd8055e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6bec00d000 tx=0x7f6bec009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.672+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6be8043ce0 con 0x7f6bf80b4b80 2022-07-20T02:29:01.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf80b4440 con 0x7f6bf80b4b80 2022-07-20T02:29:01.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bf5ffb700 1 -- 172.21.15.26:0/3299062036 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1565 (secure 0 0 0) 0x7f6be8015070 con 0x7f6bf80b4b80 2022-07-20T02:29:01.834 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:01.834 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/3096933944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3096933944},{"type":"v1","addr":"172.21.15.26:6837","nonce":3096933944}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-20T02:29:01.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 msgr2=0x7f6bd8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 0x7f6bd8055e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6bec00d000 tx=0x7f6bec009040).stop 2022-07-20T02:29:01.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 msgr2=0x7f6bf809f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f6be80032f0 tx=0x7f6be8016000).stop 2022-07-20T02:29:01.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 shutdown_connections 2022-07-20T02:29:01.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bd8053950 0x7f6bd8055e10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bf809f6b0 0x7f6bf80af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bf8149ce0 0x7f6bf80af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 --2- 172.21.15.26:0/3299062036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bf80b4b80 0x7f6bf809f1c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 >> 172.21.15.26:0/3299062036 conn(0x7f6bf805f840 msgr2=0x7f6bf80b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:01.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.828+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 shutdown_connections 2022-07-20T02:29:01.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.832+0000 7f6bfdfc2700 1 -- 172.21.15.26:0/3299062036 wait complete. 2022-07-20T02:29:01.855 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 20 2022-07-20T02:29:01.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:01.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.960+0000 7f96cc756700 1 Processor -- start 2022-07-20T02:29:01.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.960+0000 7f96cc756700 1 -- start start 2022-07-20T02:29:01.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96c4067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c40b1bc0 con 0x7f96c41484a0 2022-07-20T02:29:01.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c4147700 con 0x7f96c40b2920 2022-07-20T02:29:01.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c414b040 con 0x7f96c4061050 2022-07-20T02:29:01.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c9cf1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f96c41484a0 0x7f96c4066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36870/0 (socket says 172.21.15.26:36870) 2022-07-20T02:29:01.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c9cf1700 1 -- 172.21.15.26:0/2524390622 learned_addr learned my addr 172.21.15.26:0/2524390622 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:01.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cacf3700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96c4067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399250165 0 0) 0x7f96c40b1bc0 con 0x7f96c41484a0 2022-07-20T02:29:01.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac003260 con 0x7f96c41484a0 2022-07-20T02:29:01.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296703635 0 0) 0x7f96c4147700 con 0x7f96c40b2920 2022-07-20T02:29:01.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c40b1bc0 con 0x7f96c40b2920 2022-07-20T02:29:01.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602775813 0 0) 0x7f96ac003260 con 0x7f96c41484a0 2022-07-20T02:29:01.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 msgr2=0x7f96c4067190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:01.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96c4067190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 >> v1:172.21.15.120:6789/0 conn(0x7f96c40b2920 legacy=0x7f96c4147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4147930 con 0x7f96c41484a0 2022-07-20T02:29:01.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f96c0001660 con 0x7f96c41484a0 2022-07-20T02:29:01.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 >> v1:172.21.15.26:6789/0 conn(0x7f96c41484a0 legacy=0x7f96c4066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96ac003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac003e20 0x7f96ac004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96ac0049f0 0x7f96ac004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c414b040 con 0x7f96ac0049f0 2022-07-20T02:29:01.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac00b800 con 0x7f96ac003e20 2022-07-20T02:29:01.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac00b9c0 con 0x7f96c4061050 2022-07-20T02:29:01.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96ac003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cacf3700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96ac0049f0 0x7f96ac004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c9cf1700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac003e20 0x7f96ac004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:01.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac003e20 msgr2=0x7f96ac004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:01.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96ac003e20 0x7f96ac004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2524390622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96ac0049f0 msgr2=0x7f96ac004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:01.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96ac0049f0 0x7f96ac004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:01.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2524390622 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac003260 con 0x7f96c4061050 2022-07-20T02:29:01.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96ac003910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f96b4001110 tx=0x7f96b4007a00).ready entity=mon.2 client_cookie=64f3ad606029ed8b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:01.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b4007d10 con 0x7f96c4061050 2022-07-20T02:29:01.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b4007030 con 0x7f96c4061050 2022-07-20T02:29:01.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96c94f0700 1 -- 172.21.15.26:0/2524390622 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b4007390 con 0x7f96c4061050 2022-07-20T02:29:01.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 msgr2=0x7f96ac003910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:01.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 --2- 172.21.15.26:0/2524390622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c4061050 0x7f96ac003910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f96b4001110 tx=0x7f96b4007a00).stop 2022-07-20T02:29:01.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cacf3700 1 -- 172.21.15.26:0/2524390622 reap_dead start 2022-07-20T02:29:01.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- 172.21.15.26:0/2524390622 shutdown_connections 2022-07-20T02:29:01.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.964+0000 7f96cc756700 1 -- 172.21.15.26:0/2524390622 >> 172.21.15.26:0/2524390622 conn(0x7f96c40b60a0 msgr2=0x7f96c40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:01.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- 172.21.15.26:0/2524390622 shutdown_connections 2022-07-20T02:29:01.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- 172.21.15.26:0/2524390622 wait complete. 2022-07-20T02:29:01.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 Processor -- start 2022-07-20T02:29:01.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- start start 2022-07-20T02:29:02.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c40ac790 0x7f96c4156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c415a790 0x7f96c414e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40ae840 con 0x7f96c4061050 2022-07-20T02:29:02.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40aed10 con 0x7f96c415a790 2022-07-20T02:29:02.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cc756700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4156e60 con 0x7f96c40ac790 2022-07-20T02:29:02.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37602/0 (socket says 172.21.15.26:37602) 2022-07-20T02:29:02.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2036567195 learned_addr learned my addr 172.21.15.26:0/2036567195 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:02.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96c9cf1700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c40ac790 0x7f96c4156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96cacf3700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c415a790 0x7f96c414e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c40ac790 msgr2=0x7f96c4156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c40ac790 0x7f96c4156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c415a790 msgr2=0x7f96c414e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c415a790 0x7f96c414e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 -- 172.21.15.26:0/2036567195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c40aea70 con 0x7f96c4061050 2022-07-20T02:29:02.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.968+0000 7f96ca4f2700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f96c40ac1c0 tx=0x7f96b4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:02.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b401f480 con 0x7f96c4061050 2022-07-20T02:29:02.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b401f5e0 con 0x7f96c4061050 2022-07-20T02:29:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c415a050 con 0x7f96c4061050 2022-07-20T02:29:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b401f940 con 0x7f96c4061050 2022-07-20T02:29:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c40ac400 con 0x7f96c4061050 2022-07-20T02:29:02.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c414ada0 con 0x7f96c4061050 2022-07-20T02:29:02.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f96b4026030 con 0x7f96c4061050 2022-07-20T02:29:02.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 0x7f96a4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f96b406d0c0 con 0x7f96c4061050 2022-07-20T02:29:02.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.972+0000 7f96c9cf1700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 0x7f96a4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.976+0000 7f96c9cf1700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 0x7f96a4055f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f96c0005000 tx=0x7f96c414ac90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:02.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:01.980+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f96b40440c0 con 0x7f96c4061050 2022-07-20T02:29:02.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.140+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c414ada0 con 0x7f96c4061050 2022-07-20T02:29:02.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.144+0000 7f96b37fe700 1 -- 172.21.15.26:0/2036567195 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1565 (secure 0 0 0) 0x7f96b4024020 con 0x7f96c4061050 2022-07-20T02:29:02.147 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:02.147 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5209,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2829658465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2829658465},{"type":"v1","addr":"172.21.15.120:6833","nonce":2829658465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1827291841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1827291841},{"type":"v1","addr":"172.21.15.26:6835","nonce":1827291841}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/3096933944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3096933944},{"type":"v1","addr":"172.21.15.26:6837","nonce":3096933944}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-20T02:29:02.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 msgr2=0x7f96a4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 0x7f96a4055f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f96c0005000 tx=0x7f96c414ac90).stop 2022-07-20T02:29:02.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 msgr2=0x7f96c40b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f96c40ac1c0 tx=0x7f96b4016000).stop 2022-07-20T02:29:02.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 shutdown_connections 2022-07-20T02:29:02.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f96a4053a70 0x7f96a4055f30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c415a790 0x7f96c414e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96c40ac790 0x7f96c4156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 --2- 172.21.15.26:0/2036567195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f96c4061050 0x7f96c40b4c30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 >> 172.21.15.26:0/2036567195 conn(0x7f96c40b60a0 msgr2=0x7f96c4066bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:02.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 shutdown_connections 2022-07-20T02:29:02.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.148+0000 7f96cc756700 1 -- 172.21.15.26:0/2036567195 wait complete. 2022-07-20T02:29:02.160 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 20 2022-07-20T02:29:02.171 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:29:02.171 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:29:02.172 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:29:02.172 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:29:02.173 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:29:02.173 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:29:02.173 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:29:02.174 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:29:02.174 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:29:02.174 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:29:02.175 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:29:02.175 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:29:02.187 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:29:02.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:29:02.196 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:29:02.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:29:02.199 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:29:02.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:29:02.203 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:29:02.204 DEBUG:teuthology.parallel:result is None 2022-07-20T02:29:02.250 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:29:02.251 DEBUG:teuthology.parallel:result is None 2022-07-20T02:29:02.253 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:29:02.253 DEBUG:teuthology.parallel:result is None 2022-07-20T02:29:02.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:29:02.280 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:29:02.318 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:29:02.326 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:29:02.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 Processor -- start 2022-07-20T02:29:02.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 -- start start 2022-07-20T02:29:02.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62c80aff40 0x7f62c8066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80b0f50 con 0x7f62c80621f0 2022-07-20T02:29:02.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c805f8e0 con 0x7f62c8060250 2022-07-20T02:29:02.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cedbb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c8060dd0 con 0x7f62c80aff40 2022-07-20T02:29:02.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cddb9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62c80aff40 0x7f62c8066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cddb9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62c80aff40 0x7f62c8066b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37634/0 (socket says 172.21.15.26:37634) 2022-07-20T02:29:02.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cddb9700 1 -- 172.21.15.26:0/1389955437 learned_addr learned my addr 172.21.15.26:0/1389955437 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:02.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.352+0000 7f62cd5b8700 1 --1- 172.21.15.26:0/1389955437 >> v1:172.21.15.26:6789/0 conn(0x7f62c80621f0 0x7f62c8067020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36916/0 (socket says 172.21.15.26:36916) 2022-07-20T02:29:02.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cddb9700 1 -- 172.21.15.26:0/1389955437 >> v1:172.21.15.120:6789/0 conn(0x7f62c8060250 legacy=0x7f62c80606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cddb9700 1 -- 172.21.15.26:0/1389955437 >> v1:172.21.15.26:6789/0 conn(0x7f62c80621f0 legacy=0x7f62c8067020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cddb9700 1 -- 172.21.15.26:0/1389955437 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c813e7f0 con 0x7f62c80aff40 2022-07-20T02:29:02.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62ccdb7700 1 -- 172.21.15.26:0/1389955437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283336407 0 0) 0x7f62c80b0f50 con 0x7f62c80621f0 2022-07-20T02:29:02.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cddb9700 1 --2- 172.21.15.26:0/1389955437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80aff40 0x7f62c8066b30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f62c4002f70 tx=0x7f62c4006eb0).ready entity=mon.0 client_cookie=a766f64313ca3145 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:02.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62ccdb7700 1 -- 172.21.15.26:0/1389955437 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62c400d660 con 0x7f62c80aff40 2022-07-20T02:29:02.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62ccdb7700 1 -- 172.21.15.26:0/1389955437 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c4000e90 con 0x7f62c80aff40 2022-07-20T02:29:02.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62ccdb7700 1 -- 172.21.15.26:0/1389955437 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62c400d9b0 con 0x7f62c80aff40 2022-07-20T02:29:02.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- 172.21.15.26:0/1389955437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80aff40 msgr2=0x7f62c8066b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1389955437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80aff40 0x7f62c8066b30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f62c4002f70 tx=0x7f62c4006eb0).stop 2022-07-20T02:29:02.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- 172.21.15.26:0/1389955437 shutdown_connections 2022-07-20T02:29:02.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1389955437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80aff40 0x7f62c8066b30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- 172.21.15.26:0/1389955437 >> 172.21.15.26:0/1389955437 conn(0x7f62c80b3050 msgr2=0x7f62c80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:02.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- 172.21.15.26:0/1389955437 shutdown_connections 2022-07-20T02:29:02.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- 172.21.15.26:0/1389955437 wait complete. 2022-07-20T02:29:02.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 Processor -- start 2022-07-20T02:29:02.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.356+0000 7f62cedbb700 1 -- start start 2022-07-20T02:29:02.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8060250 0x7f62c8064370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62c80aff40 0x7f62c8153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8144cb0 con 0x7f62c80621f0 2022-07-20T02:29:02.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8150a70 con 0x7f62c8060250 2022-07-20T02:29:02.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8150d10 con 0x7f62c80aff40 2022-07-20T02:29:02.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37642/0 (socket says 172.21.15.26:37642) 2022-07-20T02:29:02.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 -- 172.21.15.26:0/1024442854 learned_addr learned my addr 172.21.15.26:0/1024442854 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:02.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cddb9700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8060250 0x7f62c8064370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62ce5ba700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62c80aff40 0x7f62c8153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 -- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62c80aff40 msgr2=0x7f62c8153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62c80aff40 0x7f62c8153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 -- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8060250 msgr2=0x7f62c8064370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:02.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8060250 0x7f62c8064370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:02.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 -- 172.21.15.26:0/1024442854 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c400b1e0 con 0x7f62c80621f0 2022-07-20T02:29:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cd5b8700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f62bc000bc0 tx=0x7f62bc007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:02.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62bc0070d0 con 0x7f62c80621f0 2022-07-20T02:29:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc007230 con 0x7f62c80621f0 2022-07-20T02:29:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c8063c30 con 0x7f62c80621f0 2022-07-20T02:29:02.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62bc005740 con 0x7f62c80621f0 2022-07-20T02:29:02.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.360+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c80ace40 con 0x7f62c80621f0 2022-07-20T02:29:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c805fb10 con 0x7f62c80621f0 2022-07-20T02:29:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f62bc01a060 con 0x7f62c80621f0 2022-07-20T02:29:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62baffd700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 0x7f62a8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f62bc06b480 con 0x7f62c80621f0 2022-07-20T02:29:02.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62cddb9700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 0x7f62a8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.364+0000 7f62cddb9700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 0x7f62a8055ff0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f62c401c000 tx=0x7f62c4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.372+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62bc042520 con 0x7f62c80621f0 2022-07-20T02:29:02.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:02.512+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f62c80574e0 con 0x7f62c80621f0 2022-07-20T02:29:03.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.436+0000 7f62baffd700 1 -- 172.21.15.26:0/1024442854 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v50) v1 ==== 97+0+0 (secure 0 0 0) 0x7f62bc018070 con 0x7f62c80621f0 2022-07-20T02:29:03.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 msgr2=0x7f62a8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 0x7f62a8055ff0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f62c401c000 tx=0x7f62c4015000).stop 2022-07-20T02:29:03.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 msgr2=0x7f62c8147c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f62bc000bc0 tx=0x7f62bc007b00).stop 2022-07-20T02:29:03.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 shutdown_connections 2022-07-20T02:29:03.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62c80621f0 0x7f62c8147c20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62a8053b30 0x7f62a8055ff0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62c80aff40 0x7f62c8153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 --2- 172.21.15.26:0/1024442854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8060250 0x7f62c8064370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 >> 172.21.15.26:0/1024442854 conn(0x7f62c80b3050 msgr2=0x7f62c8140ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 shutdown_connections 2022-07-20T02:29:03.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.444+0000 7f62cedbb700 1 -- 172.21.15.26:0/1024442854 wait complete. 2022-07-20T02:29:03.448 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:29:03.461 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:29:03.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:29:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.560+0000 7f8da7fff700 1 Processor -- start 2022-07-20T02:29:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.560+0000 7f8da7fff700 1 -- start start 2022-07-20T02:29:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.560+0000 7f8da7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8da80b3260 0x7f8da8067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 0x7f8da8142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da805ff70 con 0x7f8da8061860 2022-07-20T02:29:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da80b3640 con 0x7f8da80b3260 2022-07-20T02:29:03.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da80601a0 con 0x7f8da80980d0 2022-07-20T02:29:03.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8da80b3260 0x7f8da8067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da77fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 0x7f8da8142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da77fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 0x7f8da8142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:57346/0 (socket says 172.21.15.26:57346) 2022-07-20T02:29:03.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da67fc700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8da8061860 0x7f8da80979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51142/0 (socket says 172.21.15.26:51142) 2022-07-20T02:29:03.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da77fe700 1 -- 172.21.15.26:0/1187335175 learned_addr learned my addr 172.21.15.26:0/1187335175 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:03.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 -- 172.21.15.26:0/1187335175 >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 msgr2=0x7f8da8142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:03.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 --2- 172.21.15.26:0/1187335175 >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 0x7f8da8142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 -- 172.21.15.26:0/1187335175 >> v1:172.21.15.120:6789/0 conn(0x7f8da8061860 legacy=0x7f8da80979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 -- 172.21.15.26:0/1187335175 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da80606e0 con 0x7f8da80b3260 2022-07-20T02:29:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da6ffd700 1 --2- 172.21.15.26:0/1187335175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da8067900 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f8d98002f70 tx=0x7f8d98006df0).ready entity=mon.0 client_cookie=fda882d60dad6bd5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da5ffb700 1 -- 172.21.15.26:0/1187335175 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d9800d570 con 0x7f8da80b3260 2022-07-20T02:29:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da5ffb700 1 -- 172.21.15.26:0/1187335175 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d98000e90 con 0x7f8da80b3260 2022-07-20T02:29:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da5ffb700 1 -- 172.21.15.26:0/1187335175 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d9800d8c0 con 0x7f8da80b3260 2022-07-20T02:29:03.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- 172.21.15.26:0/1187335175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 msgr2=0x7f8da8067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 --2- 172.21.15.26:0/1187335175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da8067900 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f8d98002f70 tx=0x7f8d98006df0).stop 2022-07-20T02:29:03.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- 172.21.15.26:0/1187335175 shutdown_connections 2022-07-20T02:29:03.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 --2- 172.21.15.26:0/1187335175 >> v2:172.21.15.120:3300/0 conn(0x7f8da80980d0 0x7f8da8142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 --2- 172.21.15.26:0/1187335175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da8067900 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- 172.21.15.26:0/1187335175 >> 172.21.15.26:0/1187335175 conn(0x7f8da80b67f0 msgr2=0x7f8da80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- 172.21.15.26:0/1187335175 shutdown_connections 2022-07-20T02:29:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.564+0000 7f8da7fff700 1 -- 172.21.15.26:0/1187335175 wait complete. 2022-07-20T02:29:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 Processor -- start 2022-07-20T02:29:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 -- start start 2022-07-20T02:29:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8da8061860 0x7f8da8066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8da80980d0 0x7f8da8150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8066750 con 0x7f8da80b3260 2022-07-20T02:29:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8151690 con 0x7f8da8061860 2022-07-20T02:29:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.568+0000 7f8da7fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8066980 con 0x7f8da80980d0 2022-07-20T02:29:03.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da6ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8da8061860 0x7f8da8066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37660/0 (socket says 172.21.15.26:37660) 2022-07-20T02:29:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da67fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8da80980d0 0x7f8da8150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 -- 172.21.15.26:0/365359793 learned_addr learned my addr 172.21.15.26:0/365359793 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 -- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8da80980d0 msgr2=0x7f8da8150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8da80980d0 0x7f8da8150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 -- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8da8061860 msgr2=0x7f8da8066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8da8061860 0x7f8da8066260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 -- 172.21.15.26:0/365359793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9800b180 con 0x7f8da80b3260 2022-07-20T02:29:03.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da77fe700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f8da000d000 tx=0x7f8da000a430).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da0010630 con 0x7f8da80b3260 2022-07-20T02:29:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da0010790 con 0x7f8da80b3260 2022-07-20T02:29:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da80afea0 con 0x7f8da80b3260 2022-07-20T02:29:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da0010af0 con 0x7f8da80b3260 2022-07-20T02:29:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da8150810 con 0x7f8da80b3260 2022-07-20T02:29:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da81509d0 con 0x7f8da80b3260 2022-07-20T02:29:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.572+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8da0021070 con 0x7f8da80b3260 2022-07-20T02:29:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.576+0000 7f8d97fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 0x7f8d84055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.576+0000 7f8da6ffd700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 0x7f8d84055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.576+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f8da006c0e0 con 0x7f8da80b3260 2022-07-20T02:29:03.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.576+0000 7f8da6ffd700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 0x7f8d84055ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8d9801a000 tx=0x7f8d98014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.580+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8da0043140 con 0x7f8da80b3260 2022-07-20T02:29:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8da8067d10 con 0x7f8da80b3260 2022-07-20T02:29:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8d97fff700 1 -- 172.21.15.26:0/365359793 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f8da0015090 con 0x7f8da80b3260 2022-07-20T02:29:03.749 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:03.750 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:29:03.750 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:29:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.751 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:29:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:29:03.752 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.753 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:29:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.754 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:29:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:29:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:29:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:29:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:29:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:29:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:29:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:29:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:29:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:29:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.784 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:29:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:29:03.785 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.787 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.789 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:29:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:29:03.790 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.791 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:29:03.794 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:29:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.795 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.796 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.798 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:29:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:29:03.799 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:29:03.803 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:29:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.804 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.805 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.807 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:29:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:29:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:29:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:29:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:29:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:29:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:29:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:29:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:29:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:29:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:29:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:29:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:29:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:29:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:29:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:29:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:29:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:29:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:29:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:29:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:29:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:29:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:29:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:29:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:29:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:29:03.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:29:03.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:03.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:29:03.839 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:29:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 msgr2=0x7f8d84055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 0x7f8d84055ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8d9801a000 tx=0x7f8d98014000).stop 2022-07-20T02:29:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 msgr2=0x7f8da81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f8da000d000 tx=0x7f8da000a430).stop 2022-07-20T02:29:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 shutdown_connections 2022-07-20T02:29:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8da80b3260 0x7f8da81511a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d84053b30 0x7f8d84055ff0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8da80980d0 0x7f8da8150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 --2- 172.21.15.26:0/365359793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8da8061860 0x7f8da8066260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 >> 172.21.15.26:0/365359793 conn(0x7f8da80b67f0 msgr2=0x7f8da8143410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 shutdown_connections 2022-07-20T02:29:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.748+0000 7f8da7fff700 1 -- 172.21.15.26:0/365359793 wait complete. 2022-07-20T02:29:03.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:03.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.948+0000 7f09d9601700 1 Processor -- start 2022-07-20T02:29:03.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.948+0000 7f09d9601700 1 -- start start 2022-07-20T02:29:03.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09d40b2920 0x7f09d4147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b1bc0 con 0x7f09d4067190 2022-07-20T02:29:03.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4147700 con 0x7f09d4061050 2022-07-20T02:29:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d414b040 con 0x7f09d40b2920 2022-07-20T02:29:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f09d4067190 0x7f09d40ab680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36948/0 (socket says 172.21.15.26:36948) 2022-07-20T02:29:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d37fe700 1 -- 172.21.15.26:0/4029352757 learned_addr learned my addr 172.21.15.26:0/4029352757 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d2ffd700 1 --2- 172.21.15.26:0/4029352757 >> v2:172.21.15.120:3300/0 conn(0x7f09d40b2920 0x7f09d4147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202997776 0 0) 0x7f09d40b1bc0 con 0x7f09d4067190 2022-07-20T02:29:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4003260 con 0x7f09d4067190 2022-07-20T02:29:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521932689 0 0) 0x7f09b4003260 con 0x7f09d4067190 2022-07-20T02:29:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 >> v2:172.21.15.120:3300/0 conn(0x7f09d40b2920 msgr2=0x7f09d4147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 --2- 172.21.15.26:0/4029352757 >> v2:172.21.15.120:3300/0 conn(0x7f09d40b2920 0x7f09d4147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 >> v1:172.21.15.120:6789/0 conn(0x7f09d4061050 legacy=0x7f09d4066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d4147930 con 0x7f09d4067190 2022-07-20T02:29:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f09cc0019e0 con 0x7f09d4067190 2022-07-20T02:29:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 >> v1:172.21.15.26:6789/0 conn(0x7f09d4067190 legacy=0x7f09d40ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4003750 0x7f09b4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 0x7f09b4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b4004e70 0x7f09b400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d414b040 con 0x7f09b4004500 2022-07-20T02:29:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b400bd20 con 0x7f09b4003750 2022-07-20T02:29:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b400bee0 con 0x7f09b4004e70 2022-07-20T02:29:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 0x7f09b4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d37fe700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b4004e70 0x7f09b400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d2ffd700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4003750 0x7f09b4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 -- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b4004e70 msgr2=0x7f09b400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b4004e70 0x7f09b400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 -- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4003750 msgr2=0x7f09b4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4003750 0x7f09b4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 -- 172.21.15.26:0/4029352757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4003260 con 0x7f09b4004500 2022-07-20T02:29:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d37fe700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09b4004e70 0x7f09b400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d37fe700 1 -- 172.21.15.26:0/4029352757 reap_dead start 2022-07-20T02:29:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09cbfff700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 0x7f09b4004980 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f09d40b30f0 tx=0x7f09c4007a00).ready entity=mon.0 client_cookie=1abe91fbbfca83ee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09c4007d10 con 0x7f09b4004500 2022-07-20T02:29:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c4007030 con 0x7f09b4004500 2022-07-20T02:29:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d27fc700 1 -- 172.21.15.26:0/4029352757 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09c4007390 con 0x7f09b4004500 2022-07-20T02:29:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 msgr2=0x7f09b4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 0x7f09b4004980 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f09d40b30f0 tx=0x7f09c4007a00).stop 2022-07-20T02:29:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- 172.21.15.26:0/4029352757 shutdown_connections 2022-07-20T02:29:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 --2- 172.21.15.26:0/4029352757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09b4004500 0x7f09b4004980 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- 172.21.15.26:0/4029352757 >> 172.21.15.26:0/4029352757 conn(0x7f09d40b60a0 msgr2=0x7f09d40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- 172.21.15.26:0/4029352757 shutdown_connections 2022-07-20T02:29:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.952+0000 7f09d9601700 1 -- 172.21.15.26:0/4029352757 wait complete. 2022-07-20T02:29:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 Processor -- start 2022-07-20T02:29:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 -- start start 2022-07-20T02:29:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4004500 0x7f09d414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d4062bb0 0x7f09d4142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40655c0 con 0x7f09d4066cd0 2022-07-20T02:29:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4064ae0 con 0x7f09b4004500 2022-07-20T02:29:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d9601700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4149760 con 0x7f09d4062bb0 2022-07-20T02:29:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d2ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4004500 0x7f09d414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09d37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d4062bb0 0x7f09d4142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37678/0 (socket says 172.21.15.26:37678) 2022-07-20T02:29:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 -- 172.21.15.26:0/312647537 learned_addr learned my addr 172.21.15.26:0/312647537 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 -- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d4062bb0 msgr2=0x7f09d4142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d4062bb0 0x7f09d4142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 -- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4004500 msgr2=0x7f09d414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4004500 0x7f09d414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.956+0000 7f09cbfff700 1 -- 172.21.15.26:0/312647537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40657b0 con 0x7f09d4066cd0 2022-07-20T02:29:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09cbfff700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f09d40b30f0 tx=0x7f09c4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09c4007d10 con 0x7f09d4066cd0 2022-07-20T02:29:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c40127b0 con 0x7f09d4066cd0 2022-07-20T02:29:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d4065a50 con 0x7f09d4066cd0 2022-07-20T02:29:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09c4012b10 con 0x7f09d4066cd0 2022-07-20T02:29:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d4065f90 con 0x7f09d4066cd0 2022-07-20T02:29:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d414ada0 con 0x7f09d4066cd0 2022-07-20T02:29:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f09c401d070 con 0x7f09d4066cd0 2022-07-20T02:29:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 0x7f09b8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f09c4001e40 con 0x7f09d4066cd0 2022-07-20T02:29:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.960+0000 7f09d2ffd700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 0x7f09b8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.964+0000 7f09d2ffd700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 0x7f09b8055f30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f09bc00b000 tx=0x7f09bc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:03.968+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f09c4043980 con 0x7f09d4066cd0 2022-07-20T02:29:04.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.092+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09d40574e0 con 0x7f09d4066cd0 2022-07-20T02:29:04.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.092+0000 7f09d0ff9700 1 -- 172.21.15.26:0/312647537 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13462 (secure 0 0 0) 0x7f09c4025070 con 0x7f09d4066cd0 2022-07-20T02:29:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:04.126 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":50,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:03.431457+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 msgr2=0x7f09b8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 0x7f09b8055f30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f09bc00b000 tx=0x7f09bc008040).stop 2022-07-20T02:29:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 msgr2=0x7f09d4142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f09d40b30f0 tx=0x7f09c4015000).stop 2022-07-20T02:29:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 shutdown_connections 2022-07-20T02:29:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f09d4066cd0 0x7f09d4142ae0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f09b8053a70 0x7f09b8055f30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d4062bb0 0x7f09d4142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 --2- 172.21.15.26:0/312647537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09b4004500 0x7f09d414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 >> 172.21.15.26:0/312647537 conn(0x7f09d40b60a0 msgr2=0x7f09d4149e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 shutdown_connections 2022-07-20T02:29:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.096+0000 7f09d9601700 1 -- 172.21.15.26:0/312647537 wait complete. 2022-07-20T02:29:04.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:29:04.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:29:04.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.256+0000 7f51affff700 1 Processor -- start 2022-07-20T02:29:04.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.256+0000 7f51affff700 1 -- start start 2022-07-20T02:29:04.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51b00b5ef0 0x7f51b01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51b00a8c40 0x7f51b00af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00b62d0 con 0x7f51b00a7fc0 2022-07-20T02:29:04.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b014aac0 con 0x7f51b00a8c40 2022-07-20T02:29:04.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b014ad60 con 0x7f51b00b5ef0 2022-07-20T02:29:04.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51b00a8c40 0x7f51b00af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51ae7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f51b00a7fc0 0x7f51b00aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36964/0 (socket says 172.21.15.26:36964) 2022-07-20T02:29:04.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51ae7fc700 1 -- 172.21.15.26:0/416827772 learned_addr learned my addr 172.21.15.26:0/416827772 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:04.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51aeffd700 1 --2- 172.21.15.26:0/416827772 >> v2:172.21.15.120:3300/0 conn(0x7f51b00b5ef0 0x7f51b01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 -- 172.21.15.26:0/416827772 >> v2:172.21.15.120:3300/0 conn(0x7f51b00b5ef0 msgr2=0x7f51b01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:04.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 --2- 172.21.15.26:0/416827772 >> v2:172.21.15.120:3300/0 conn(0x7f51b00b5ef0 0x7f51b01474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 -- 172.21.15.26:0/416827772 >> v1:172.21.15.26:6789/0 conn(0x7f51b00a7fc0 legacy=0x7f51b00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 -- 172.21.15.26:0/416827772 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b014b000 con 0x7f51b00a8c40 2022-07-20T02:29:04.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51adffb700 1 -- 172.21.15.26:0/416827772 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995101707 0 0) 0x7f51b00b62d0 con 0x7f51b00a7fc0 2022-07-20T02:29:04.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51af7fe700 1 --2- 172.21.15.26:0/416827772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b00af550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f51a8002f70 tx=0x7f51a8006df0).ready entity=mon.0 client_cookie=c174338de1fba653 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51adffb700 1 -- 172.21.15.26:0/416827772 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51a800d5c0 con 0x7f51b00a8c40 2022-07-20T02:29:04.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51adffb700 1 -- 172.21.15.26:0/416827772 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a8000e90 con 0x7f51b00a8c40 2022-07-20T02:29:04.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51adffb700 1 -- 172.21.15.26:0/416827772 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51a800d910 con 0x7f51b00a8c40 2022-07-20T02:29:04.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- 172.21.15.26:0/416827772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 msgr2=0x7f51b00af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 --2- 172.21.15.26:0/416827772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b00af550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f51a8002f70 tx=0x7f51a8006df0).stop 2022-07-20T02:29:04.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- 172.21.15.26:0/416827772 shutdown_connections 2022-07-20T02:29:04.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 --2- 172.21.15.26:0/416827772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b00af550 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 --2- 172.21.15.26:0/416827772 >> v2:172.21.15.120:3300/0 conn(0x7f51b00b5ef0 0x7f51b01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- 172.21.15.26:0/416827772 >> 172.21.15.26:0/416827772 conn(0x7f51b0060310 msgr2=0x7f51b00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- 172.21.15.26:0/416827772 shutdown_connections 2022-07-20T02:29:04.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.260+0000 7f51affff700 1 -- 172.21.15.26:0/416827772 wait complete. 2022-07-20T02:29:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 Processor -- start 2022-07-20T02:29:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 -- start start 2022-07-20T02:29:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 0x7f51b014cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b00b5ef0 0x7f51b00a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00643d0 con 0x7f51b00a8c40 2022-07-20T02:29:04.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b0159bc0 con 0x7f51b00a7fc0 2022-07-20T02:29:04.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51affff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b0159df0 con 0x7f51b00b5ef0 2022-07-20T02:29:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51ae7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51ae7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37692/0 (socket says 172.21.15.26:37692) 2022-07-20T02:29:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51ae7fc700 1 -- 172.21.15.26:0/443009838 learned_addr learned my addr 172.21.15.26:0/443009838 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.264+0000 7f51aeffd700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 0x7f51b014cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51af7fe700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b00b5ef0 0x7f51b00a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 -- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b00b5ef0 msgr2=0x7f51b00a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b00b5ef0 0x7f51b00a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 -- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 msgr2=0x7f51b014cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 0x7f51b014cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 -- 172.21.15.26:0/443009838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a800b180 con 0x7f51b00a8c40 2022-07-20T02:29:04.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51aeffd700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 0x7f51b014cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:04.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51ae7fc700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f51b0063c80 tx=0x7f51a0007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51a0005b60 con 0x7f51b00a8c40 2022-07-20T02:29:04.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a0005cc0 con 0x7f51b00a8c40 2022-07-20T02:29:04.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b015a090 con 0x7f51b00a8c40 2022-07-20T02:29:04.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51a0007300 con 0x7f51b00a8c40 2022-07-20T02:29:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b00abe70 con 0x7f51b00a8c40 2022-07-20T02:29:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b015a220 con 0x7f51b00a8c40 2022-07-20T02:29:04.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51a0010030 con 0x7f51b00a8c40 2022-07-20T02:29:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 0x7f518c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f51a006bdc0 con 0x7f51b00a8c40 2022-07-20T02:29:04.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.268+0000 7f51aeffd700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 0x7f518c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.272+0000 7f51aeffd700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 0x7f518c055f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f51a400c000 tx=0x7f51a4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.276+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f51a0042ec0 con 0x7f51b00a8c40 2022-07-20T02:29:04.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f51b014b2a0 con 0x7f51b00a8c40 2022-07-20T02:29:04.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f519ffff700 1 -- 172.21.15.26:0/443009838 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v50) v1 ==== 110+0+10 (secure 0 0 0) 0x7f51a00464e0 con 0x7f51b00a8c40 2022-07-20T02:29:04.412 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:29:04.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 msgr2=0x7f518c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 0x7f518c055f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f51a400c000 tx=0x7f51a4008040).stop 2022-07-20T02:29:04.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 msgr2=0x7f51b015a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f51b0063c80 tx=0x7f51a0007e90).stop 2022-07-20T02:29:04.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 shutdown_connections 2022-07-20T02:29:04.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51b00a8c40 0x7f51b015a530 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f518c053a70 0x7f518c055f30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b00b5ef0 0x7f51b00a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 --2- 172.21.15.26:0/443009838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b00a7fc0 0x7f51b014cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.404+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 >> 172.21.15.26:0/443009838 conn(0x7f51b0060310 msgr2=0x7f51b00aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.408+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 shutdown_connections 2022-07-20T02:29:04.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.408+0000 7f51affff700 1 -- 172.21.15.26:0/443009838 wait complete. 2022-07-20T02:29:04.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:29:04.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.536+0000 7f400f59e700 1 Processor -- start 2022-07-20T02:29:04.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.536+0000 7f400f59e700 1 -- start start 2022-07-20T02:29:04.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f401005cfd0 0x7f40100aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4010142fa0 0x7f4010143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401005d3b0 con 0x7f40100b1cb0 2022-07-20T02:29:04.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40100b1a60 con 0x7f4010142fa0 2022-07-20T02:29:04.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40100b2140 con 0x7f401005cfd0 2022-07-20T02:29:04.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4010142fa0 0x7f4010143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f4007fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f40100b1cb0 0x7f4010142890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36978/0 (socket says 172.21.15.26:36978) 2022-07-20T02:29:04.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f4007fff700 1 -- 172.21.15.26:0/2042308751 learned_addr learned my addr 172.21.15.26:0/2042308751 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:04.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400e59c700 1 --2- 172.21.15.26:0/2042308751 >> v2:172.21.15.120:3300/0 conn(0x7f401005cfd0 0x7f40100aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400dd9b700 1 -- 172.21.15.26:0/2042308751 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575434809 0 0) 0x7f401005d3b0 con 0x7f40100b1cb0 2022-07-20T02:29:04.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400dd9b700 1 -- 172.21.15.26:0/2042308751 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff0003260 con 0x7f40100b1cb0 2022-07-20T02:29:04.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 -- 172.21.15.26:0/2042308751 >> v2:172.21.15.120:3300/0 conn(0x7f401005cfd0 msgr2=0x7f40100aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:04.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 --2- 172.21.15.26:0/2042308751 >> v2:172.21.15.120:3300/0 conn(0x7f401005cfd0 0x7f40100aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 -- 172.21.15.26:0/2042308751 >> v1:172.21.15.26:6789/0 conn(0x7f40100b1cb0 legacy=0x7f4010142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 -- 172.21.15.26:0/2042308751 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100b1520 con 0x7f4010142fa0 2022-07-20T02:29:04.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400ed9d700 1 --2- 172.21.15.26:0/2042308751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4010142fa0 0x7f4010143810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f40080032f0 tx=0x7f40080091d0).ready entity=mon.0 client_cookie=bd59c682ac8b3a49 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400dd9b700 1 -- 172.21.15.26:0/2042308751 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400800f8a0 con 0x7f4010142fa0 2022-07-20T02:29:04.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400dd9b700 1 -- 172.21.15.26:0/2042308751 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400800fa00 con 0x7f4010142fa0 2022-07-20T02:29:04.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400dd9b700 1 -- 172.21.15.26:0/2042308751 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400800fd60 con 0x7f4010142fa0 2022-07-20T02:29:04.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- 172.21.15.26:0/2042308751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4010142fa0 msgr2=0x7f4010143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 --2- 172.21.15.26:0/2042308751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4010142fa0 0x7f4010143810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f40080032f0 tx=0x7f40080091d0).stop 2022-07-20T02:29:04.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- 172.21.15.26:0/2042308751 shutdown_connections 2022-07-20T02:29:04.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 --2- 172.21.15.26:0/2042308751 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4010142fa0 0x7f4010143810 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 --2- 172.21.15.26:0/2042308751 >> v2:172.21.15.120:3300/0 conn(0x7f401005cfd0 0x7f40100aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- 172.21.15.26:0/2042308751 >> 172.21.15.26:0/2042308751 conn(0x7f40100b7090 msgr2=0x7f401005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- 172.21.15.26:0/2042308751 shutdown_connections 2022-07-20T02:29:04.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.540+0000 7f400f59e700 1 -- 172.21.15.26:0/2042308751 wait complete. 2022-07-20T02:29:04.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 Processor -- start 2022-07-20T02:29:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 -- start start 2022-07-20T02:29:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f401005cfd0 0x7f4010065fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4010142fa0 0x7f4010152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100afea0 con 0x7f40100b1cb0 2022-07-20T02:29:04.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100b0620 con 0x7f4010142fa0 2022-07-20T02:29:04.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400f59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100afc00 con 0x7f401005cfd0 2022-07-20T02:29:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f4007fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f4007fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37706/0 (socket says 172.21.15.26:37706) 2022-07-20T02:29:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f4007fff700 1 -- 172.21.15.26:0/1142146981 learned_addr learned my addr 172.21.15.26:0/1142146981 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400ed9d700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4010142fa0 0x7f4010152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.544+0000 7f400e59c700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f401005cfd0 0x7f4010065fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 -- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f401005cfd0 msgr2=0x7f4010065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f401005cfd0 0x7f4010065fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 -- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4010142fa0 msgr2=0x7f4010152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4010142fa0 0x7f4010152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 -- 172.21.15.26:0/1142146981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008009470 con 0x7f40100b1cb0 2022-07-20T02:29:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f4007fff700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f3ff8001110 tx=0x7f3ff8007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff800ce20 con 0x7f40100b1cb0 2022-07-20T02:29:04.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff80072e0 con 0x7f40100b1cb0 2022-07-20T02:29:04.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40101502c0 con 0x7f40100b1cb0 2022-07-20T02:29:04.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff8005800 con 0x7f40100b1cb0 2022-07-20T02:29:04.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40100af3b0 con 0x7f40100b1cb0 2022-07-20T02:29:04.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40100af170 con 0x7f40100b1cb0 2022-07-20T02:29:04.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.548+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ff801a070 con 0x7f40100b1cb0 2022-07-20T02:29:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.552+0000 7f40077fe700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 0x7f3ff4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.552+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f3ff806b180 con 0x7f40100b1cb0 2022-07-20T02:29:04.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.552+0000 7f400e59c700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 0x7f3ff4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.552+0000 7f400e59c700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 0x7f3ff4055ea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f400000b000 tx=0x7f4000006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.556+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ff8068e90 con 0x7f40100b1cb0 2022-07-20T02:29:04.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.680+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f40100b17c0 con 0x7f40100b1cb0 2022-07-20T02:29:04.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f40077fe700 1 -- 172.21.15.26:0/1142146981 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v50) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3ff8018020 con 0x7f40100b1cb0 2022-07-20T02:29:04.686 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:04.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 msgr2=0x7f3ff4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 0x7f3ff4055ea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f400000b000 tx=0x7f4000006df0).stop 2022-07-20T02:29:04.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 msgr2=0x7f4010152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f3ff8001110 tx=0x7f3ff8007b20).stop 2022-07-20T02:29:04.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 shutdown_connections 2022-07-20T02:29:04.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40100b1cb0 0x7f4010152680 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ff40539e0 0x7f3ff4055ea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4010142fa0 0x7f4010152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 --2- 172.21.15.26:0/1142146981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f401005cfd0 0x7f4010065fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 >> 172.21.15.26:0/1142146981 conn(0x7f40100b7090 msgr2=0x7f40100ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 shutdown_connections 2022-07-20T02:29:04.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.684+0000 7f400f59e700 1 -- 172.21.15.26:0/1142146981 wait complete. 2022-07-20T02:29:04.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:29:04.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 Processor -- start 2022-07-20T02:29:04.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 -- start start 2022-07-20T02:29:04.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe7101465b0 0x7fe710066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7100672a0 0x7fe710146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7100b1ae0 con 0x7fe7100b24c0 2022-07-20T02:29:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100b2e20 con 0x7fe7101465b0 2022-07-20T02:29:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe716a0b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100b1d80 con 0x7fe7100672a0 2022-07-20T02:29:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe715208700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe7100b24c0 0x7fe7100604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36992/0 (socket says 172.21.15.26:36992) 2022-07-20T02:29:04.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe715208700 1 -- 172.21.15.26:0/3028874419 learned_addr learned my addr 172.21.15.26:0/3028874419 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:04.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe715a09700 1 --2- 172.21.15.26:0/3028874419 >> v2:172.21.15.26:3300/0 conn(0x7fe7101465b0 0x7fe710066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.800+0000 7fe71620a700 1 --2- 172.21.15.26:0/3028874419 >> v2:172.21.15.120:3300/0 conn(0x7fe7100672a0 0x7fe710146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe714a07700 1 -- 172.21.15.26:0/3028874419 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99359071 0 0) 0x7fe7100b1ae0 con 0x7fe7100b24c0 2022-07-20T02:29:04.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe714a07700 1 -- 172.21.15.26:0/3028874419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f8003260 con 0x7fe7100b24c0 2022-07-20T02:29:04.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe715a09700 1 -- 172.21.15.26:0/3028874419 >> v2:172.21.15.120:3300/0 conn(0x7fe7100672a0 msgr2=0x7fe710146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:04.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe715a09700 1 --2- 172.21.15.26:0/3028874419 >> v2:172.21.15.120:3300/0 conn(0x7fe7100672a0 0x7fe710146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe715a09700 1 -- 172.21.15.26:0/3028874419 >> v1:172.21.15.26:6789/0 conn(0x7fe7100b24c0 legacy=0x7fe7100604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe715a09700 1 -- 172.21.15.26:0/3028874419 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100b2020 con 0x7fe7101465b0 2022-07-20T02:29:04.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe715a09700 1 --2- 172.21.15.26:0/3028874419 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7101465b0 0x7fe710066db0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c006df0).ready entity=mon.0 client_cookie=f243bfcdb2bf648a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe714a07700 1 -- 172.21.15.26:0/3028874419 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c00d4f0 con 0x7fe7101465b0 2022-07-20T02:29:04.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe714a07700 1 -- 172.21.15.26:0/3028874419 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c000e90 con 0x7fe7101465b0 2022-07-20T02:29:04.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe714a07700 1 -- 172.21.15.26:0/3028874419 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c00d840 con 0x7fe7101465b0 2022-07-20T02:29:04.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- 172.21.15.26:0/3028874419 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7101465b0 msgr2=0x7fe710066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 --2- 172.21.15.26:0/3028874419 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7101465b0 0x7fe710066db0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c006df0).stop 2022-07-20T02:29:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- 172.21.15.26:0/3028874419 shutdown_connections 2022-07-20T02:29:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 --2- 172.21.15.26:0/3028874419 >> v2:172.21.15.120:3300/0 conn(0x7fe7100672a0 0x7fe710146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 --2- 172.21.15.26:0/3028874419 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7101465b0 0x7fe710066db0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- 172.21.15.26:0/3028874419 >> 172.21.15.26:0/3028874419 conn(0x7fe7100b3050 msgr2=0x7fe7100b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- 172.21.15.26:0/3028874419 shutdown_connections 2022-07-20T02:29:04.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- 172.21.15.26:0/3028874419 wait complete. 2022-07-20T02:29:04.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 Processor -- start 2022-07-20T02:29:04.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.804+0000 7fe716a0b700 1 -- start start 2022-07-20T02:29:04.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7101465b0 0x7fe710155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 0x7fe710064c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7100b5630 con 0x7fe7100672a0 2022-07-20T02:29:04.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe710155da0 con 0x7fe7100b51a0 2022-07-20T02:29:04.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe710156040 con 0x7fe7101465b0 2022-07-20T02:29:04.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37718/0 (socket says 172.21.15.26:37718) 2022-07-20T02:29:04.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 -- 172.21.15.26:0/1981503002 learned_addr learned my addr 172.21.15.26:0/1981503002 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:04.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe71620a700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 0x7fe710064c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715208700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7101465b0 0x7fe710155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 -- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7101465b0 msgr2=0x7fe710155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7101465b0 0x7fe710155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 -- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 msgr2=0x7fe710064c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 0x7fe710064c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 -- 172.21.15.26:0/1981503002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c00b180 con 0x7fe7100672a0 2022-07-20T02:29:04.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe71620a700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 0x7fe710064c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:04.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe715a09700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c013de0 con 0x7fe7100672a0 2022-07-20T02:29:04.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c00de60 con 0x7fe7100672a0 2022-07-20T02:29:04.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7101562e0 con 0x7fe7100672a0 2022-07-20T02:29:04.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c01e5f0 con 0x7fe7100672a0 2022-07-20T02:29:04.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7101418a0 con 0x7fe7100672a0 2022-07-20T02:29:04.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.808+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7101521f0 con 0x7fe7100672a0 2022-07-20T02:29:04.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.812+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe70c025030 con 0x7fe7100672a0 2022-07-20T02:29:04.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.812+0000 7fe6feffd700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 0x7fe6f0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:04.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.812+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe70c06c890 con 0x7fe7100672a0 2022-07-20T02:29:04.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.812+0000 7fe715208700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 0x7fe6f0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:04.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.812+0000 7fe715208700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 0x7fe6f0055ea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fe70000d000 tx=0x7fe700009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:04.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.816+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe7101521f0 con 0x7fe7100672a0 2022-07-20T02:29:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.940+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe7101521f0 con 0x7fe7100672a0 2022-07-20T02:29:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.940+0000 7fe6feffd700 1 -- 172.21.15.26:0/1981503002 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+143 (secure 0 0 0) 0x7fe7101521f0 con 0x7fe7100672a0 2022-07-20T02:29:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:04.946 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:29:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:29:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:29:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:29:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:29:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:29:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:04.949 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:29:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 msgr2=0x7fe6f0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 0x7fe6f0055ea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fe70000d000 tx=0x7fe700009040).stop 2022-07-20T02:29:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 msgr2=0x7fe7100acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c014000).stop 2022-07-20T02:29:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 shutdown_connections 2022-07-20T02:29:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f00539e0 0x7fe6f0055ea0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7100b51a0 0x7fe710064c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7101465b0 0x7fe710155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 --2- 172.21.15.26:0/1981503002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100672a0 0x7fe7100acd50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 >> 172.21.15.26:0/1981503002 conn(0x7fe7100b3050 msgr2=0x7fe7100b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 shutdown_connections 2022-07-20T02:29:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:04.944+0000 7fe716a0b700 1 -- 172.21.15.26:0/1981503002 wait complete. 2022-07-20T02:29:04.957 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:29:04.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:29:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.064+0000 7fcbbffff700 1 Processor -- start 2022-07-20T02:29:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.064+0000 7fcbbffff700 1 -- start start 2022-07-20T02:29:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcbc00b24c0 0x7fcbc00604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbc005f840 0x7fcbc00609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00b1ae0 con 0x7fcbc01465b0 2022-07-20T02:29:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc00b2e20 con 0x7fcbc005f840 2022-07-20T02:29:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc00b1d80 con 0x7fcbc00b24c0 2022-07-20T02:29:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbc005f840 0x7fcbc00609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbc005f840 0x7fcbc00609d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37730/0 (socket says 172.21.15.26:37730) 2022-07-20T02:29:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbeffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcbc01465b0 0x7fcbc0066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37006/0 (socket says 172.21.15.26:37006) 2022-07-20T02:29:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/4121656078 learned_addr learned my addr 172.21.15.26:0/4121656078 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbe7fc700 1 --2- 172.21.15.26:0/4121656078 >> v2:172.21.15.120:3300/0 conn(0x7fcbc00b24c0 0x7fcbc00604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/4121656078 >> v2:172.21.15.120:3300/0 conn(0x7fcbc00b24c0 msgr2=0x7fcbc00604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 --2- 172.21.15.26:0/4121656078 >> v2:172.21.15.120:3300/0 conn(0x7fcbc00b24c0 0x7fcbc00604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/4121656078 >> v1:172.21.15.26:6789/0 conn(0x7fcbc01465b0 legacy=0x7fcbc0066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbdffb700 1 -- 172.21.15.26:0/4121656078 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157886667 0 0) 0x7fcbc00b1ae0 con 0x7fcbc01465b0 2022-07-20T02:29:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/4121656078 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00b2020 con 0x7fcbc005f840 2022-07-20T02:29:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbf7fe700 1 --2- 172.21.15.26:0/4121656078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc005f840 0x7fcbc00609d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fcbb80032f0 tx=0x7fcbb8009290).ready entity=mon.0 client_cookie=26606119698cc889 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbdffb700 1 -- 172.21.15.26:0/4121656078 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb800b870 con 0x7fcbc005f840 2022-07-20T02:29:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbdffb700 1 -- 172.21.15.26:0/4121656078 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb800b9d0 con 0x7fcbc005f840 2022-07-20T02:29:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbdffb700 1 -- 172.21.15.26:0/4121656078 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb800bb40 con 0x7fcbc005f840 2022-07-20T02:29:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- 172.21.15.26:0/4121656078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc005f840 msgr2=0x7fcbc00609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- 172.21.15.26:0/4121656078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc005f840 0x7fcbc00609d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fcbb80032f0 tx=0x7fcbb8009290).stop 2022-07-20T02:29:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- 172.21.15.26:0/4121656078 shutdown_connections 2022-07-20T02:29:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- 172.21.15.26:0/4121656078 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc005f840 0x7fcbc00609d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- 172.21.15.26:0/4121656078 >> v2:172.21.15.120:3300/0 conn(0x7fcbc00b24c0 0x7fcbc00604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- 172.21.15.26:0/4121656078 >> 172.21.15.26:0/4121656078 conn(0x7fcbc00b3050 msgr2=0x7fcbc00b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- 172.21.15.26:0/4121656078 shutdown_connections 2022-07-20T02:29:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- 172.21.15.26:0/4121656078 wait complete. 2022-07-20T02:29:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 Processor -- start 2022-07-20T02:29:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- start start 2022-07-20T02:29:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc005f840 0x7fcbc0155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc01465b0 0x7fcbc00ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0156040 con 0x7fcbc0063800 2022-07-20T02:29:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00ae9b0 con 0x7fcbc005f840 2022-07-20T02:29:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.068+0000 7fcbbffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00642e0 con 0x7fcbc01465b0 2022-07-20T02:29:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37736/0 (socket says 172.21.15.26:37736) 2022-07-20T02:29:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/3183963334 learned_addr learned my addr 172.21.15.26:0/3183963334 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbeffd700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc005f840 0x7fcbc0155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbe7fc700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc01465b0 0x7fcbc00ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc01465b0 msgr2=0x7fcbc00ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc01465b0 0x7fcbc00ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc005f840 msgr2=0x7fcbc0155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc005f840 0x7fcbc0155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 -- 172.21.15.26:0/3183963334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8009530 con 0x7fcbc0063800 2022-07-20T02:29:05.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbf7fe700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fcbb80032f0 tx=0x7fcbb8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb800b870 con 0x7fcbc0063800 2022-07-20T02:29:05.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb8021630 con 0x7fcbc0063800 2022-07-20T02:29:05.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc0156270 con 0x7fcbc0063800 2022-07-20T02:29:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb8021990 con 0x7fcbc0063800 2022-07-20T02:29:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc0064580 con 0x7fcbc0063800 2022-07-20T02:29:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.072+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0065300 con 0x7fcbc0063800 2022-07-20T02:29:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.076+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbb800f070 con 0x7fcbc0063800 2022-07-20T02:29:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.076+0000 7fcbaffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.076+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fcbb806c0b0 con 0x7fcbc0063800 2022-07-20T02:29:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.076+0000 7fcbbeffd700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.076+0000 7fcbbeffd700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 0x7fcb9c055e10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fcbb400d000 tx=0x7fcbb4007b80).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.080+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbb80430a0 con 0x7fcbc0063800 2022-07-20T02:29:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.208+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fcbc00674c0 con 0x7fcbc0063800 2022-07-20T02:29:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.468+0000 7fcbaffff700 1 -- 172.21.15.26:0/3183963334 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v51) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcbb8015020 con 0x7fcbc0063800 2022-07-20T02:29:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 msgr2=0x7fcb9c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 0x7fcb9c055e10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fcbb400d000 tx=0x7fcbb4007b80).stop 2022-07-20T02:29:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 msgr2=0x7fcbc0065530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fcbb80032f0 tx=0x7fcbb8016000).stop 2022-07-20T02:29:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 shutdown_connections 2022-07-20T02:29:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbc0063800 0x7fcbc0065530 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbc01465b0 0x7fcbc00ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 --2- 172.21.15.26:0/3183963334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc005f840 0x7fcbc0155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 >> 172.21.15.26:0/3183963334 conn(0x7fcbc00b3050 msgr2=0x7fcbc013f920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.472+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 shutdown_connections 2022-07-20T02:29:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.476+0000 7fcbbffff700 1 -- 172.21.15.26:0/3183963334 wait complete. 2022-07-20T02:29:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:29:05.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:29:05.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.600+0000 7f63f5abd700 1 Processor -- start 2022-07-20T02:29:05.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.600+0000 7f63f5abd700 1 -- start start 2022-07-20T02:29:05.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f5abd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63f00b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f5abd700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f005d3b0 con 0x7f63f0142fa0 2022-07-20T02:29:05.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f5abd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00b1a60 con 0x7f63f005cfd0 2022-07-20T02:29:05.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f5abd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00b21a0 con 0x7f63f00aaf70 2022-07-20T02:29:05.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63effff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f63f0142fa0 0x7f63f0143810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37022/0 (socket says 172.21.15.26:37022) 2022-07-20T02:29:05.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63effff700 1 -- 172.21.15.26:0/1804796553 learned_addr learned my addr 172.21.15.26:0/1804796553 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:05.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63f00b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303867608 0 0) 0x7f63f005d3b0 con 0x7f63f0142fa0 2022-07-20T02:29:05.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d4003260 con 0x7f63f0142fa0 2022-07-20T02:29:05.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142692554 0 0) 0x7f63f00b1a60 con 0x7f63f005cfd0 2022-07-20T02:29:05.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f005d3b0 con 0x7f63f005cfd0 2022-07-20T02:29:05.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366486334 0 0) 0x7f63d4003260 con 0x7f63f0142fa0 2022-07-20T02:29:05.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 msgr2=0x7f63f00b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:05.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63f00b1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 >> v1:172.21.15.120:6789/0 conn(0x7f63f005cfd0 legacy=0x7f63f0142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00b1520 con 0x7f63f0142fa0 2022-07-20T02:29:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f63dc001660 con 0x7f63f0142fa0 2022-07-20T02:29:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 >> v1:172.21.15.26:6789/0 conn(0x7f63f0142fa0 legacy=0x7f63f0143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d4003910 0x7f63d4003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63d4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63d40049f0 0x7f63d4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00b21a0 con 0x7f63d40049f0 2022-07-20T02:29:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d400b780 con 0x7f63d4003910 2022-07-20T02:29:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d400b940 con 0x7f63f00aaf70 2022-07-20T02:29:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63d40049f0 0x7f63d4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63effff700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63d4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f4abb700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d4003910 0x7f63d4003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 -- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 msgr2=0x7f63d4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f00aaf70 0x7f63d4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 -- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d4003910 msgr2=0x7f63d4003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d4003910 0x7f63d4003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 -- 172.21.15.26:0/1804796553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d4003260 con 0x7f63d40049f0 2022-07-20T02:29:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.604+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63d40049f0 0x7f63d4004e70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f63e800c000 tx=0x7f63e80091d0).ready entity=mon.0 client_cookie=aeb733424032577b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e800e7e0 con 0x7f63d40049f0 2022-07-20T02:29:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e800e940 con 0x7f63d40049f0 2022-07-20T02:29:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63ef7fe700 1 -- 172.21.15.26:0/1804796553 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e800eca0 con 0x7f63d40049f0 2022-07-20T02:29:05.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- 172.21.15.26:0/1804796553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63d40049f0 msgr2=0x7f63d4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1804796553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63d40049f0 0x7f63d4004e70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f63e800c000 tx=0x7f63e80091d0).stop 2022-07-20T02:29:05.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f52bc700 1 -- 172.21.15.26:0/1804796553 reap_dead start 2022-07-20T02:29:05.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- 172.21.15.26:0/1804796553 shutdown_connections 2022-07-20T02:29:05.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- 172.21.15.26:0/1804796553 >> 172.21.15.26:0/1804796553 conn(0x7f63f00b7090 msgr2=0x7f63f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:05.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- 172.21.15.26:0/1804796553 shutdown_connections 2022-07-20T02:29:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- 172.21.15.26:0/1804796553 wait complete. 2022-07-20T02:29:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 Processor -- start 2022-07-20T02:29:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.608+0000 7f63f5abd700 1 -- start start 2022-07-20T02:29:05.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f0060d00 0x7f63f013f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f00a3990 0x7f63f013fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00610e0 con 0x7f63f00654c0 2022-07-20T02:29:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f0140330 con 0x7f63f00a3990 2022-07-20T02:29:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00a3020 con 0x7f63f0060d00 2022-07-20T02:29:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37756/0 (socket says 172.21.15.26:37756) 2022-07-20T02:29:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 -- 172.21.15.26:0/1218227055 learned_addr learned my addr 172.21.15.26:0/1218227055 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63effff700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f00a3990 0x7f63f013fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f4abb700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f0060d00 0x7f63f013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 -- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f0060d00 msgr2=0x7f63f013f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f0060d00 0x7f63f013f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 -- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f00a3990 msgr2=0x7f63f013fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f00a3990 0x7f63f013fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 -- 172.21.15.26:0/1218227055 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00a32c0 con 0x7f63f00654c0 2022-07-20T02:29:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f52bc700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f63f0061af0 tx=0x7f63e801a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e800e7e0 con 0x7f63f00654c0 2022-07-20T02:29:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e80127f0 con 0x7f63f00654c0 2022-07-20T02:29:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f00a3560 con 0x7f63f00654c0 2022-07-20T02:29:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e8009600 con 0x7f63f00654c0 2022-07-20T02:29:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f0156270 con 0x7f63f00654c0 2022-07-20T02:29:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.612+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f00a36f0 con 0x7f63f00654c0 2022-07-20T02:29:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.616+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f63e802d070 con 0x7f63f00654c0 2022-07-20T02:29:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.616+0000 7f63edffb700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 0x7f63d0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.616+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f63e806dd70 con 0x7f63f00654c0 2022-07-20T02:29:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.616+0000 7f63f4abb700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 0x7f63d0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.616+0000 7f63f4abb700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 0x7f63d0055f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f63e4003000 tx=0x7f63e4007a00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:05.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.624+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63e8044bf0 con 0x7f63f00654c0 2022-07-20T02:29:05.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:05.752+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f63f00b17c0 con 0x7f63f00654c0 2022-07-20T02:29:06.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.296+0000 7f63edffb700 1 -- 172.21.15.26:0/1218227055 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v52) v1 ==== 125+0+0 (secure 0 0 0) 0x7f63e8016070 con 0x7f63f00654c0 2022-07-20T02:29:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 msgr2=0x7f63d0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 0x7f63d0055f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f63e4003000 tx=0x7f63e4007a00).stop 2022-07-20T02:29:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 msgr2=0x7f63f0155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f63f0061af0 tx=0x7f63e801a000).stop 2022-07-20T02:29:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 shutdown_connections 2022-07-20T02:29:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63f00654c0 0x7f63f0155d80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63d0053a70 0x7f63d0055f30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f00a3990 0x7f63f013fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 --2- 172.21.15.26:0/1218227055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f0060d00 0x7f63f013f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 >> 172.21.15.26:0/1218227055 conn(0x7f63f00b7090 msgr2=0x7f63f0143590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 shutdown_connections 2022-07-20T02:29:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.304+0000 7f63f5abd700 1 -- 172.21.15.26:0/1218227055 wait complete. 2022-07-20T02:29:06.313 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:29:06.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:29:06.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 Processor -- start 2022-07-20T02:29:06.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 -- start start 2022-07-20T02:29:06.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f780add30 0x7f1f780b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 0x7f1f78060dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780aa1e0 con 0x7f1f780612b0 2022-07-20T02:29:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780ae110 con 0x7f1f780add30 2022-07-20T02:29:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f7d6ee700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780a9f40 con 0x7f1f78060950 2022-07-20T02:29:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f77fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f780add30 0x7f1f780b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f777fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 0x7f1f78060dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f77fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f780add30 0x7f1f780b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37760/0 (socket says 172.21.15.26:37760) 2022-07-20T02:29:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f77fff700 1 -- 172.21.15.26:0/552508094 learned_addr learned my addr 172.21.15.26:0/552508094 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.432+0000 7f1f77fff700 1 -- 172.21.15.26:0/552508094 >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 msgr2=0x7f1f78060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f77fff700 1 --2- 172.21.15.26:0/552508094 >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 0x7f1f78060dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f77fff700 1 -- 172.21.15.26:0/552508094 >> v1:172.21.15.120:6789/0 conn(0x7f1f780612b0 legacy=0x7f1f78143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f77fff700 1 -- 172.21.15.26:0/552508094 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780b1110 con 0x7f1f780add30 2022-07-20T02:29:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f76ffd700 1 -- 172.21.15.26:0/552508094 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454280594 0 0) 0x7f1f780aa1e0 con 0x7f1f780612b0 2022-07-20T02:29:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f777fe700 1 --2- 172.21.15.26:0/552508094 >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 0x7f1f78060dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f77fff700 1 --2- 172.21.15.26:0/552508094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780add30 0x7f1f780b2ab0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f1f68002f70 tx=0x7f1f68006eb0).ready entity=mon.0 client_cookie=4fff32bbb6bbf55a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f76ffd700 1 -- 172.21.15.26:0/552508094 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f6800d5c0 con 0x7f1f780add30 2022-07-20T02:29:06.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f76ffd700 1 -- 172.21.15.26:0/552508094 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68000e90 con 0x7f1f780add30 2022-07-20T02:29:06.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f76ffd700 1 -- 172.21.15.26:0/552508094 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f6800d910 con 0x7f1f780add30 2022-07-20T02:29:06.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/552508094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780add30 msgr2=0x7f1f780b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/552508094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780add30 0x7f1f780b2ab0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f1f68002f70 tx=0x7f1f68006eb0).stop 2022-07-20T02:29:06.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/552508094 shutdown_connections 2022-07-20T02:29:06.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/552508094 >> v2:172.21.15.120:3300/0 conn(0x7f1f78060950 0x7f1f78060dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/552508094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780add30 0x7f1f780b2ab0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/552508094 >> 172.21.15.26:0/552508094 conn(0x7f1f780b3050 msgr2=0x7f1f780b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:06.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/552508094 shutdown_connections 2022-07-20T02:29:06.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/552508094 wait complete. 2022-07-20T02:29:06.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 Processor -- start 2022-07-20T02:29:06.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- start start 2022-07-20T02:29:06.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f78060950 0x7f1f7813f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f780add30 0x7f1f7814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78153460 con 0x7f1f780612b0 2022-07-20T02:29:06.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78153700 con 0x7f1f780add30 2022-07-20T02:29:06.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.436+0000 7f1f7d6ee700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781539a0 con 0x7f1f78060950 2022-07-20T02:29:06.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37768/0 (socket says 172.21.15.26:37768) 2022-07-20T02:29:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 -- 172.21.15.26:0/4017991433 learned_addr learned my addr 172.21.15.26:0/4017991433 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f77fff700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f78060950 0x7f1f7813f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f7ceed700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f780add30 0x7f1f7814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 -- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f78060950 msgr2=0x7f1f7813f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f78060950 0x7f1f7813f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 -- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f780add30 msgr2=0x7f1f7814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f780add30 0x7f1f7814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 -- 172.21.15.26:0/4017991433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c00b180 con 0x7f1f780612b0 2022-07-20T02:29:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f777fe700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f1f6c00c000 tx=0x7f1f6c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f6c00c7e0 con 0x7f1f780612b0 2022-07-20T02:29:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c00c940 con 0x7f1f780612b0 2022-07-20T02:29:06.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6800b1e0 con 0x7f1f780612b0 2022-07-20T02:29:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f6c00cca0 con 0x7f1f780612b0 2022-07-20T02:29:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f781401e0 con 0x7f1f780612b0 2022-07-20T02:29:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f781473d0 con 0x7f1f780612b0 2022-07-20T02:29:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f6c01c460 con 0x7f1f780612b0 2022-07-20T02:29:06.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 0x7f1f58055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f1f6c06c600 con 0x7f1f780612b0 2022-07-20T02:29:06.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.440+0000 7f1f77fff700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 0x7f1f58055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.444+0000 7f1f77fff700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 0x7f1f58055f30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1f6801c000 tx=0x7f1f68015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.452+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f6c043340 con 0x7f1f780612b0 2022-07-20T02:29:06.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.616+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1f780b13b0 con 0x7f1f780612b0 2022-07-20T02:29:06.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.628+0000 7f1f757fa700 1 -- 172.21.15.26:0/4017991433 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v23) v1 ==== 188+0+0 (secure 0 0 0) 0x7f1f6c046960 con 0x7f1f780612b0 2022-07-20T02:29:06.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 msgr2=0x7f1f58055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 0x7f1f58055f30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1f6801c000 tx=0x7f1f68015000).stop 2022-07-20T02:29:06.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 msgr2=0x7f1f7813fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f1f6c00c000 tx=0x7f1f6c006df0).stop 2022-07-20T02:29:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 shutdown_connections 2022-07-20T02:29:06.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f780612b0 0x7f1f7813fcf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f58053a70 0x7f1f58055f30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f780add30 0x7f1f7814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 --2- 172.21.15.26:0/4017991433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f78060950 0x7f1f7813f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 >> 172.21.15.26:0/4017991433 conn(0x7f1f780b3050 msgr2=0x7f1f78142e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:06.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 shutdown_connections 2022-07-20T02:29:06.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.632+0000 7f1f7d6ee700 1 -- 172.21.15.26:0/4017991433 wait complete. 2022-07-20T02:29:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 9 and data pool 10 2022-07-20T02:29:06.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:06.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 Processor -- start 2022-07-20T02:29:06.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 -- start start 2022-07-20T02:29:06.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e08147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3e08067190 0x7f3e08148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080b1bc0 con 0x7f3e08061050 2022-07-20T02:29:06.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0814b040 con 0x7f3e08067190 2022-07-20T02:29:06.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0cf1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080a8860 con 0x7f3e080b2920 2022-07-20T02:29:06.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3dfdd9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3e08061050 0x7f3e08066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37068/0 (socket says 172.21.15.26:37068) 2022-07-20T02:29:06.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3dfdd9b700 1 -- 172.21.15.26:0/3944344700 learned_addr learned my addr 172.21.15.26:0/3944344700 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 --2- 172.21.15.26:0/3944344700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e08147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e06d9d700 1 --2- 172.21.15.26:0/3944344700 >> v2:172.21.15.120:3300/0 conn(0x7f3e08067190 0x7f3e08148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e05d9b700 1 -- 172.21.15.26:0/3944344700 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483363945 0 0) 0x7f3e080b1bc0 con 0x7f3e08061050 2022-07-20T02:29:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e05d9b700 1 -- 172.21.15.26:0/3944344700 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dec003260 con 0x7f3e08061050 2022-07-20T02:29:06.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 -- 172.21.15.26:0/3944344700 >> v2:172.21.15.120:3300/0 conn(0x7f3e08067190 msgr2=0x7f3e08148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:06.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 --2- 172.21.15.26:0/3944344700 >> v2:172.21.15.120:3300/0 conn(0x7f3e08067190 0x7f3e08148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 -- 172.21.15.26:0/3944344700 >> v1:172.21.15.26:6789/0 conn(0x7f3e08061050 legacy=0x7f3e08066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 -- 172.21.15.26:0/3944344700 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e08147700 con 0x7f3e080b2920 2022-07-20T02:29:06.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e0659c700 1 --2- 172.21.15.26:0/3944344700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e08147d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3df8002f70 tx=0x7f3df8006df0).ready entity=mon.2 client_cookie=bc61acf46fbfea99 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e05d9b700 1 -- 172.21.15.26:0/3944344700 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df800d4a0 con 0x7f3e080b2920 2022-07-20T02:29:06.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e05d9b700 1 -- 172.21.15.26:0/3944344700 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df8000e90 con 0x7f3e080b2920 2022-07-20T02:29:06.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.752+0000 7f3e05d9b700 1 -- 172.21.15.26:0/3944344700 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df800d7f0 con 0x7f3e080b2920 2022-07-20T02:29:06.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/3944344700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 msgr2=0x7f3e08147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/3944344700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e08147d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3df8002f70 tx=0x7f3df8006df0).stop 2022-07-20T02:29:06.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/3944344700 shutdown_connections 2022-07-20T02:29:06.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/3944344700 >> v2:172.21.15.120:3300/0 conn(0x7f3e08067190 0x7f3e08148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/3944344700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e08147d90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/3944344700 >> 172.21.15.26:0/3944344700 conn(0x7f3e080b60a0 msgr2=0x7f3e080b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/3944344700 shutdown_connections 2022-07-20T02:29:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/3944344700 wait complete. 2022-07-20T02:29:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 Processor -- start 2022-07-20T02:29:06.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- start start 2022-07-20T02:29:06.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e08061050 0x7f3e080aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e080aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0815a5d0 con 0x7f3e08067190 2022-07-20T02:29:06.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08154370 con 0x7f3e08061050 2022-07-20T02:29:06.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.756+0000 7f3e0cf1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080a98d0 con 0x7f3e080b2920 2022-07-20T02:29:06.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37794/0 (socket says 172.21.15.26:37794) 2022-07-20T02:29:06.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3e0659c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e08061050 0x7f3e080aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 -- 172.21.15.26:0/1770617041 learned_addr learned my addr 172.21.15.26:0/1770617041 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:06.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3e06d9d700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e080aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 -- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 msgr2=0x7f3e080aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e080aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 -- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e08061050 msgr2=0x7f3e080aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e08061050 0x7f3e080aa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 -- 172.21.15.26:0/1770617041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df800b180 con 0x7f3e08067190 2022-07-20T02:29:06.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfdd9b700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f3df0001110 tx=0x7f3df0007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df00070d0 con 0x7f3e08067190 2022-07-20T02:29:06.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df0007230 con 0x7f3e08067190 2022-07-20T02:29:06.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0815a850 con 0x7f3e08067190 2022-07-20T02:29:06.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df0005610 con 0x7f3e08067190 2022-07-20T02:29:06.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e08066440 con 0x7f3e08067190 2022-07-20T02:29:06.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.760+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0814ada0 con 0x7f3e08067190 2022-07-20T02:29:06.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.764+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3df0010070 con 0x7f3e08067190 2022-07-20T02:29:06.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.764+0000 7f3dfffff700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 0x7f3de0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:06.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.764+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3df006b7a0 con 0x7f3e08067190 2022-07-20T02:29:06.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.764+0000 7f3e0659c700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 0x7f3de0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:06.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.764+0000 7f3e0659c700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 0x7f3de0055ed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f3df801a000 tx=0x7f3df8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:06.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.768+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3df0042560 con 0x7f3e08067190 2022-07-20T02:29:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.896+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e080574e0 con 0x7f3e08067190 2022-07-20T02:29:06.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.896+0000 7f3dfffff700 1 -- 172.21.15.26:0/1770617041 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16334 (secure 0 0 0) 0x7f3df0045b80 con 0x7f3e08067190 2022-07-20T02:29:06.900 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:06.901 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":54,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:06.619435+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:29:05.266754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T02:29:05.807643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:06.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 msgr2=0x7f3de0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 0x7f3de0055ed0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f3df801a000 tx=0x7f3df8014000).stop 2022-07-20T02:29:06.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 msgr2=0x7f3e081566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:06.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f3df0001110 tx=0x7f3df0007b50).stop 2022-07-20T02:29:06.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 shutdown_connections 2022-07-20T02:29:06.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3e08067190 0x7f3e081566f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3de0053a10 0x7f3de0055ed0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e080b2920 0x7f3e080aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 --2- 172.21.15.26:0/1770617041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e08061050 0x7f3e080aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:06.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 >> 172.21.15.26:0/1770617041 conn(0x7f3e080b60a0 msgr2=0x7f3e08148a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:06.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 shutdown_connections 2022-07-20T02:29:06.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:06.900+0000 7f3e0cf1e700 1 -- 172.21.15.26:0/1770617041 wait complete. 2022-07-20T02:29:06.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:07.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 Processor -- start 2022-07-20T02:29:07.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 -- start start 2022-07-20T02:29:07.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70fc05cfd0 0x7f70fc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 0x7f70fc13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc1474e0 con 0x7f70fc147000 2022-07-20T02:29:07.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70fc0b4e40 con 0x7f70fc05cfd0 2022-07-20T02:29:07.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fbfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc05d3b0 con 0x7f70fc060f00 2022-07-20T02:29:07.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70faffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70fc05cfd0 0x7f70fc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fb7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 0x7f70fc13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70faffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70fc05cfd0 0x7f70fc143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37802/0 (socket says 172.21.15.26:37802) 2022-07-20T02:29:07.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70fa7fc700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f70fc147000 0x7f70fc0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51292/0 (socket says 172.21.15.26:51292) 2022-07-20T02:29:07.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.032+0000 7f70faffd700 1 -- 172.21.15.26:0/3151095056 learned_addr learned my addr 172.21.15.26:0/3151095056 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70f9ffb700 1 -- 172.21.15.26:0/3151095056 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128542321 0 0) 0x7f70fc1474e0 con 0x7f70fc147000 2022-07-20T02:29:07.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70f9ffb700 1 -- 172.21.15.26:0/3151095056 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc003260 con 0x7f70fc147000 2022-07-20T02:29:07.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70faffd700 1 -- 172.21.15.26:0/3151095056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 msgr2=0x7f70fc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:07.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70faffd700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 0x7f70fc13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70faffd700 1 -- 172.21.15.26:0/3151095056 >> v1:172.21.15.120:6789/0 conn(0x7f70fc147000 legacy=0x7f70fc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70faffd700 1 -- 172.21.15.26:0/3151095056 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc1431e0 con 0x7f70fc05cfd0 2022-07-20T02:29:07.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fb7fe700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 0x7f70fc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70faffd700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc05cfd0 0x7f70fc143bc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f70ec002f70 tx=0x7f70ec006eb0).ready entity=mon.0 client_cookie=8c11d0bc1af8e590 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70f9ffb700 1 -- 172.21.15.26:0/3151095056 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70ec00d5c0 con 0x7f70fc05cfd0 2022-07-20T02:29:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70f9ffb700 1 -- 172.21.15.26:0/3151095056 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec000e90 con 0x7f70fc05cfd0 2022-07-20T02:29:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70f9ffb700 1 -- 172.21.15.26:0/3151095056 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70ec00d950 con 0x7f70fc05cfd0 2022-07-20T02:29:07.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- 172.21.15.26:0/3151095056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc05cfd0 msgr2=0x7f70fc143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc05cfd0 0x7f70fc143bc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f70ec002f70 tx=0x7f70ec006eb0).stop 2022-07-20T02:29:07.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- 172.21.15.26:0/3151095056 shutdown_connections 2022-07-20T02:29:07.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc060f00 0x7f70fc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 --2- 172.21.15.26:0/3151095056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc05cfd0 0x7f70fc143bc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- 172.21.15.26:0/3151095056 >> 172.21.15.26:0/3151095056 conn(0x7f70fc0620c0 msgr2=0x7f70fc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- 172.21.15.26:0/3151095056 shutdown_connections 2022-07-20T02:29:07.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- 172.21.15.26:0/3151095056 wait complete. 2022-07-20T02:29:07.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 Processor -- start 2022-07-20T02:29:07.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.036+0000 7f70fbfff700 1 -- start start 2022-07-20T02:29:07.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70fc0ad8a0 0x7f70fc155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc155fd0 0x7f70fc156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0add00 con 0x7f70fc147000 2022-07-20T02:29:07.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc138cc0 con 0x7f70fc0ad8a0 2022-07-20T02:29:07.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc138f50 con 0x7f70fc155fd0 2022-07-20T02:29:07.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37806/0 (socket says 172.21.15.26:37806) 2022-07-20T02:29:07.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 -- 172.21.15.26:0/2316907231 learned_addr learned my addr 172.21.15.26:0/2316907231 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fa7fc700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70fc0ad8a0 0x7f70fc155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fb7fe700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc155fd0 0x7f70fc156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 -- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc155fd0 msgr2=0x7f70fc156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc155fd0 0x7f70fc156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 -- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70fc0ad8a0 msgr2=0x7f70fc155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70fc0ad8a0 0x7f70fc155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 -- 172.21.15.26:0/2316907231 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f400b180 con 0x7f70fc147000 2022-07-20T02:29:07.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70faffd700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f70ec002f70 tx=0x7f70ec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70ec00de80 con 0x7f70fc147000 2022-07-20T02:29:07.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0068b0 con 0x7f70fc147000 2022-07-20T02:29:07.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec00b1e0 con 0x7f70fc147000 2022-07-20T02:29:07.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70ec011d70 con 0x7f70fc147000 2022-07-20T02:29:07.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc0adf30 con 0x7f70fc147000 2022-07-20T02:29:07.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc0b81a0 con 0x7f70fc147000 2022-07-20T02:29:07.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.040+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70ec025030 con 0x7f70fc147000 2022-07-20T02:29:07.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.044+0000 7f70f3fff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 0x7f70e0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.044+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f70ec06cfe0 con 0x7f70fc147000 2022-07-20T02:29:07.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.044+0000 7f70fa7fc700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 0x7f70e0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.044+0000 7f70fa7fc700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 0x7f70e0055ed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f70e400d000 tx=0x7f70e4000f40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.048+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70ec043da0 con 0x7f70fc147000 2022-07-20T02:29:07.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.172+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70fc143480 con 0x7f70fc147000 2022-07-20T02:29:07.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.172+0000 7f70f3fff700 1 -- 172.21.15.26:0/2316907231 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16334 (secure 0 0 0) 0x7f70ec023070 con 0x7f70fc147000 2022-07-20T02:29:07.175 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:07.175 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":54,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:06.619435+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:29:05.266754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T02:29:05.807643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 msgr2=0x7f70e0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 0x7f70e0055ed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f70e400d000 tx=0x7f70e4000f40).stop 2022-07-20T02:29:07.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 msgr2=0x7f70fc145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f70ec002f70 tx=0x7f70ec015000).stop 2022-07-20T02:29:07.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 shutdown_connections 2022-07-20T02:29:07.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f70e0053a10 0x7f70e0055ed0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70fc155fd0 0x7f70fc156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70fc0ad8a0 0x7f70fc155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 --2- 172.21.15.26:0/2316907231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70fc147000 0x7f70fc145970 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 >> 172.21.15.26:0/2316907231 conn(0x7f70fc060f00 msgr2=0x7f70fc1365f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 shutdown_connections 2022-07-20T02:29:07.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.176+0000 7f70fbfff700 1 -- 172.21.15.26:0/2316907231 wait complete. 2022-07-20T02:29:07.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:29:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 Processor -- start 2022-07-20T02:29:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 -- start start 2022-07-20T02:29:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29ec05cfd0 0x7f29ec1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29ec0b1160 0x7f29ec140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec146e80 0x7f29ec1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ec1435b0 con 0x7f29ec0b1160 2022-07-20T02:29:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ec05d3b0 con 0x7f29ec05cfd0 2022-07-20T02:29:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29f2fd6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0b4bc0 con 0x7f29ec146e80 2022-07-20T02:29:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.316+0000 7f29ebfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29ec0b1160 0x7f29ec140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f29ec0b1160 0x7f29ec140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37814/0 (socket says 172.21.15.26:37814) 2022-07-20T02:29:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 -- 172.21.15.26:0/1669248497 learned_addr learned my addr 172.21.15.26:0/1669248497 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f0d72700 1 --2- 172.21.15.26:0/1669248497 >> v2:172.21.15.120:3300/0 conn(0x7f29ec05cfd0 0x7f29ec1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f1573700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec146e80 0x7f29ec1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 -- 172.21.15.26:0/1669248497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec146e80 msgr2=0x7f29ec1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec146e80 0x7f29ec1472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 -- 172.21.15.26:0/1669248497 >> v2:172.21.15.120:3300/0 conn(0x7f29ec05cfd0 msgr2=0x7f29ec1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 --2- 172.21.15.26:0/1669248497 >> v2:172.21.15.120:3300/0 conn(0x7f29ec05cfd0 0x7f29ec1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 -- 172.21.15.26:0/1669248497 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec0b4e60 con 0x7f29ec0b1160 2022-07-20T02:29:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29ebfff700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec140cb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f29e0002f70 tx=0x7f29e0006eb0).ready entity=mon.0 client_cookie=bed29a78edef62e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29eb7fe700 1 -- 172.21.15.26:0/1669248497 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e000d570 con 0x7f29ec0b1160 2022-07-20T02:29:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29eb7fe700 1 -- 172.21.15.26:0/1669248497 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e0000e90 con 0x7f29ec0b1160 2022-07-20T02:29:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29eb7fe700 1 -- 172.21.15.26:0/1669248497 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e000d900 con 0x7f29ec0b1160 2022-07-20T02:29:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 -- 172.21.15.26:0/1669248497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 msgr2=0x7f29ec140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec140cb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f29e0002f70 tx=0x7f29e0006eb0).stop 2022-07-20T02:29:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 -- 172.21.15.26:0/1669248497 shutdown_connections 2022-07-20T02:29:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec146e80 0x7f29ec1472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/1669248497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec140cb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/1669248497 >> v2:172.21.15.120:3300/0 conn(0x7f29ec05cfd0 0x7f29ec1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 -- 172.21.15.26:0/1669248497 >> 172.21.15.26:0/1669248497 conn(0x7f29ec066f70 msgr2=0x7f29ec05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 -- 172.21.15.26:0/1669248497 shutdown_connections 2022-07-20T02:29:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.320+0000 7f29f2fd6700 1 -- 172.21.15.26:0/1669248497 wait complete. 2022-07-20T02:29:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 Processor -- start 2022-07-20T02:29:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- start start 2022-07-20T02:29:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec05cfd0 0x7f29ec0affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec146e80 0x7f29ec0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0b89a0 con 0x7f29ec0b1160 2022-07-20T02:29:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0b8050 con 0x7f29ec146e80 2022-07-20T02:29:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec1534a0 con 0x7f29ec05cfd0 2022-07-20T02:29:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37822/0 (socket says 172.21.15.26:37822) 2022-07-20T02:29:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 -- 172.21.15.26:0/188122608 learned_addr learned my addr 172.21.15.26:0/188122608 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f1573700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec146e80 0x7f29ec0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f0d72700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec05cfd0 0x7f29ec0affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 -- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec05cfd0 msgr2=0x7f29ec0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec05cfd0 0x7f29ec0affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 -- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec146e80 msgr2=0x7f29ec0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec146e80 0x7f29ec0b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 -- 172.21.15.26:0/188122608 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e000b1e0 con 0x7f29ec0b1160 2022-07-20T02:29:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29ebfff700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f29e0002f70 tx=0x7f29e0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e00116e0 con 0x7f29ec0b1160 2022-07-20T02:29:07.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e0011840 con 0x7f29ec0b1160 2022-07-20T02:29:07.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec153740 con 0x7f29ec0b1160 2022-07-20T02:29:07.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29e0011ba0 con 0x7f29ec0b1160 2022-07-20T02:29:07.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.324+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec13f570 con 0x7f29ec0b1160 2022-07-20T02:29:07.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec153250 con 0x7f29ec0b1160 2022-07-20T02:29:07.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29e0025030 con 0x7f29ec0b1160 2022-07-20T02:29:07.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29e9ffb700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 0x7f29cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f29e006d120 con 0x7f29ec0b1160 2022-07-20T02:29:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29f0d72700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 0x7f29cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.328+0000 7f29f0d72700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 0x7f29cc055f30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f29d800b000 tx=0x7f29d8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.336+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29e0043ee0 con 0x7f29ec0b1160 2022-07-20T02:29:07.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.496+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f29ec0574e0 con 0x7f29ec0b1160 2022-07-20T02:29:07.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29e9ffb700 1 -- 172.21.15.26:0/188122608 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f29e00230b0 con 0x7f29ec0b1160 2022-07-20T02:29:07.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 msgr2=0x7f29cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 0x7f29cc055f30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f29d800b000 tx=0x7f29d8006df0).stop 2022-07-20T02:29:07.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 msgr2=0x7f29ec0b7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f29e0002f70 tx=0x7f29e0015000).stop 2022-07-20T02:29:07.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 shutdown_connections 2022-07-20T02:29:07.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29ec0b1160 0x7f29ec0b7670 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f29cc053a70 0x7f29cc055f30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29ec146e80 0x7f29ec0b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 --2- 172.21.15.26:0/188122608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29ec05cfd0 0x7f29ec0affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 >> 172.21.15.26:0/188122608 conn(0x7f29ec066f70 msgr2=0x7f29ec0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 shutdown_connections 2022-07-20T02:29:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.640+0000 7f29f2fd6700 1 -- 172.21.15.26:0/188122608 wait complete. 2022-07-20T02:29:07.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:07.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.780+0000 7fe7165a9700 1 Processor -- start 2022-07-20T02:29:07.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.780+0000 7fe7165a9700 1 -- start start 2022-07-20T02:29:07.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe7100add30 0x7fe7100b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe7100afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7100aa1e0 con 0x7fe710060950 2022-07-20T02:29:07.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100ae110 con 0x7fe7100add30 2022-07-20T02:29:07.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7100a9f40 con 0x7fe7100af6d0 2022-07-20T02:29:07.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe7100add30 0x7fe7100b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe7100add30 0x7fe7100b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37828/0 (socket says 172.21.15.26:37828) 2022-07-20T02:29:07.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 -- 172.21.15.26:0/1103644297 learned_addr learned my addr 172.21.15.26:0/1103644297 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 -- 172.21.15.26:0/1103644297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 msgr2=0x7fe7100afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:07.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 --2- 172.21.15.26:0/1103644297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe7100afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:29:07.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 -- 172.21.15.26:0/1103644297 >> v1:172.21.15.26:6789/0 conn(0x7fe710060950 legacy=0x7fe710060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 -- 172.21.15.26:0/1103644297 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100b1110 con 0x7fe7100add30 2022-07-20T02:29:07.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7155a7700 1 --2- 172.21.15.26:0/1103644297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe7100b2ab0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c006f40).ready entity=mon.0 client_cookie=26a690aa682dbbbb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe6fffff700 1 -- 172.21.15.26:0/1103644297 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c00dc70 con 0x7fe7100add30 2022-07-20T02:29:07.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe6fffff700 1 -- 172.21.15.26:0/1103644297 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c000e90 con 0x7fe7100add30 2022-07-20T02:29:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe6fffff700 1 -- 172.21.15.26:0/1103644297 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70c00be90 con 0x7fe7100add30 2022-07-20T02:29:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- 172.21.15.26:0/1103644297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 msgr2=0x7fe7100b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 --2- 172.21.15.26:0/1103644297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe7100b2ab0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fe70c002f70 tx=0x7fe70c006f40).stop 2022-07-20T02:29:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- 172.21.15.26:0/1103644297 shutdown_connections 2022-07-20T02:29:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 --2- 172.21.15.26:0/1103644297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe7100afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:29:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 --2- 172.21.15.26:0/1103644297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe7100b2ab0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- 172.21.15.26:0/1103644297 >> 172.21.15.26:0/1103644297 conn(0x7fe7100b3050 msgr2=0x7fe7100b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- 172.21.15.26:0/1103644297 shutdown_connections 2022-07-20T02:29:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.784+0000 7fe7165a9700 1 -- 172.21.15.26:0/1103644297 wait complete. 2022-07-20T02:29:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 Processor -- start 2022-07-20T02:29:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 -- start start 2022-07-20T02:29:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe710060950 0x7fe710063800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe71013ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7100650f0 con 0x7fe7100add30 2022-07-20T02:29:07.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71013e340 con 0x7fe710060950 2022-07-20T02:29:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7165a9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71013e5e0 con 0x7fe7100af6d0 2022-07-20T02:29:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37836/0 (socket says 172.21.15.26:37836) 2022-07-20T02:29:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 -- 172.21.15.26:0/2243951698 learned_addr learned my addr 172.21.15.26:0/2243951698 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe7155a7700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe710060950 0x7fe710063800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe715da8700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe71013ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 -- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 msgr2=0x7fe71013ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe71013ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 -- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe710060950 msgr2=0x7fe710063800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe710060950 0x7fe710063800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 -- 172.21.15.26:0/2243951698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c00b460 con 0x7fe7100add30 2022-07-20T02:29:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.788+0000 7fe714da6700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fe7100630a0 tx=0x7fe700007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe700012400 con 0x7fe7100add30 2022-07-20T02:29:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe700012560 con 0x7fe7100add30 2022-07-20T02:29:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71013e880 con 0x7fe7100add30 2022-07-20T02:29:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7000128c0 con 0x7fe7100add30 2022-07-20T02:29:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71013dea0 con 0x7fe7100add30 2022-07-20T02:29:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7100b13b0 con 0x7fe7100add30 2022-07-20T02:29:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe700019030 con 0x7fe7100add30 2022-07-20T02:29:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 0x7fe6f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe70006b450 con 0x7fe7100add30 2022-07-20T02:29:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.792+0000 7fe7155a7700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 0x7fe6f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.796+0000 7fe7155a7700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 0x7fe6f0055f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe70c01e000 tx=0x7fe7100b3db0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.800+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe700042100 con 0x7fe7100add30 2022-07-20T02:29:07.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.964+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7100574e0 con 0x7fe7100add30 2022-07-20T02:29:07.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.964+0000 7fe6fe7fc700 1 -- 172.21.15.26:0/2243951698 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7fe700045720 con 0x7fe7100add30 2022-07-20T02:29:07.973 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:07.973 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:07.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 msgr2=0x7fe6f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 0x7fe6f0055f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe70c01e000 tx=0x7fe7100b3db0).stop 2022-07-20T02:29:07.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 msgr2=0x7fe710141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:07.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fe7100630a0 tx=0x7fe700007c50).stop 2022-07-20T02:29:07.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 shutdown_connections 2022-07-20T02:29:07.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe7100add30 0x7fe710141490 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe6f0053a70 0x7fe6f0055f30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7100af6d0 0x7fe71013ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 --2- 172.21.15.26:0/2243951698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe710060950 0x7fe710063800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:07.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 >> 172.21.15.26:0/2243951698 conn(0x7fe7100b3050 msgr2=0x7fe7100b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:07.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 shutdown_connections 2022-07-20T02:29:07.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:07.968+0000 7fe7165a9700 1 -- 172.21.15.26:0/2243951698 wait complete. 2022-07-20T02:29:07.997 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:07.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:08.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.108+0000 7f8bfb52d700 1 Processor -- start 2022-07-20T02:29:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.108+0000 7f8bfb52d700 1 -- start start 2022-07-20T02:29:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf40b2920 0x7f8bf4147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8bf4061050 0x7f8bf4148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bf4066a80 0x7f8bf4066ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf414b040 con 0x7f8bf4066a80 2022-07-20T02:29:08.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf40a8860 con 0x7f8bf4061050 2022-07-20T02:29:08.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40b1bc0 con 0x7f8bf40b2920 2022-07-20T02:29:08.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bf4066a80 0x7f8bf4066ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bf4066a80 0x7f8bf4066ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37846/0 (socket says 172.21.15.26:37846) 2022-07-20T02:29:08.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 -- 172.21.15.26:0/4166667577 learned_addr learned my addr 172.21.15.26:0/4166667577 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:08.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf8ac8700 1 --2- 172.21.15.26:0/4166667577 >> v2:172.21.15.120:3300/0 conn(0x7f8bf4061050 0x7f8bf4148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf92c9700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf40b2920 0x7f8bf4147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 -- 172.21.15.26:0/4166667577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf40b2920 msgr2=0x7f8bf4147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:08.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf40b2920 0x7f8bf4147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 -- 172.21.15.26:0/4166667577 >> v2:172.21.15.120:3300/0 conn(0x7f8bf4061050 msgr2=0x7f8bf4148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:08.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/4166667577 >> v2:172.21.15.120:3300/0 conn(0x7f8bf4061050 0x7f8bf4148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 -- 172.21.15.26:0/4166667577 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4147700 con 0x7f8bf4066a80 2022-07-20T02:29:08.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf4066a80 0x7f8bf4066ec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f8bec002f70 tx=0x7f8bec006eb0).ready entity=mon.0 client_cookie=e0c358cbeeeb8602 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bebfff700 1 -- 172.21.15.26:0/4166667577 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bec00d570 con 0x7f8bf4066a80 2022-07-20T02:29:08.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bebfff700 1 -- 172.21.15.26:0/4166667577 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec000e90 con 0x7f8bf4066a80 2022-07-20T02:29:08.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bebfff700 1 -- 172.21.15.26:0/4166667577 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bec00d900 con 0x7f8bf4066a80 2022-07-20T02:29:08.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- 172.21.15.26:0/4166667577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf4066a80 msgr2=0x7f8bf4066ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf4066a80 0x7f8bf4066ec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f8bec002f70 tx=0x7f8bec006eb0).stop 2022-07-20T02:29:08.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- 172.21.15.26:0/4166667577 shutdown_connections 2022-07-20T02:29:08.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf4066a80 0x7f8bf4066ec0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/4166667577 >> v2:172.21.15.120:3300/0 conn(0x7f8bf4061050 0x7f8bf4148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/4166667577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf40b2920 0x7f8bf4147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- 172.21.15.26:0/4166667577 >> 172.21.15.26:0/4166667577 conn(0x7f8bf40b60a0 msgr2=0x7f8bf40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- 172.21.15.26:0/4166667577 shutdown_connections 2022-07-20T02:29:08.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.112+0000 7f8bfb52d700 1 -- 172.21.15.26:0/4166667577 wait complete. 2022-07-20T02:29:08.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 Processor -- start 2022-07-20T02:29:08.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 -- start start 2022-07-20T02:29:08.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf4061050 0x7f8bf414e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf4066a80 0x7f8bf4065a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40b4c80 con 0x7f8bf40b2920 2022-07-20T02:29:08.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4064140 con 0x7f8bf4066a80 2022-07-20T02:29:08.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bfb52d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4066460 con 0x7f8bf4061050 2022-07-20T02:29:08.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bf9aca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bf9aca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37852/0 (socket says 172.21.15.26:37852) 2022-07-20T02:29:08.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bf92c9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf4061050 0x7f8bf414e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bf9aca700 1 -- 172.21.15.26:0/2623448316 learned_addr learned my addr 172.21.15.26:0/2623448316 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:08.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.116+0000 7f8bf8ac8700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf4066a80 0x7f8bf4065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 -- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf4061050 msgr2=0x7f8bf414e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf4061050 0x7f8bf414e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 -- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf4066a80 msgr2=0x7f8bf4065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf4066a80 0x7f8bf4065a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 -- 172.21.15.26:0/2623448316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bec00b1e0 con 0x7f8bf40b2920 2022-07-20T02:29:08.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf9aca700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f8bec002f70 tx=0x7f8bec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bec006b40 con 0x7f8bf40b2920 2022-07-20T02:29:08.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec0068b0 con 0x7f8bf40b2920 2022-07-20T02:29:08.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4065170 con 0x7f8bf40b2920 2022-07-20T02:29:08.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bec011d70 con 0x7f8bf40b2920 2022-07-20T02:29:08.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf4064610 con 0x7f8bf40b2920 2022-07-20T02:29:08.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8bec025030 con 0x7f8bf40b2920 2022-07-20T02:29:08.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf40647d0 con 0x7f8bf40b2920 2022-07-20T02:29:08.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 0x7f8bdc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8bec06d080 con 0x7f8bf40b2920 2022-07-20T02:29:08.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf92c9700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 0x7f8bdc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.120+0000 7f8bf92c9700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 0x7f8bdc055e40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f8be400c000 tx=0x7f8be4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.124+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8bec043e40 con 0x7f8bf40b2920 2022-07-20T02:29:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.248+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bf414ada0 con 0x7f8bf40b2920 2022-07-20T02:29:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bea7fc700 1 -- 172.21.15.26:0/2623448316 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16321 (secure 0 0 0) 0x7f8bec023020 con 0x7f8bf40b2920 2022-07-20T02:29:08.259 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:08.259 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:07.622520+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:29:05.266754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T02:29:05.807643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 msgr2=0x7f8bdc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 0x7f8bdc055e40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f8be400c000 tx=0x7f8be4008040).stop 2022-07-20T02:29:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 msgr2=0x7f8bf4065f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f8bec002f70 tx=0x7f8bec015000).stop 2022-07-20T02:29:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 shutdown_connections 2022-07-20T02:29:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bf40b2920 0x7f8bf4065f70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bdc053980 0x7f8bdc055e40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf4066a80 0x7f8bf4065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 --2- 172.21.15.26:0/2623448316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bf4061050 0x7f8bf414e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 >> 172.21.15.26:0/2623448316 conn(0x7f8bf40b60a0 msgr2=0x7f8bf40b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 shutdown_connections 2022-07-20T02:29:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.252+0000 7f8bfb52d700 1 -- 172.21.15.26:0/2623448316 wait complete. 2022-07-20T02:29:08.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:29:08.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 Processor -- start 2022-07-20T02:29:08.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 -- start start 2022-07-20T02:29:08.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 0x7f60f01431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60f00b5b70 0x7f60f01403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f00ba060 con 0x7f60f0062710 2022-07-20T02:29:08.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f012bee0 con 0x7f60f00b9c80 2022-07-20T02:29:08.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f7bb2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f0061f80 con 0x7f60f00b5b70 2022-07-20T02:29:08.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f594e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 0x7f60f01431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f594e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 0x7f60f01431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37856/0 (socket says 172.21.15.26:37856) 2022-07-20T02:29:08.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f594e700 1 -- 172.21.15.26:0/3771286696 learned_addr learned my addr 172.21.15.26:0/3771286696 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:08.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 --2- 172.21.15.26:0/3771286696 >> v2:172.21.15.120:3300/0 conn(0x7f60f00b5b70 0x7f60f01403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 -- 172.21.15.26:0/3771286696 >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 msgr2=0x7f60f01431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 --2- 172.21.15.26:0/3771286696 >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 0x7f60f01431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 -- 172.21.15.26:0/3771286696 >> v1:172.21.15.120:6789/0 conn(0x7f60f0062710 legacy=0x7f60f00b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 -- 172.21.15.26:0/3771286696 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0062220 con 0x7f60f00b5b70 2022-07-20T02:29:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f614f700 1 --2- 172.21.15.26:0/3771286696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f00b5b70 0x7f60f01403e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f60e80032f0 tx=0x7f60e80091d0).ready entity=mon.1 client_cookie=ade29e3bba7cef40 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f494c700 1 -- 172.21.15.26:0/3771286696 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e800b680 con 0x7f60f00b5b70 2022-07-20T02:29:08.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f494c700 1 -- 172.21.15.26:0/3771286696 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e800b7e0 con 0x7f60f00b5b70 2022-07-20T02:29:08.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.384+0000 7f60f494c700 1 -- 172.21.15.26:0/3771286696 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e800b950 con 0x7f60f00b5b70 2022-07-20T02:29:08.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3771286696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f00b5b70 msgr2=0x7f60f01403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3771286696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f00b5b70 0x7f60f01403e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f60e80032f0 tx=0x7f60e80091d0).stop 2022-07-20T02:29:08.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3771286696 shutdown_connections 2022-07-20T02:29:08.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3771286696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f00b5b70 0x7f60f01403e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3771286696 >> v2:172.21.15.26:3300/0 conn(0x7f60f00b9c80 0x7f60f01431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3771286696 >> 172.21.15.26:0/3771286696 conn(0x7f60f0066f70 msgr2=0x7f60f005faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3771286696 shutdown_connections 2022-07-20T02:29:08.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3771286696 wait complete. 2022-07-20T02:29:08.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 Processor -- start 2022-07-20T02:29:08.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- start start 2022-07-20T02:29:08.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f0062710 0x7f60f0147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f00b5b70 0x7f60f0140270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00b2680 con 0x7f60f00b9c80 2022-07-20T02:29:08.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0153450 con 0x7f60f0062710 2022-07-20T02:29:08.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f7bb2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f012c050 con 0x7f60f00b5b70 2022-07-20T02:29:08.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f614f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f594e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f0062710 0x7f60f0147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f614f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37866/0 (socket says 172.21.15.26:37866) 2022-07-20T02:29:08.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f514d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f00b5b70 0x7f60f0140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.388+0000 7f60f614f700 1 -- 172.21.15.26:0/3145782451 learned_addr learned my addr 172.21.15.26:0/3145782451 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:08.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 -- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f00b5b70 msgr2=0x7f60f0140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f00b5b70 0x7f60f0140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 -- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f0062710 msgr2=0x7f60f0147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f0062710 0x7f60f0147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 -- 172.21.15.26:0/3145782451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e8009470 con 0x7f60f00b9c80 2022-07-20T02:29:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f614f700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f60e80032f0 tx=0x7f60e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e8011e20 con 0x7f60f00b9c80 2022-07-20T02:29:08.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f012c2f0 con 0x7f60f00b9c80 2022-07-20T02:29:08.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e8021440 con 0x7f60f00b9c80 2022-07-20T02:29:08.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e80217a0 con 0x7f60f00b9c80 2022-07-20T02:29:08.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f00b2a90 con 0x7f60f00b9c80 2022-07-20T02:29:08.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f00b2c50 con 0x7f60f00b9c80 2022-07-20T02:29:08.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.392+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60e8028070 con 0x7f60f00b9c80 2022-07-20T02:29:08.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.396+0000 7f60e6ffd700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 0x7f60d8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.396+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f60e806d0f0 con 0x7f60f00b9c80 2022-07-20T02:29:08.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.396+0000 7f60f594e700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 0x7f60d8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.396+0000 7f60f594e700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 0x7f60d8055ea0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f60ec008000 tx=0x7f60ec008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.400+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60e8043e10 con 0x7f60f00b9c80 2022-07-20T02:29:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.568+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f60f00b6650 con 0x7f60f00b9c80 2022-07-20T02:29:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.572+0000 7f60e6ffd700 1 -- 172.21.15.26:0/3145782451 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f60e8015020 con 0x7f60f00b9c80 2022-07-20T02:29:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 msgr2=0x7f60d8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 0x7f60d8055ea0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f60ec008000 tx=0x7f60ec008040).stop 2022-07-20T02:29:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 msgr2=0x7f60f0152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f60e80032f0 tx=0x7f60e8016000).stop 2022-07-20T02:29:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 shutdown_connections 2022-07-20T02:29:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60f00b9c80 0x7f60f0152f60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60d80539e0 0x7f60d8055ea0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f00b5b70 0x7f60f0140270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 --2- 172.21.15.26:0/3145782451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60f0062710 0x7f60f0147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 >> 172.21.15.26:0/3145782451 conn(0x7f60f0066f70 msgr2=0x7f60f00b8720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 shutdown_connections 2022-07-20T02:29:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.576+0000 7f60f7bb2700 1 -- 172.21.15.26:0/3145782451 wait complete. 2022-07-20T02:29:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:29:08.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:29:08.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 Processor -- start 2022-07-20T02:29:08.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 -- start start 2022-07-20T02:29:08.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faafc0b3260 0x7faafc0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc0b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc0b3640 con 0x7faafc097c50 2022-07-20T02:29:08.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faafc060e90 con 0x7faafc0b3260 2022-07-20T02:29:08.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafb59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc095300 con 0x7faafc0609e0 2022-07-20T02:29:08.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faaf9d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7faafc097c50 0x7faafc0b1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37152/0 (socket says 172.21.15.26:37152) 2022-07-20T02:29:08.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faaf9d9b700 1 -- 172.21.15.26:0/1128736125 learned_addr learned my addr 172.21.15.26:0/1128736125 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:08.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.716+0000 7faafa59c700 1 --2- 172.21.15.26:0/1128736125 >> v2:172.21.15.26:3300/0 conn(0x7faafc0b3260 0x7faafc0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafad9d700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc0b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faaf959a700 1 -- 172.21.15.26:0/1128736125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042358762 0 0) 0x7faafc0b3640 con 0x7faafc097c50 2022-07-20T02:29:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faaf959a700 1 -- 172.21.15.26:0/1128736125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faadc003260 con 0x7faafc097c50 2022-07-20T02:29:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafa59c700 1 -- 172.21.15.26:0/1128736125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 msgr2=0x7faafc0b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafa59c700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc0b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafa59c700 1 -- 172.21.15.26:0/1128736125 >> v1:172.21.15.26:6789/0 conn(0x7faafc097c50 legacy=0x7faafc0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafa59c700 1 -- 172.21.15.26:0/1128736125 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0955a0 con 0x7faafc0b3260 2022-07-20T02:29:08.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafa59c700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc0b3260 0x7faafc0604f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7faae4002f70 tx=0x7faae4006df0).ready entity=mon.0 client_cookie=d81586b9fa2d8de7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faaf959a700 1 -- 172.21.15.26:0/1128736125 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faae400d4a0 con 0x7faafc0b3260 2022-07-20T02:29:08.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faaf959a700 1 -- 172.21.15.26:0/1128736125 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4000e90 con 0x7faafc0b3260 2022-07-20T02:29:08.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faaf959a700 1 -- 172.21.15.26:0/1128736125 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faae400d7f0 con 0x7faafc0b3260 2022-07-20T02:29:08.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 -- 172.21.15.26:0/1128736125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc0b3260 msgr2=0x7faafc0604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc0b3260 0x7faafc0604f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7faae4002f70 tx=0x7faae4006df0).stop 2022-07-20T02:29:08.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 -- 172.21.15.26:0/1128736125 shutdown_connections 2022-07-20T02:29:08.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc0b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 --2- 172.21.15.26:0/1128736125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc0b3260 0x7faafc0604f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 -- 172.21.15.26:0/1128736125 >> 172.21.15.26:0/1128736125 conn(0x7faafc0b67f0 msgr2=0x7faafc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 -- 172.21.15.26:0/1128736125 shutdown_connections 2022-07-20T02:29:08.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.720+0000 7faafb59e700 1 -- 172.21.15.26:0/1128736125 wait complete. 2022-07-20T02:29:08.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 Processor -- start 2022-07-20T02:29:08.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- start start 2022-07-20T02:29:08.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc09af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc0b3260 0x7faafc090580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc066ff0 con 0x7faafc097c50 2022-07-20T02:29:08.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc066610 con 0x7faafc0b3260 2022-07-20T02:29:08.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc0668b0 con 0x7faafc0609e0 2022-07-20T02:29:08.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37878/0 (socket says 172.21.15.26:37878) 2022-07-20T02:29:08.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 -- 172.21.15.26:0/619896451 learned_addr learned my addr 172.21.15.26:0/619896451 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:08.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafad9d700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc0b3260 0x7faafc090580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafa59c700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc09af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 -- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 msgr2=0x7faafc09af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc09af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 -- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc0b3260 msgr2=0x7faafc090580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc0b3260 0x7faafc090580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 -- 172.21.15.26:0/619896451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae400b180 con 0x7faafc097c50 2022-07-20T02:29:08.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafa59c700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc09af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:08.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf9d9b700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7faaec001110 tx=0x7faaec007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaec00ce20 con 0x7faafc097c50 2022-07-20T02:29:08.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec0072e0 con 0x7faafc097c50 2022-07-20T02:29:08.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc066b50 con 0x7faafc097c50 2022-07-20T02:29:08.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaec005800 con 0x7faafc097c50 2022-07-20T02:29:08.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc065400 con 0x7faafc097c50 2022-07-20T02:29:08.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.724+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc066ce0 con 0x7faafc097c50 2022-07-20T02:29:08.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.728+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faaec01a070 con 0x7faafc097c50 2022-07-20T02:29:08.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.728+0000 7faaf37fe700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 0x7faae0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:08.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.728+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faaec06b3c0 con 0x7faafc097c50 2022-07-20T02:29:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.728+0000 7faafa59c700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 0x7faae0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.728+0000 7faafa59c700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 0x7faae0055fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7faae4018000 tx=0x7faae4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.736+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faaec042180 con 0x7faafc097c50 2022-07-20T02:29:08.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.904+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faafc095840 con 0x7faafc097c50 2022-07-20T02:29:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.912+0000 7faaf37fe700 1 -- 172.21.15.26:0/619896451 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7faaec0457a0 con 0x7faafc097c50 2022-07-20T02:29:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.912+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 msgr2=0x7faae0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.912+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 0x7faae0055fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7faae4018000 tx=0x7faae4014000).stop 2022-07-20T02:29:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.912+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 msgr2=0x7faafc090090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.912+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7faaec001110 tx=0x7faaec007b20).stop 2022-07-20T02:29:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 shutdown_connections 2022-07-20T02:29:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faafc097c50 0x7faafc090090 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faae0053b00 0x7faae0055fc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc0b3260 0x7faafc090580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 --2- 172.21.15.26:0/619896451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faafc0609e0 0x7faafc09af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 >> 172.21.15.26:0/619896451 conn(0x7faafc0b67f0 msgr2=0x7faafc0998a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 shutdown_connections 2022-07-20T02:29:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:08.916+0000 7faafb59e700 1 -- 172.21.15.26:0/619896451 wait complete. 2022-07-20T02:29:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:29:08.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 Processor -- start 2022-07-20T02:29:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 -- start start 2022-07-20T02:29:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21d40b7920 0x7f21d4067480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d4144670 con 0x7f21d40b49c0 2022-07-20T02:29:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d40b4f40 con 0x7f21d4141000 2022-07-20T02:29:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21dc455700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d40b44a0 con 0x7f21d40b7920 2022-07-20T02:29:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21da1f1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21d40b7920 0x7f21d4067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21da1f1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f21d40b7920 0x7f21d4067480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37884/0 (socket says 172.21.15.26:37884) 2022-07-20T02:29:09.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21d99f0700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f21d40b49c0 0x7f21d4144d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37164/0 (socket says 172.21.15.26:37164) 2022-07-20T02:29:09.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.032+0000 7f21da1f1700 1 -- 172.21.15.26:0/337254153 learned_addr learned my addr 172.21.15.26:0/337254153 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:09.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21d91ef700 1 -- 172.21.15.26:0/337254153 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765364967 0 0) 0x7f21d40b4f40 con 0x7f21d4141000 2022-07-20T02:29:09.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21da1f1700 1 -- 172.21.15.26:0/337254153 >> v1:172.21.15.120:6789/0 conn(0x7f21d4141000 legacy=0x7f21d4141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21da1f1700 1 -- 172.21.15.26:0/337254153 >> v1:172.21.15.26:6789/0 conn(0x7f21d40b49c0 legacy=0x7f21d4144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21da1f1700 1 -- 172.21.15.26:0/337254153 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d41448a0 con 0x7f21d40b7920 2022-07-20T02:29:09.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21da1f1700 1 --2- 172.21.15.26:0/337254153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d4067480 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f21d0001110 tx=0x7f21c4006eb0).ready entity=mon.0 client_cookie=fd7381ea5ee26366 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21d91ef700 1 -- 172.21.15.26:0/337254153 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c400d610 con 0x7f21d40b7920 2022-07-20T02:29:09.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21d91ef700 1 -- 172.21.15.26:0/337254153 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4000e90 con 0x7f21d40b7920 2022-07-20T02:29:09.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21d91ef700 1 -- 172.21.15.26:0/337254153 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c400d960 con 0x7f21d40b7920 2022-07-20T02:29:09.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- 172.21.15.26:0/337254153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 msgr2=0x7f21d4067480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 --2- 172.21.15.26:0/337254153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d4067480 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f21d0001110 tx=0x7f21c4006eb0).stop 2022-07-20T02:29:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- 172.21.15.26:0/337254153 shutdown_connections 2022-07-20T02:29:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 --2- 172.21.15.26:0/337254153 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d4067480 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- 172.21.15.26:0/337254153 >> 172.21.15.26:0/337254153 conn(0x7f21d4064b10 msgr2=0x7f21d4064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- 172.21.15.26:0/337254153 shutdown_connections 2022-07-20T02:29:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- 172.21.15.26:0/337254153 wait complete. 2022-07-20T02:29:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 Processor -- start 2022-07-20T02:29:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.036+0000 7f21dc455700 1 -- start start 2022-07-20T02:29:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d40b49c0 0x7f21d40b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d4141000 0x7f21d41436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d4150de0 con 0x7f21d40b7920 2022-07-20T02:29:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40aecf0 con 0x7f21d4141000 2022-07-20T02:29:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40aea50 con 0x7f21d40b49c0 2022-07-20T02:29:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37892/0 (socket says 172.21.15.26:37892) 2022-07-20T02:29:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 -- 172.21.15.26:0/2922088571 learned_addr learned my addr 172.21.15.26:0/2922088571 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21da1f1700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d40b49c0 0x7f21d40b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21da9f2700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d4141000 0x7f21d41436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 -- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d40b49c0 msgr2=0x7f21d40b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d40b49c0 0x7f21d40b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 -- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d4141000 msgr2=0x7f21d41436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d4141000 0x7f21d41436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 -- 172.21.15.26:0/2922088571 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c400b1e0 con 0x7f21d40b7920 2022-07-20T02:29:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21d99f0700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f21d00011d0 tx=0x7f21d0007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d000ce40 con 0x7f21d40b7920 2022-07-20T02:29:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d00072e0 con 0x7f21d40b7920 2022-07-20T02:29:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d4151040 con 0x7f21d40b7920 2022-07-20T02:29:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d0005740 con 0x7f21d40b7920 2022-07-20T02:29:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.040+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d4143ba0 con 0x7f21d40b7920 2022-07-20T02:29:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d41511d0 con 0x7f21d40b7920 2022-07-20T02:29:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f21d0010070 con 0x7f21d40b7920 2022-07-20T02:29:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21c37fe700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 0x7f21b4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f21d006b7b0 con 0x7f21d40b7920 2022-07-20T02:29:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21da1f1700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 0x7f21b4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.044+0000 7f21da1f1700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 0x7f21b4055f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f21c401c000 tx=0x7f21c4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.048+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21d0042460 con 0x7f21d40b7920 2022-07-20T02:29:09.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21d4145450 con 0x7f21d40b7920 2022-07-20T02:29:09.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21c37fe700 1 -- 172.21.15.26:0/2922088571 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f21d0045a80 con 0x7f21d40b7920 2022-07-20T02:29:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:09.215 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:09.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 msgr2=0x7f21b4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 0x7f21b4055f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f21c401c000 tx=0x7f21c4015000).stop 2022-07-20T02:29:09.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 msgr2=0x7f21d40b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.212+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f21d00011d0 tx=0x7f21d0007c00).stop 2022-07-20T02:29:09.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 shutdown_connections 2022-07-20T02:29:09.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d40b7920 0x7f21d40b1470 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b4053aa0 0x7f21b4055f60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d4141000 0x7f21d41436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 --2- 172.21.15.26:0/2922088571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d40b49c0 0x7f21d40b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 >> 172.21.15.26:0/2922088571 conn(0x7f21d4064b10 msgr2=0x7f21d405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:09.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 shutdown_connections 2022-07-20T02:29:09.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.216+0000 7f21dc455700 1 -- 172.21.15.26:0/2922088571 wait complete. 2022-07-20T02:29:09.234 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:09.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:29:06.619736+0000', 'modified': '2022-07-20T02:29:07.623517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5616}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5616': {'gid': 5616, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3215220955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3215220955}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3215220955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:29:09.540 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:29:09.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:29:09.659 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5616,"want_state":"up:active","state":"up:active","rank_uptime":3.0277103030000001,"mdsmap_epoch":24,"osdmap_epoch":55,"osdmap_epoch_barrier":55,"uptime":7.3231190589999997} 2022-07-20T02:29:09.660 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5616, "mdsmap_epoch": 24, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "rank_uptime": 3.027710303, "state": "up:active", "uptime": 7.323119059, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:29:09.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:09.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 Processor -- start 2022-07-20T02:29:09.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 -- start start 2022-07-20T02:29:09.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 0x7f4ac0066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00b1bc0 con 0x7f4ac0061050 2022-07-20T02:29:09.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac0147700 con 0x7f4ac00b2920 2022-07-20T02:29:09.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.764+0000 7f4ac55ed700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac014b040 con 0x7f4ac01484a0 2022-07-20T02:29:09.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abf7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4ac0061050 0x7f4ac0066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37182/0 (socket says 172.21.15.26:37182) 2022-07-20T02:29:09.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abf7fe700 1 -- 172.21.15.26:0/999243579 learned_addr learned my addr 172.21.15.26:0/999243579 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:09.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abe7fc700 1 --2- 172.21.15.26:0/999243579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 0x7f4ac0066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653202105 0 0) 0x7f4ac00b1bc0 con 0x7f4ac0061050 2022-07-20T02:29:09.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4003260 con 0x7f4ac0061050 2022-07-20T02:29:09.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472584590 0 0) 0x7f4ac0147700 con 0x7f4ac00b2920 2022-07-20T02:29:09.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac00b1bc0 con 0x7f4ac00b2920 2022-07-20T02:29:09.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abe7fc700 1 -- 172.21.15.26:0/999243579 >> v1:172.21.15.120:6789/0 conn(0x7f4ac00b2920 legacy=0x7f4ac0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612240904 0 0) 0x7f4aa4003260 con 0x7f4ac0061050 2022-07-20T02:29:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abe7fc700 1 -- 172.21.15.26:0/999243579 >> v1:172.21.15.26:6789/0 conn(0x7f4ac0061050 legacy=0x7f4ac0066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abe7fc700 1 -- 172.21.15.26:0/999243579 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0147930 con 0x7f4ac01484a0 2022-07-20T02:29:09.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abe7fc700 1 --2- 172.21.15.26:0/999243579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 0x7f4ac0066a80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4ab8001110 tx=0x7f4ab00091d0).ready entity=mon.2 client_cookie=5240151b115445c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab8001660 con 0x7f4ac01484a0 2022-07-20T02:29:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab000f510 con 0x7f4ac01484a0 2022-07-20T02:29:09.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4abdffb700 1 -- 172.21.15.26:0/999243579 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab000f680 con 0x7f4ac01484a0 2022-07-20T02:29:09.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 -- 172.21.15.26:0/999243579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 msgr2=0x7f4ac0066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/999243579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 0x7f4ac0066a80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4ab8001110 tx=0x7f4ab00091d0).stop 2022-07-20T02:29:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 -- 172.21.15.26:0/999243579 shutdown_connections 2022-07-20T02:29:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/999243579 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac01484a0 0x7f4ac0066a80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 -- 172.21.15.26:0/999243579 >> 172.21.15.26:0/999243579 conn(0x7f4ac00b60a0 msgr2=0x7f4ac00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 -- 172.21.15.26:0/999243579 shutdown_connections 2022-07-20T02:29:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.768+0000 7f4ac55ed700 1 -- 172.21.15.26:0/999243579 wait complete. 2022-07-20T02:29:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 Processor -- start 2022-07-20T02:29:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- start start 2022-07-20T02:29:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac0061050 0x7f4ac00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac00b2920 0x7f4ac00abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0156c00 con 0x7f4ac01484a0 2022-07-20T02:29:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac00ac170 con 0x7f4ac0061050 2022-07-20T02:29:09.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac00ac410 con 0x7f4ac00b2920 2022-07-20T02:29:09.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abe7fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac00b2920 0x7f4ac00abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37908/0 (socket says 172.21.15.26:37908) 2022-07-20T02:29:09.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 -- 172.21.15.26:0/2040973820 learned_addr learned my addr 172.21.15.26:0/2040973820 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:09.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abeffd700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac0061050 0x7f4ac00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 -- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac00b2920 msgr2=0x7f4ac00abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac00b2920 0x7f4ac00abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 -- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac0061050 msgr2=0x7f4ac00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac0061050 0x7f4ac00b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 -- 172.21.15.26:0/2040973820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab0009470 con 0x7f4ac01484a0 2022-07-20T02:29:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4abf7fe700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4ab8001840 tx=0x7f4ab8001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab8007170 con 0x7f4ac01484a0 2022-07-20T02:29:09.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab80072d0 con 0x7f4ac01484a0 2022-07-20T02:29:09.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab8004920 con 0x7f4ac01484a0 2022-07-20T02:29:09.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac0156df0 con 0x7f4ac01484a0 2022-07-20T02:29:09.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.772+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac0065f30 con 0x7f4ac01484a0 2022-07-20T02:29:09.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac00ac5d0 con 0x7f4ac01484a0 2022-07-20T02:29:09.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ab8004a80 con 0x7f4ac01484a0 2022-07-20T02:29:09.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4aaffff700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4ab806bc50 con 0x7f4ac01484a0 2022-07-20T02:29:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4abeffd700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:09.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.776+0000 7f4abeffd700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f4ab400d000 tx=0x7f4ab4007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:09.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.784+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ab80428a0 con 0x7f4ac01484a0 2022-07-20T02:29:09.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.908+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ac0156f80 con 0x7f4ac01484a0 2022-07-20T02:29:09.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.908+0000 7f4aaffff700 1 -- 172.21.15.26:0/2040973820 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16321 (secure 0 0 0) 0x7f4ab8045ec0 con 0x7f4ac01484a0 2022-07-20T02:29:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:09.912 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:07.622520+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:29:05.266754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T02:29:05.807643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:09.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 msgr2=0x7f4a9c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f4ab400d000 tx=0x7f4ab4007ad0).stop 2022-07-20T02:29:09.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 msgr2=0x7f4ac0157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:09.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4ab8001840 tx=0x7f4ab8001400).stop 2022-07-20T02:29:09.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 shutdown_connections 2022-07-20T02:29:09.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ac01484a0 0x7f4ac0157290 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a9c053a70 0x7f4a9c055f30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ac00b2920 0x7f4ac00abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 --2- 172.21.15.26:0/2040973820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac0061050 0x7f4ac00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:09.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 >> 172.21.15.26:0/2040973820 conn(0x7f4ac00b60a0 msgr2=0x7f4ac014b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 shutdown_connections 2022-07-20T02:29:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:09.912+0000 7f4ac55ed700 1 -- 172.21.15.26:0/2040973820 wait complete. 2022-07-20T02:29:09.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:29:05.266754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:29:05.807643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:29:09.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:29:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 Processor -- start 2022-07-20T02:29:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 -- start start 2022-07-20T02:29:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcc5005f840 0x7fcc50146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500b1ae0 con 0x7fcc501465b0 2022-07-20T02:29:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500b1d80 con 0x7fcc500b24c0 2022-07-20T02:29:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc56615700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc500b2e20 con 0x7fcc5005f840 2022-07-20T02:29:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc55e14700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcc5005f840 0x7fcc50146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc55613700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcc501465b0 0x7fcc50066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37192/0 (socket says 172.21.15.26:37192) 2022-07-20T02:29:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.044+0000 7fcc55613700 1 -- 172.21.15.26:0/997546443 learned_addr learned my addr 172.21.15.26:0/997546443 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 -- 172.21.15.26:0/997546443 >> v1:172.21.15.120:6789/0 conn(0x7fcc500b24c0 legacy=0x7fcc500604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 -- 172.21.15.26:0/997546443 >> v1:172.21.15.26:6789/0 conn(0x7fcc501465b0 legacy=0x7fcc50066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc3ffff700 1 -- 172.21.15.26:0/997546443 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943014321 0 0) 0x7fcc500b1ae0 con 0x7fcc501465b0 2022-07-20T02:29:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 -- 172.21.15.26:0/997546443 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500b2020 con 0x7fcc5005f840 2022-07-20T02:29:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 --2- 172.21.15.26:0/997546443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc5005f840 0x7fcc50146990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fcc480032f0 tx=0x7fcc480091d0).ready entity=mon.0 client_cookie=3d1190d8a433abab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc3ffff700 1 -- 172.21.15.26:0/997546443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4800b8a0 con 0x7fcc5005f840 2022-07-20T02:29:10.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc3ffff700 1 -- 172.21.15.26:0/997546443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800ba00 con 0x7fcc5005f840 2022-07-20T02:29:10.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc3ffff700 1 -- 172.21.15.26:0/997546443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4800bb70 con 0x7fcc5005f840 2022-07-20T02:29:10.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- 172.21.15.26:0/997546443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc5005f840 msgr2=0x7fcc50146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 --2- 172.21.15.26:0/997546443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc5005f840 0x7fcc50146990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fcc480032f0 tx=0x7fcc480091d0).stop 2022-07-20T02:29:10.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- 172.21.15.26:0/997546443 shutdown_connections 2022-07-20T02:29:10.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 --2- 172.21.15.26:0/997546443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc5005f840 0x7fcc50146990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- 172.21.15.26:0/997546443 >> 172.21.15.26:0/997546443 conn(0x7fcc500b3050 msgr2=0x7fcc500b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- 172.21.15.26:0/997546443 shutdown_connections 2022-07-20T02:29:10.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- 172.21.15.26:0/997546443 wait complete. 2022-07-20T02:29:10.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 Processor -- start 2022-07-20T02:29:10.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- start start 2022-07-20T02:29:10.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 0x7fcc50064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc501465b0 0x7fcc50155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500638a0 con 0x7fcc50155d40 2022-07-20T02:29:10.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc501566c0 con 0x7fcc5005f840 2022-07-20T02:29:10.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc56615700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc50063ad0 con 0x7fcc501465b0 2022-07-20T02:29:10.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37922/0 (socket says 172.21.15.26:37922) 2022-07-20T02:29:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55e14700 1 -- 172.21.15.26:0/2578501616 learned_addr learned my addr 172.21.15.26:0/2578501616 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:10.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc55613700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 0x7fcc50064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.048+0000 7fcc54e12700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc501465b0 0x7fcc50155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 -- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc501465b0 msgr2=0x7fcc50155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc501465b0 0x7fcc50155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 -- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 msgr2=0x7fcc50064210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 0x7fcc50064210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 -- 172.21.15.26:0/2578501616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48009470 con 0x7fcc50155d40 2022-07-20T02:29:10.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55613700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 0x7fcc50064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:10.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc55e14700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7fcc480032f0 tx=0x7fcc48016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4801f3f0 con 0x7fcc50155d40 2022-07-20T02:29:10.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4801f550 con 0x7fcc50155d40 2022-07-20T02:29:10.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc50063d70 con 0x7fcc50155d40 2022-07-20T02:29:10.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4801f8b0 con 0x7fcc50155d40 2022-07-20T02:29:10.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc50152300 con 0x7fcc50155d40 2022-07-20T02:29:10.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc501474a0 con 0x7fcc50155d40 2022-07-20T02:29:10.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcc48026030 con 0x7fcc50155d40 2022-07-20T02:29:10.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 0x7fcc30055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.052+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcc4806cf60 con 0x7fcc50155d40 2022-07-20T02:29:10.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.056+0000 7fcc55613700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 0x7fcc30055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.056+0000 7fcc55613700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 0x7fcc30055e10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcc4c00b000 tx=0x7fcc4c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.060+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcc48043c80 con 0x7fcc50155d40 2022-07-20T02:29:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.184+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fcc50063f00 con 0x7fcc50155d40 2022-07-20T02:29:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc3e7fc700 1 -- 172.21.15.26:0/2578501616 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v55) v1 ==== 110+0+10 (secure 0 0 0) 0x7fcc480240b0 con 0x7fcc50155d40 2022-07-20T02:29:10.196 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:29:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 msgr2=0x7fcc30055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 0x7fcc30055e10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcc4c00b000 tx=0x7fcc4c009040).stop 2022-07-20T02:29:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 msgr2=0x7fcc501561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7fcc480032f0 tx=0x7fcc48016000).stop 2022-07-20T02:29:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 shutdown_connections 2022-07-20T02:29:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc50155d40 0x7fcc501561d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc30053950 0x7fcc30055e10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc501465b0 0x7fcc50155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 --2- 172.21.15.26:0/2578501616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5005f840 0x7fcc50064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 >> 172.21.15.26:0/2578501616 conn(0x7fcc500b3050 msgr2=0x7fcc500b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 shutdown_connections 2022-07-20T02:29:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.188+0000 7fcc56615700 1 -- 172.21.15.26:0/2578501616 wait complete. 2022-07-20T02:29:10.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:29:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 Processor -- start 2022-07-20T02:29:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 -- start start 2022-07-20T02:29:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c05f840 con 0x7f274c0a95a0 2022-07-20T02:29:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c0b2e80 con 0x7f274c0b2aa0 2022-07-20T02:29:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2752210700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0b0a50 con 0x7f274c095800 2022-07-20T02:29:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f275120e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2751a0f700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f274c0a95a0 0x7f274c0b15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51422/0 (socket says 172.21.15.26:51422) 2022-07-20T02:29:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:59444/0 (socket says 172.21.15.26:59444) 2022-07-20T02:29:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f275120e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:57618/0 (socket says 172.21.15.26:57618) 2022-07-20T02:29:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2751a0f700 1 -- 172.21.15.26:0/2443425059 learned_addr learned my addr 172.21.15.26:0/2443425059 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f273bfff700 1 -- 172.21.15.26:0/2443425059 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824693861 0 0) 0x7f274c05f840 con 0x7f274c0a95a0 2022-07-20T02:29:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f273bfff700 1 -- 172.21.15.26:0/2443425059 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2734003260 con 0x7f274c0a95a0 2022-07-20T02:29:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 -- 172.21.15.26:0/2443425059 >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 msgr2=0x7f274c1438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 --2- 172.21.15.26:0/2443425059 >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 -- 172.21.15.26:0/2443425059 >> v1:172.21.15.120:6789/0 conn(0x7f274c0a95a0 legacy=0x7f274c0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 -- 172.21.15.26:0/2443425059 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c05fa70 con 0x7f274c095800 2022-07-20T02:29:10.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f275120e700 1 --2- 172.21.15.26:0/2443425059 >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:10.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f2750a0d700 1 --2- 172.21.15.26:0/2443425059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f273c002f70 tx=0x7f273c006df0).ready entity=mon.2 client_cookie=eb6d407775958cab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f273bfff700 1 -- 172.21.15.26:0/2443425059 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f273c00d4a0 con 0x7f274c095800 2022-07-20T02:29:10.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f273bfff700 1 -- 172.21.15.26:0/2443425059 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c000e90 con 0x7f274c095800 2022-07-20T02:29:10.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.320+0000 7f273bfff700 1 -- 172.21.15.26:0/2443425059 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f273c00d7f0 con 0x7f274c095800 2022-07-20T02:29:10.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- 172.21.15.26:0/2443425059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 msgr2=0x7f274c143db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 --2- 172.21.15.26:0/2443425059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f273c002f70 tx=0x7f273c006df0).stop 2022-07-20T02:29:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- 172.21.15.26:0/2443425059 shutdown_connections 2022-07-20T02:29:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 --2- 172.21.15.26:0/2443425059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c095800 0x7f274c143db0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 --2- 172.21.15.26:0/2443425059 >> v2:172.21.15.120:3300/0 conn(0x7f274c0b2aa0 0x7f274c1438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- 172.21.15.26:0/2443425059 >> 172.21.15.26:0/2443425059 conn(0x7f274c063030 msgr2=0x7f274c063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- 172.21.15.26:0/2443425059 shutdown_connections 2022-07-20T02:29:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- 172.21.15.26:0/2443425059 wait complete. 2022-07-20T02:29:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 Processor -- start 2022-07-20T02:29:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.324+0000 7f2752210700 1 -- start start 2022-07-20T02:29:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f274c0a95a0 0x7f274c140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c0b2aa0 0x7f274c140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0ac290 con 0x7f274c095800 2022-07-20T02:29:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c153220 con 0x7f274c0a95a0 2022-07-20T02:29:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0ac4c0 con 0x7f274c0b2aa0 2022-07-20T02:29:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37932/0 (socket says 172.21.15.26:37932) 2022-07-20T02:29:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 -- 172.21.15.26:0/1355133569 learned_addr learned my addr 172.21.15.26:0/1355133569 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2750a0d700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f274c0a95a0 0x7f274c140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2751a0f700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c0b2aa0 0x7f274c140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 -- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c0b2aa0 msgr2=0x7f274c140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c0b2aa0 0x7f274c140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 -- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f274c0a95a0 msgr2=0x7f274c140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f274c0a95a0 0x7f274c140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 -- 172.21.15.26:0/1355133569 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274800b180 con 0x7f274c095800 2022-07-20T02:29:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f275120e700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f274800c000 tx=0x7f2748006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2748010780 con 0x7f274c095800 2022-07-20T02:29:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27480108e0 con 0x7f274c095800 2022-07-20T02:29:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c00b180 con 0x7f274c095800 2022-07-20T02:29:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2748010c40 con 0x7f274c095800 2022-07-20T02:29:10.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.328+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c141270 con 0x7f274c095800 2022-07-20T02:29:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c05fd10 con 0x7f274c095800 2022-07-20T02:29:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f274801c460 con 0x7f274c095800 2022-07-20T02:29:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f273a7fc700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 0x7f272c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f274806c4d0 con 0x7f274c095800 2022-07-20T02:29:10.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f2750a0d700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 0x7f272c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.332+0000 7f2750a0d700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 0x7f272c055fc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f273c01a000 tx=0x7f273c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.340+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2748043230 con 0x7f274c095800 2022-07-20T02:29:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.468+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f274c069ea0 con 0x7f274c095800 2022-07-20T02:29:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.468+0000 7f273a7fc700 1 -- 172.21.15.26:0/1355133569 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2748046850 con 0x7f274c095800 2022-07-20T02:29:10.472 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 msgr2=0x7f272c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 0x7f272c055fc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f273c01a000 tx=0x7f273c014000).stop 2022-07-20T02:29:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 msgr2=0x7f274c0947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f274800c000 tx=0x7f2748006df0).stop 2022-07-20T02:29:10.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 shutdown_connections 2022-07-20T02:29:10.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f272c053b00 0x7f272c055fc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f274c0b2aa0 0x7f274c140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f274c0a95a0 0x7f274c140890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 --2- 172.21.15.26:0/1355133569 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f274c095800 0x7f274c0947e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 >> 172.21.15.26:0/1355133569 conn(0x7f274c063030 msgr2=0x7f274c141750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 shutdown_connections 2022-07-20T02:29:10.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.472+0000 7f2752210700 1 -- 172.21.15.26:0/1355133569 wait complete. 2022-07-20T02:29:10.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:29:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.596+0000 7f366ffff700 1 Processor -- start 2022-07-20T02:29:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.596+0000 7f366ffff700 1 -- start start 2022-07-20T02:29:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.596+0000 7f366ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36700b3260 0x7f3670067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.596+0000 7f366ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f36700980d0 0x7f3670142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367005ff70 con 0x7f3670061860 2022-07-20T02:29:10.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36700b3640 con 0x7f36700980d0 2022-07-20T02:29:10.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36700601a0 con 0x7f36700b3260 2022-07-20T02:29:10.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f36700980d0 0x7f3670142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f36700980d0 0x7f3670142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37944/0 (socket says 172.21.15.26:37944) 2022-07-20T02:29:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 -- 172.21.15.26:0/3660888623 learned_addr learned my addr 172.21.15.26:0/3660888623 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366effd700 1 --2- 172.21.15.26:0/3660888623 >> v2:172.21.15.120:3300/0 conn(0x7f36700b3260 0x7f3670067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 -- 172.21.15.26:0/3660888623 >> v2:172.21.15.120:3300/0 conn(0x7f36700b3260 msgr2=0x7f3670067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 --2- 172.21.15.26:0/3660888623 >> v2:172.21.15.120:3300/0 conn(0x7f36700b3260 0x7f3670067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 -- 172.21.15.26:0/3660888623 >> v1:172.21.15.120:6789/0 conn(0x7f3670061860 legacy=0x7f36700979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 -- 172.21.15.26:0/3660888623 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700606e0 con 0x7f36700980d0 2022-07-20T02:29:10.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366f7fe700 1 --2- 172.21.15.26:0/3660888623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670142d30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f36680032f0 tx=0x7f3668009290).ready entity=mon.0 client_cookie=e09bb3ec9a17c064 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366dffb700 1 -- 172.21.15.26:0/3660888623 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f366800b870 con 0x7f36700980d0 2022-07-20T02:29:10.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366dffb700 1 -- 172.21.15.26:0/3660888623 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800b9d0 con 0x7f36700980d0 2022-07-20T02:29:10.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366dffb700 1 -- 172.21.15.26:0/3660888623 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f366800bb40 con 0x7f36700980d0 2022-07-20T02:29:10.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- 172.21.15.26:0/3660888623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 msgr2=0x7f3670142d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 --2- 172.21.15.26:0/3660888623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670142d30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f36680032f0 tx=0x7f3668009290).stop 2022-07-20T02:29:10.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- 172.21.15.26:0/3660888623 shutdown_connections 2022-07-20T02:29:10.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 --2- 172.21.15.26:0/3660888623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670142d30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 --2- 172.21.15.26:0/3660888623 >> v2:172.21.15.120:3300/0 conn(0x7f36700b3260 0x7f3670067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- 172.21.15.26:0/3660888623 >> 172.21.15.26:0/3660888623 conn(0x7f36700b67f0 msgr2=0x7f36700b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- 172.21.15.26:0/3660888623 shutdown_connections 2022-07-20T02:29:10.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- 172.21.15.26:0/3660888623 wait complete. 2022-07-20T02:29:10.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 Processor -- start 2022-07-20T02:29:10.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.600+0000 7f366ffff700 1 -- start start 2022-07-20T02:29:10.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670061860 0x7f3670094970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36700b3260 0x7f3670063810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36700644e0 con 0x7f36700980d0 2022-07-20T02:29:10.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367009bc30 con 0x7f3670061860 2022-07-20T02:29:10.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670064240 con 0x7f36700b3260 2022-07-20T02:29:10.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37948/0 (socket says 172.21.15.26:37948) 2022-07-20T02:29:10.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 -- 172.21.15.26:0/3177471017 learned_addr learned my addr 172.21.15.26:0/3177471017 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:10.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366effd700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670061860 0x7f3670094970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366f7fe700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36700b3260 0x7f3670063810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 -- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36700b3260 msgr2=0x7f3670063810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36700b3260 0x7f3670063810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 -- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670061860 msgr2=0x7f3670094970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670061860 0x7f3670094970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 -- 172.21.15.26:0/3177471017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668009530 con 0x7f36700980d0 2022-07-20T02:29:10.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366e7fc700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f3670097600 tx=0x7f3658007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365800ce30 con 0x7f36700980d0 2022-07-20T02:29:10.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658007240 con 0x7f36700980d0 2022-07-20T02:29:10.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367009b4f0 con 0x7f36700980d0 2022-07-20T02:29:10.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3658006700 con 0x7f36700980d0 2022-07-20T02:29:10.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3670099f80 con 0x7f36700980d0 2022-07-20T02:29:10.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.604+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36700574e0 con 0x7f36700980d0 2022-07-20T02:29:10.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.608+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3658010030 con 0x7f36700980d0 2022-07-20T02:29:10.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.608+0000 7f3667fff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 0x7f3654055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.608+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f365806bcf0 con 0x7f36700980d0 2022-07-20T02:29:10.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.608+0000 7f366effd700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 0x7f3654055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.608+0000 7f366effd700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 0x7f3654055ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f366000c000 tx=0x7f3660008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.616+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3658042ab0 con 0x7f36700980d0 2022-07-20T02:29:10.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.744+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f367006afa0 con 0x7f36700980d0 2022-07-20T02:29:10.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.744+0000 7f3667fff700 1 -- 172.21.15.26:0/3177471017 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7f36580460d0 con 0x7f36700980d0 2022-07-20T02:29:10.749 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:10.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 msgr2=0x7f3654055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 0x7f3654055ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f366000c000 tx=0x7f3660008040).stop 2022-07-20T02:29:10.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 msgr2=0x7f3670063320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f3670097600 tx=0x7f3658007c10).stop 2022-07-20T02:29:10.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 shutdown_connections 2022-07-20T02:29:10.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36700980d0 0x7f3670063320 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3654053b30 0x7f3654055ff0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36700b3260 0x7f3670063810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 --2- 172.21.15.26:0/3177471017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3670061860 0x7f3670094970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 >> 172.21.15.26:0/3177471017 conn(0x7f36700b67f0 msgr2=0x7f3670143410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 shutdown_connections 2022-07-20T02:29:10.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.748+0000 7f366ffff700 1 -- 172.21.15.26:0/3177471017 wait complete. 2022-07-20T02:29:10.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:29:10.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.868+0000 7f1956d69700 1 Processor -- start 2022-07-20T02:29:10.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.868+0000 7f1956d69700 1 -- start start 2022-07-20T02:29:10.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956d69700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f19500b1200 0x7f19500ac300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956d69700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195005d4d0 con 0x7f195005c720 2022-07-20T02:29:10.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956d69700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500a9ff0 con 0x7f19500afe40 2022-07-20T02:29:10.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956d69700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1950067e60 con 0x7f19500b1200 2022-07-20T02:29:10.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f195005c720 0x7f1950147550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37234/0 (socket says 172.21.15.26:37234) 2022-07-20T02:29:10.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 -- 172.21.15.26:0/2483103391 learned_addr learned my addr 172.21.15.26:0/2483103391 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:10.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956568700 1 --2- 172.21.15.26:0/2483103391 >> v2:172.21.15.120:3300/0 conn(0x7f19500b1200 0x7f19500ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420796475 0 0) 0x7f195005d4d0 con 0x7f195005c720 2022-07-20T02:29:10.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1938003260 con 0x7f195005c720 2022-07-20T02:29:10.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14133201 0 0) 0x7f1938003260 con 0x7f195005c720 2022-07-20T02:29:10.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 >> v2:172.21.15.120:3300/0 conn(0x7f19500b1200 msgr2=0x7f19500ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:10.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 --2- 172.21.15.26:0/2483103391 >> v2:172.21.15.120:3300/0 conn(0x7f19500b1200 0x7f19500ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 >> v1:172.21.15.120:6789/0 conn(0x7f19500afe40 legacy=0x7f19500abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950067bc0 con 0x7f195005c720 2022-07-20T02:29:10.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f194c001660 con 0x7f195005c720 2022-07-20T02:29:10.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956568700 1 --2- 172.21.15.26:0/2483103391 >> v2:172.21.15.120:3300/0 conn(0x7f19500b1200 0x7f19500ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:10.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 >> v1:172.21.15.26:6789/0 conn(0x7f195005c720 legacy=0x7f1950147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 0x7f1938003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1938004500 0x7f1938004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938004e70 0x7f193800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1950067e60 con 0x7f1938003750 2022-07-20T02:29:10.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193800bd20 con 0x7f1938004500 2022-07-20T02:29:10.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193800bee0 con 0x7f1938004e70 2022-07-20T02:29:10.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 0x7f1938003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956568700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938004e70 0x7f193800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955566700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1938004500 0x7f1938004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 -- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938004e70 msgr2=0x7f193800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:10.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938004e70 0x7f193800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 -- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1938004500 msgr2=0x7f1938004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:10.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1938004500 0x7f1938004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 -- 172.21.15.26:0/2483103391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938003260 con 0x7f1938003750 2022-07-20T02:29:10.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956568700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938004e70 0x7f193800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:10.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955566700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1938004500 0x7f1938004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:10.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1956568700 1 -- 172.21.15.26:0/2483103391 reap_dead start 2022-07-20T02:29:10.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1955d67700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 0x7f1938003b70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f194c001370 tx=0x7f1950067d50).ready entity=mon.0 client_cookie=e0619d6b4fc8fa39 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c0070d0 con 0x7f1938003750 2022-07-20T02:29:10.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c007230 con 0x7f1938003750 2022-07-20T02:29:10.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.872+0000 7f1954d65700 1 -- 172.21.15.26:0/2483103391 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c00c8f0 con 0x7f1938003750 2022-07-20T02:29:10.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 msgr2=0x7f1938003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 0x7f1938003b70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f194c001370 tx=0x7f1950067d50).stop 2022-07-20T02:29:10.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- 172.21.15.26:0/2483103391 shutdown_connections 2022-07-20T02:29:10.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 --2- 172.21.15.26:0/2483103391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1938003750 0x7f1938003b70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- 172.21.15.26:0/2483103391 >> 172.21.15.26:0/2483103391 conn(0x7f1950060310 msgr2=0x7f19500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:10.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- 172.21.15.26:0/2483103391 shutdown_connections 2022-07-20T02:29:10.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- 172.21.15.26:0/2483103391 wait complete. 2022-07-20T02:29:10.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 Processor -- start 2022-07-20T02:29:10.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.876+0000 7f1956d69700 1 -- start start 2022-07-20T02:29:10.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938003750 0x7f195014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1950145cc0 0x7f1950159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19500add80 con 0x7f19500ac870 2022-07-20T02:29:10.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19500ad040 con 0x7f1950145cc0 2022-07-20T02:29:10.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19500ad2e0 con 0x7f1938003750 2022-07-20T02:29:10.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37968/0 (socket says 172.21.15.26:37968) 2022-07-20T02:29:10.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 -- 172.21.15.26:0/3721311514 learned_addr learned my addr 172.21.15.26:0/3721311514 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:10.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955d67700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938003750 0x7f195014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956568700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1950145cc0 0x7f1950159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 -- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938003750 msgr2=0x7f195014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938003750 0x7f195014e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 -- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1950145cc0 msgr2=0x7f1950159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:10.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1950145cc0 0x7f1950159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:10.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 -- 172.21.15.26:0/3721311514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500ad580 con 0x7f19500ac870 2022-07-20T02:29:10.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1955566700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f194400b000 tx=0x7f1944009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1944014620 con 0x7f19500ac870 2022-07-20T02:29:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1944014780 con 0x7f19500ac870 2022-07-20T02:29:10.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19500adfe0 con 0x7f19500ac870 2022-07-20T02:29:10.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1944014ae0 con 0x7f19500ac870 2022-07-20T02:29:10.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195014dc70 con 0x7f19500ac870 2022-07-20T02:29:10.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.880+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1944014c40 con 0x7f19500ac870 2022-07-20T02:29:10.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.884+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19500ae170 con 0x7f19500ac870 2022-07-20T02:29:10.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.884+0000 7f1942ffd700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 0x7f1930055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:10.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.884+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f194406c200 con 0x7f19500ac870 2022-07-20T02:29:10.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.884+0000 7f1955d67700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 0x7f1930055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:10.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.884+0000 7f1955d67700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 0x7f1930055ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f194c019000 tx=0x7f194c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:10.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:10.892+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1944042fc0 con 0x7f19500ac870 2022-07-20T02:29:11.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.016+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19500aa250 con 0x7f19500ac870 2022-07-20T02:29:11.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.016+0000 7f1942ffd700 1 -- 172.21.15.26:0/3721311514 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7f19440465e0 con 0x7f19500ac870 2022-07-20T02:29:11.019 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:11.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 msgr2=0x7f1930055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 0x7f1930055ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f194c019000 tx=0x7f194c015000).stop 2022-07-20T02:29:11.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 msgr2=0x7f19501457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f194400b000 tx=0x7f1944009040).stop 2022-07-20T02:29:11.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 shutdown_connections 2022-07-20T02:29:11.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19500ac870 0x7f19501457d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1930053b30 0x7f1930055ff0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1950145cc0 0x7f1950159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 --2- 172.21.15.26:0/3721311514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1938003750 0x7f195014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 >> 172.21.15.26:0/3721311514 conn(0x7f1950060310 msgr2=0x7f1950146bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:11.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 shutdown_connections 2022-07-20T02:29:11.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.020+0000 7f1956d69700 1 -- 172.21.15.26:0/3721311514 wait complete. 2022-07-20T02:29:11.032 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:29:11.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:11.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 Processor -- start 2022-07-20T02:29:11.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 -- start start 2022-07-20T02:29:11.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdbf005cfd0 0x7fdbf0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 0x7fdbf00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf01474e0 con 0x7fdbf0060f00 2022-07-20T02:29:11.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf005d3b0 con 0x7fdbf005cfd0 2022-07-20T02:29:11.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.136+0000 7fdbf67ca700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf01431e0 con 0x7fdbf0147000 2022-07-20T02:29:11.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdbf005cfd0 0x7fdbf0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdbf005cfd0 0x7fdbf0143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37978/0 (socket says 172.21.15.26:37978) 2022-07-20T02:29:11.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 -- 172.21.15.26:0/1123920133 learned_addr learned my addr 172.21.15.26:0/1123920133 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:11.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf4fc7700 1 --2- 172.21.15.26:0/1123920133 >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 0x7fdbf00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 -- 172.21.15.26:0/1123920133 >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 msgr2=0x7fdbf00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:11.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 --2- 172.21.15.26:0/1123920133 >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 0x7fdbf00b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 -- 172.21.15.26:0/1123920133 >> v1:172.21.15.120:6789/0 conn(0x7fdbf0060f00 legacy=0x7fdbf013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 -- 172.21.15.26:0/1123920133 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0143480 con 0x7fdbf005cfd0 2022-07-20T02:29:11.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf4fc7700 1 --2- 172.21.15.26:0/1123920133 >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 0x7fdbf00b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:11.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf57c8700 1 --2- 172.21.15.26:0/1123920133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf005cfd0 0x7fdbf0143bc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fdbe80032f0 tx=0x7fdbe8009290).ready entity=mon.0 client_cookie=396084dd79e47eae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:11.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbdffff700 1 -- 172.21.15.26:0/1123920133 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbe800b870 con 0x7fdbf005cfd0 2022-07-20T02:29:11.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbdffff700 1 -- 172.21.15.26:0/1123920133 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe800b9d0 con 0x7fdbf005cfd0 2022-07-20T02:29:11.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbdffff700 1 -- 172.21.15.26:0/1123920133 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbe800bb40 con 0x7fdbf005cfd0 2022-07-20T02:29:11.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- 172.21.15.26:0/1123920133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf005cfd0 msgr2=0x7fdbf0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/1123920133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf005cfd0 0x7fdbf0143bc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fdbe80032f0 tx=0x7fdbe8009290).stop 2022-07-20T02:29:11.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- 172.21.15.26:0/1123920133 shutdown_connections 2022-07-20T02:29:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/1123920133 >> v2:172.21.15.120:3300/0 conn(0x7fdbf0147000 0x7fdbf00b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/1123920133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf005cfd0 0x7fdbf0143bc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- 172.21.15.26:0/1123920133 >> 172.21.15.26:0/1123920133 conn(0x7fdbf00620c0 msgr2=0x7fdbf005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- 172.21.15.26:0/1123920133 shutdown_connections 2022-07-20T02:29:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- 172.21.15.26:0/1123920133 wait complete. 2022-07-20T02:29:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 Processor -- start 2022-07-20T02:29:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.140+0000 7fdbf67ca700 1 -- start start 2022-07-20T02:29:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbf0060f00 0x7fdbf00ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbf0147000 0x7fdbf00b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf00ae9d0 con 0x7fdbf00aeb30 2022-07-20T02:29:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf00b30f0 con 0x7fdbf0060f00 2022-07-20T02:29:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf00b37d0 con 0x7fdbf0147000 2022-07-20T02:29:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37982/0 (socket says 172.21.15.26:37982) 2022-07-20T02:29:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 -- 172.21.15.26:0/3566747816 learned_addr learned my addr 172.21.15.26:0/3566747816 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf57c8700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbf0060f00 0x7fdbf00ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf4fc7700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbf0147000 0x7fdbf00b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 -- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbf0147000 msgr2=0x7fdbf00b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbf0147000 0x7fdbf00b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 -- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbf0060f00 msgr2=0x7fdbf00ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbf0060f00 0x7fdbf00ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 -- 172.21.15.26:0/3566747816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8009530 con 0x7fdbf00aeb30 2022-07-20T02:29:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf5fc9700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fdbf01525d0 tx=0x7fdbe0007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbe000ce20 con 0x7fdbf00aeb30 2022-07-20T02:29:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe000f470 con 0x7fdbf00aeb30 2022-07-20T02:29:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf00b2e50 con 0x7fdbf00aeb30 2022-07-20T02:29:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbe000f7d0 con 0x7fdbf00aeb30 2022-07-20T02:29:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf00b32c0 con 0x7fdbf00aeb30 2022-07-20T02:29:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0143720 con 0x7fdbf00aeb30 2022-07-20T02:29:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdbe0019030 con 0x7fdbf00aeb30 2022-07-20T02:29:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 0x7fdbd0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.144+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdbe006b520 con 0x7fdbf00aeb30 2022-07-20T02:29:11.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.148+0000 7fdbf57c8700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 0x7fdbd0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:11.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.148+0000 7fdbf57c8700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 0x7fdbd0055ed0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdbe8019000 tx=0x7fdbe8016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:11.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.152+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdbe0042240 con 0x7fdbf00aeb30 2022-07-20T02:29:11.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.312+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf00574e0 con 0x7fdbf00aeb30 2022-07-20T02:29:11.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.312+0000 7fdbde7fc700 1 -- 172.21.15.26:0/3566747816 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7fdbe0045860 con 0x7fdbf00aeb30 2022-07-20T02:29:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:11.316 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:11.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 msgr2=0x7fdbd0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 0x7fdbd0055ed0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdbe8019000 tx=0x7fdbe8016000).stop 2022-07-20T02:29:11.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 msgr2=0x7fdbf00b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:11.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fdbf01525d0 tx=0x7fdbe0007bc0).stop 2022-07-20T02:29:11.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 shutdown_connections 2022-07-20T02:29:11.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbf00aeb30 0x7fdbf00b1c10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdbd0053a10 0x7fdbd0055ed0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbf0147000 0x7fdbf00b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 --2- 172.21.15.26:0/3566747816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbf0060f00 0x7fdbf00ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:11.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 >> 172.21.15.26:0/3566747816 conn(0x7fdbf005cfd0 msgr2=0x7fdbf00b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:11.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 shutdown_connections 2022-07-20T02:29:11.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:11.316+0000 7fdbf67ca700 1 -- 172.21.15.26:0/3566747816 wait complete. 2022-07-20T02:29:11.324 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:11.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:29:06.619736+0000', 'modified': '2022-07-20T02:29:07.623517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5616}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5616': {'gid': 5616, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3215220955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3215220955}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3215220955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:29:11.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:29:11.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:29:11.759 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5616,"want_state":"up:active","state":"up:active","rank_uptime":5.1266926919999998,"mdsmap_epoch":24,"osdmap_epoch":55,"osdmap_epoch_barrier":55,"uptime":9.4221098909999998} 2022-07-20T02:29:11.760 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5616, "mdsmap_epoch": 24, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "rank_uptime": 5.126692692, "state": "up:active", "uptime": 9.422109891, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:29:11.760 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:29:11.760 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:29:11.761 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:29:11.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:11.762 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:11.777 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:29:11.839 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:29:11.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:29:11.857 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:29:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:29:11.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:11.863 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:29:11.874 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:29:11.877 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:11.928 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:29:11.948 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:29:11.950 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:29:12.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:29:12.000+0000 7f2675d0f080 -1 init, newargv = 0x5644fe1747a0 newargc=9ceph-fuse[22567]: starting ceph client 2022-07-20T02:29:12.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:29:12.006 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:29:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:29:12.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.010 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:29:12.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22567]: starting fuse 2022-07-20T02:29:12.030 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:29:12.031 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:29:12.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.046 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:29:12.047 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:29:12.048 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:29:12.048 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:29:12.049 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:29:12.055 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.067 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:29:12.067 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:29:12.068 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:29:12.069 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.070 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:29:12.071 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:29:12.071 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:29:12.072 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:29:12.073 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.074 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:29:12.075 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:29:12.075 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:29:12.077 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:29:12.078 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:29:12.081 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.083 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:29:12.084 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:29:12.134 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22567.asok 2022-07-20T02:29:12.137 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22567.asok 2022-07-20T02:29:12.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.140 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22567.asok status 2022-07-20T02:29:12.282 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:29:12.283 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:29:12.283 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:29:12.284 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:29:12.284 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:29:12.285 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:29:12.285 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:29:12.286 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22567", 2022-07-20T02:29:12.286 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:29:12.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:12.291 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:29:12.291 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:29:12.292 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 5722, 2022-07-20T02:29:12.292 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:29:12.292 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:29:12.292 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:29:12.293 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 5722 2022-07-20T02:29:12.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:12.294 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:29:12.295 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:29:12.295 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:29:12.295 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 4216255250 2022-07-20T02:29:12.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:29:12.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:12.297 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:29:12.297 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:29:12.297 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:29:12.298 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 4216255250 2022-07-20T02:29:12.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:12.299 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.5722 172.21.15.159:0/4216255250", 2022-07-20T02:29:12.300 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/4216255250", 2022-07-20T02:29:12.300 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:29:12.301 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 24, 2022-07-20T02:29:12.301 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 55, 2022-07-20T02:29:12.315 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:29:12.317 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:29:12.318 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:29:12.318 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:29:12.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.329 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:12.329 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:12.338 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:29:12.344 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:29:12.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.347 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:12.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:12.413 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:12.414 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:12.432 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:29:12.432 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:29:12.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:29:12.434 DEBUG:teuthology.orchestra.run.smithi159:> import sys 2022-07-20T02:29:12.434 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:29:12.436 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:29:12.436 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.443 DEBUG:teuthology.orchestra.run.smithi159:> n = 1000 2022-07-20T02:29:12.444 DEBUG:teuthology.orchestra.run.smithi159:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T02:29:12.444 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.445 DEBUG:teuthology.orchestra.run.smithi159:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T02:29:12.446 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T02:29:12.447 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:12.447 DEBUG:teuthology.orchestra.run.smithi159:> for i in range(0, n): 2022-07-20T02:29:12.448 DEBUG:teuthology.orchestra.run.smithi159:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T02:29:12.448 DEBUG:teuthology.orchestra.run.smithi159:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T02:29:12.449 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:29:12.449 DEBUG:teuthology.orchestra.run.smithi159:> if True: 2022-07-20T02:29:12.450 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:29:12.451 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(f.fileno()) 2022-07-20T02:29:12.451 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:29:16.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:16.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:16.661 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:16.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:23.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:23.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.880+0000 7f1f259fb700 1 Processor -- start 2022-07-20T02:29:23.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- start start 2022-07-20T02:29:23.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 0x7f1f20147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f20061050 0x7f1f20148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b1bc0 con 0x7f1f20066a80 2022-07-20T02:29:23.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2014b040 con 0x7f1f20061050 2022-07-20T02:29:23.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f200a8860 con 0x7f1f200b2920 2022-07-20T02:29:23.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f20061050 0x7f1f20148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f20061050 0x7f1f20148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37988/0 (socket says 172.21.15.26:37988) 2022-07-20T02:29:23.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 -- 172.21.15.26:0/3441205250 learned_addr learned my addr 172.21.15.26:0/3441205250 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:23.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1effd700 1 --2- 172.21.15.26:0/3441205250 >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 0x7f1f20147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 -- 172.21.15.26:0/3441205250 >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 msgr2=0x7f1f20147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:23.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 --2- 172.21.15.26:0/3441205250 >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 0x7f1f20147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:23.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 -- 172.21.15.26:0/3441205250 >> v1:172.21.15.120:6789/0 conn(0x7f1f20066a80 legacy=0x7f1f20066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:23.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 -- 172.21.15.26:0/3441205250 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20147700 con 0x7f1f20061050 2022-07-20T02:29:23.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1effd700 1 --2- 172.21.15.26:0/3441205250 >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 0x7f1f20147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:29:23.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1e7fc700 1 --2- 172.21.15.26:0/3441205250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20061050 0x7f1f20148280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f1f20147c70 tx=0x7f1f10006eb0).ready entity=mon.0 client_cookie=e876d3ec00cec6ce server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:23.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1dffb700 1 -- 172.21.15.26:0/3441205250 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1000d5c0 con 0x7f1f20061050 2022-07-20T02:29:23.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1dffb700 1 -- 172.21.15.26:0/3441205250 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10000e90 con 0x7f1f20061050 2022-07-20T02:29:23.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f1dffb700 1 -- 172.21.15.26:0/3441205250 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1000d910 con 0x7f1f20061050 2022-07-20T02:29:23.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- 172.21.15.26:0/3441205250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20061050 msgr2=0x7f1f20148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:23.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 --2- 172.21.15.26:0/3441205250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20061050 0x7f1f20148280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f1f20147c70 tx=0x7f1f10006eb0).stop 2022-07-20T02:29:23.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- 172.21.15.26:0/3441205250 shutdown_connections 2022-07-20T02:29:23.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 --2- 172.21.15.26:0/3441205250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20061050 0x7f1f20148280 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:23.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 --2- 172.21.15.26:0/3441205250 >> v2:172.21.15.120:3300/0 conn(0x7f1f200b2920 0x7f1f20147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:23.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- 172.21.15.26:0/3441205250 >> 172.21.15.26:0/3441205250 conn(0x7f1f200b60a0 msgr2=0x7f1f200b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:23.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- 172.21.15.26:0/3441205250 shutdown_connections 2022-07-20T02:29:23.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 -- 172.21.15.26:0/3441205250 wait complete. 2022-07-20T02:29:23.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.884+0000 7f1f259fb700 1 Processor -- start 2022-07-20T02:29:23.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- start start 2022-07-20T02:29:23.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20061050 0x7f1f200b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f20066a80 0x7f1f200ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200ab5a0 con 0x7f1f200b2920 2022-07-20T02:29:23.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200632c0 con 0x7f1f20066a80 2022-07-20T02:29:23.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20063560 con 0x7f1f20061050 2022-07-20T02:29:23.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37998/0 (socket says 172.21.15.26:37998) 2022-07-20T02:29:23.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/2907330637 learned_addr learned my addr 172.21.15.26:0/2907330637 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:23.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1e7fc700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f20066a80 0x7f1f200ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1effd700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20061050 0x7f1f200b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20061050 msgr2=0x7f1f200b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:23.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20061050 0x7f1f200b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:23.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f20066a80 msgr2=0x7f1f200ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:23.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f20066a80 0x7f1f200ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:23.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/2907330637 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1000b1e0 con 0x7f1f200b2920 2022-07-20T02:29:23.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f1f18002320 tx=0x7f1f180023f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:23.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f18006aa0 con 0x7f1f200b2920 2022-07-20T02:29:23.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18006c00 con 0x7f1f200b2920 2022-07-20T02:29:23.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f200aac10 con 0x7f1f200b2920 2022-07-20T02:29:23.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f18007030 con 0x7f1f200b2920 2022-07-20T02:29:23.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.888+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f200641b0 con 0x7f1f200b2920 2022-07-20T02:29:23.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f20063720 con 0x7f1f200b2920 2022-07-20T02:29:23.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f18010030 con 0x7f1f200b2920 2022-07-20T02:29:23.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f0ffff700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 0x7f1efc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:23.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1f1806bfe0 con 0x7f1f200b2920 2022-07-20T02:29:23.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f1effd700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 0x7f1efc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:23.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.892+0000 7f1f1effd700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 0x7f1efc055e10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1f1400d000 tx=0x7f1f14008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:23.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:23.900+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f18042cb0 con 0x7f1f200b2920 2022-07-20T02:29:24.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.060+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f200aada0 con 0x7f1f200b2920 2022-07-20T02:29:24.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.060+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2907330637 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f1f180462d0 con 0x7f1f200b2920 2022-07-20T02:29:24.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:24.063 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:24.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 msgr2=0x7f1efc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 0x7f1efc055e10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1f1400d000 tx=0x7f1f14008040).stop 2022-07-20T02:29:24.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 msgr2=0x7f1f201568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f1f18002320 tx=0x7f1f180023f0).stop 2022-07-20T02:29:24.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 shutdown_connections 2022-07-20T02:29:24.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f200b2920 0x7f1f201568b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1efc053950 0x7f1efc055e10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f20066a80 0x7f1f200ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 --2- 172.21.15.26:0/2907330637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20061050 0x7f1f200b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 >> 172.21.15.26:0/2907330637 conn(0x7f1f200b60a0 msgr2=0x7f1f20149650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:24.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 shutdown_connections 2022-07-20T02:29:24.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.064+0000 7f1f259fb700 1 -- 172.21.15.26:0/2907330637 wait complete. 2022-07-20T02:29:24.070 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:24.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config set mds_recall_max_caps 20 2022-07-20T02:29:24.203 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T02:29:24.204 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T02:29:24.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:24.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.308+0000 7f476f3c8700 1 Processor -- start 2022-07-20T02:29:24.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.308+0000 7f476f3c8700 1 -- start start 2022-07-20T02:29:24.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680b5ef0 0x7f47681474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47680a7fc0 0x7f47681479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47680a8c40 0x7f47680aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4768147eb0 con 0x7f47680a7fc0 2022-07-20T02:29:24.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47680b62d0 con 0x7f47680a8c40 2022-07-20T02:29:24.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476814aac0 con 0x7f47680b5ef0 2022-07-20T02:29:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47680a7fc0 0x7f47681479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47680a7fc0 0x7f47681479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38004/0 (socket says 172.21.15.26:38004) 2022-07-20T02:29:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 -- 172.21.15.26:0/3821011497 learned_addr learned my addr 172.21.15.26:0/3821011497 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476e3c6700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680b5ef0 0x7f47681474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476ebc7700 1 --2- 172.21.15.26:0/3821011497 >> v2:172.21.15.120:3300/0 conn(0x7f47680a8c40 0x7f47680aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 -- 172.21.15.26:0/3821011497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680b5ef0 msgr2=0x7f47681474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:24.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680b5ef0 0x7f47681474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 -- 172.21.15.26:0/3821011497 >> v2:172.21.15.120:3300/0 conn(0x7f47680a8c40 msgr2=0x7f47680aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:24.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 --2- 172.21.15.26:0/3821011497 >> v2:172.21.15.120:3300/0 conn(0x7f47680a8c40 0x7f47680aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 -- 172.21.15.26:0/3821011497 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476814ad60 con 0x7f47680a7fc0 2022-07-20T02:29:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476dbc5700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680a7fc0 0x7f47681479c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f475c002f70 tx=0x7f475c006eb0).ready entity=mon.0 client_cookie=391af7c8ce9838fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476d3c4700 1 -- 172.21.15.26:0/3821011497 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475c00d570 con 0x7f47680a7fc0 2022-07-20T02:29:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476d3c4700 1 -- 172.21.15.26:0/3821011497 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c000e90 con 0x7f47680a7fc0 2022-07-20T02:29:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476d3c4700 1 -- 172.21.15.26:0/3821011497 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475c00d900 con 0x7f47680a7fc0 2022-07-20T02:29:24.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- 172.21.15.26:0/3821011497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680a7fc0 msgr2=0x7f47681479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680a7fc0 0x7f47681479c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f475c002f70 tx=0x7f475c006eb0).stop 2022-07-20T02:29:24.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- 172.21.15.26:0/3821011497 shutdown_connections 2022-07-20T02:29:24.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- 172.21.15.26:0/3821011497 >> v2:172.21.15.120:3300/0 conn(0x7f47680a8c40 0x7f47680aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680a7fc0 0x7f47681479c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 --2- 172.21.15.26:0/3821011497 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680b5ef0 0x7f47681474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- 172.21.15.26:0/3821011497 >> 172.21.15.26:0/3821011497 conn(0x7f4768060310 msgr2=0x7f47680b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:24.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- 172.21.15.26:0/3821011497 shutdown_connections 2022-07-20T02:29:24.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.312+0000 7f476f3c8700 1 -- 172.21.15.26:0/3821011497 wait complete. 2022-07-20T02:29:24.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 Processor -- start 2022-07-20T02:29:24.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- start start 2022-07-20T02:29:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680a8c40 0x7f47680aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 0x7f4768146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4768067cb0 con 0x7f47680b5ef0 2022-07-20T02:29:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47680ab3f0 con 0x7f47680a7fc0 2022-07-20T02:29:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47680ab690 con 0x7f47680a8c40 2022-07-20T02:29:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 0x7f4768146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476e3c6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476e3c6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:57700/0 (socket says 172.21.15.26:57700) 2022-07-20T02:29:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476e3c6700 1 -- 172.21.15.26:0/1671870591 learned_addr learned my addr 172.21.15.26:0/1671870591 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476dbc5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680a8c40 0x7f47680aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 -- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680a8c40 msgr2=0x7f47680aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680a8c40 0x7f47680aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 -- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 msgr2=0x7f47680abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 -- 172.21.15.26:0/1671870591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c00b1e0 con 0x7f47680b5ef0 2022-07-20T02:29:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476e3c6700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476ebc7700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 0x7f4768146840 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f475800a000 tx=0x7f4758008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758013670 con 0x7f47680b5ef0 2022-07-20T02:29:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47580137d0 con 0x7f47680b5ef0 2022-07-20T02:29:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4768063170 con 0x7f47680b5ef0 2022-07-20T02:29:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758013b30 con 0x7f47680b5ef0 2022-07-20T02:29:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.316+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4768064a20 con 0x7f47680b5ef0 2022-07-20T02:29:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f47580123e0 con 0x7f47680b5ef0 2022-07-20T02:29:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47680ab250 con 0x7f47680b5ef0 2022-07-20T02:29:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f47677fe700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 0x7f4748055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f475806bda0 con 0x7f47680b5ef0 2022-07-20T02:29:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f476e3c6700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 0x7f4748055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.320+0000 7f476e3c6700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 0x7f4748055f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4760006e00 tx=0x7f47600091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.324+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47680ab250 con 0x7f47680b5ef0 2022-07-20T02:29:24.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.488+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47680aa910 con 0x7f47680b5ef0 2022-07-20T02:29:24.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.488+0000 7f47677fe700 1 -- 172.21.15.26:0/1671870591 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f47680aa910 con 0x7f47680b5ef0 2022-07-20T02:29:24.491 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:24.491 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:24.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 msgr2=0x7f4748055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 0x7f4748055f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4760006e00 tx=0x7f47600091d0).stop 2022-07-20T02:29:24.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 msgr2=0x7f4768146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 0x7f4768146840 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f475800a000 tx=0x7f4758008040).stop 2022-07-20T02:29:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 shutdown_connections 2022-07-20T02:29:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47680b5ef0 0x7f4768146840 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4748053aa0 0x7f4748055f60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47680a8c40 0x7f47680aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 --2- 172.21.15.26:0/1671870591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47680a7fc0 0x7f47680abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 >> 172.21.15.26:0/1671870591 conn(0x7f4768060310 msgr2=0x7f47680b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 shutdown_connections 2022-07-20T02:29:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.492+0000 7f476f3c8700 1 -- 172.21.15.26:0/1671870591 wait complete. 2022-07-20T02:29:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:24.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T02:29:24.616 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T02:29:24.617 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T02:29:24.620 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:29:24.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:24.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 Processor -- start 2022-07-20T02:29:24.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 -- start start 2022-07-20T02:29:24.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef0c0adf90 0x7fef0c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c0ae370 con 0x7fef0c0614b0 2022-07-20T02:29:24.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c062050 con 0x7fef0c0b14b0 2022-07-20T02:29:24.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef1471c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c0b28b0 con 0x7fef0c0adf90 2022-07-20T02:29:24.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef12cb9700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fef0c0614b0 0x7fef0c061940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37300/0 (socket says 172.21.15.26:37300) 2022-07-20T02:29:24.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef12cb9700 1 -- 172.21.15.26:0/2800232003 learned_addr learned my addr 172.21.15.26:0/2800232003 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:24.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> v2:172.21.15.120:3300/0 conn(0x7fef0c0adf90 0x7fef0c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429650884 0 0) 0x7fef0c0ae370 con 0x7fef0c0614b0 2022-07-20T02:29:24.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef4003260 con 0x7fef0c0614b0 2022-07-20T02:29:24.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118088458 0 0) 0x7fef0c062050 con 0x7fef0c0b14b0 2022-07-20T02:29:24.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0c0ae370 con 0x7fef0c0b14b0 2022-07-20T02:29:24.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3630301680 0 0) 0x7feef4003260 con 0x7fef0c0614b0 2022-07-20T02:29:24.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 >> v2:172.21.15.120:3300/0 conn(0x7fef0c0adf90 msgr2=0x7fef0c0b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:24.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 --2- 172.21.15.26:0/2800232003 >> v2:172.21.15.120:3300/0 conn(0x7fef0c0adf90 0x7fef0c0b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 >> v1:172.21.15.120:6789/0 conn(0x7fef0c0b14b0 legacy=0x7fef0c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c0b0610 con 0x7fef0c0614b0 2022-07-20T02:29:24.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fef04001a00 con 0x7fef0c0614b0 2022-07-20T02:29:24.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> v2:172.21.15.120:3300/0 conn(0x7fef0c0adf90 0x7fef0c0b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:24.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 >> v1:172.21.15.26:6789/0 conn(0x7fef0c0614b0 legacy=0x7fef0c061940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 0x7feef4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4004500 0x7feef4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef4004e70 0x7feef400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0b28b0 con 0x7feef4003930 2022-07-20T02:29:24.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 0x7feef4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef400bd20 con 0x7feef4004e70 2022-07-20T02:29:24.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef400bee0 con 0x7feef4004500 2022-07-20T02:29:24.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef12cb9700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef4004e70 0x7feef400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef11cb7700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4004500 0x7feef4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 -- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4004500 msgr2=0x7feef4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:24.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4004500 0x7feef4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 -- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef4004e70 msgr2=0x7feef400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:24.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef4004e70 0x7feef400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef12cb9700 1 -- 172.21.15.26:0/2800232003 reap_dead start 2022-07-20T02:29:24.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 -- 172.21.15.26:0/2800232003 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4003260 con 0x7feef4003930 2022-07-20T02:29:24.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.760+0000 7fef124b8700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 0x7feef4003d30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fef0800c000 tx=0x7fef08006df0).ready entity=mon.0 client_cookie=b2f378e0513e9725 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0800e830 con 0x7feef4003930 2022-07-20T02:29:24.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0800e990 con 0x7feef4003930 2022-07-20T02:29:24.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef114b6700 1 -- 172.21.15.26:0/2800232003 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0800ecf0 con 0x7feef4003930 2022-07-20T02:29:24.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 msgr2=0x7feef4003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 0x7feef4003d30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fef0800c000 tx=0x7fef08006df0).stop 2022-07-20T02:29:24.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- 172.21.15.26:0/2800232003 shutdown_connections 2022-07-20T02:29:24.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 --2- 172.21.15.26:0/2800232003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feef4003930 0x7feef4003d30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- 172.21.15.26:0/2800232003 >> 172.21.15.26:0/2800232003 conn(0x7fef0c0b66e0 msgr2=0x7fef0c0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:24.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- 172.21.15.26:0/2800232003 shutdown_connections 2022-07-20T02:29:24.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- 172.21.15.26:0/2800232003 wait complete. 2022-07-20T02:29:24.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 Processor -- start 2022-07-20T02:29:24.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- start start 2022-07-20T02:29:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4003930 0x7fef0c141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef0c1503a0 0x7fef0c1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c066f50 con 0x7fef0c0a9930 2022-07-20T02:29:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c153af0 con 0x7fef0c1503a0 2022-07-20T02:29:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.764+0000 7fef1471c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c066cb0 con 0x7feef4003930 2022-07-20T02:29:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef124b8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4003930 0x7fef0c141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38032/0 (socket says 172.21.15.26:38032) 2022-07-20T02:29:24.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 -- 172.21.15.26:0/3298534262 learned_addr learned my addr 172.21.15.26:0/3298534262 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:24.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef11cb7700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef0c1503a0 0x7fef0c1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 -- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4003930 msgr2=0x7fef0c141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4003930 0x7fef0c141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 -- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef0c1503a0 msgr2=0x7fef0c1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef0c1503a0 0x7fef0c1541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 -- 172.21.15.26:0/3298534262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c153d50 con 0x7fef0c0a9930 2022-07-20T02:29:24.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef12cb9700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fef0c06b470 tx=0x7fef0c067420).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef040116b0 con 0x7fef0c0a9930 2022-07-20T02:29:24.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef04011810 con 0x7fef0c0a9930 2022-07-20T02:29:24.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c0aa2a0 con 0x7fef0c0a9930 2022-07-20T02:29:24.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef04011b70 con 0x7fef0c0a9930 2022-07-20T02:29:24.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c1546e0 con 0x7fef0c0a9930 2022-07-20T02:29:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.768+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c153ee0 con 0x7fef0c0a9930 2022-07-20T02:29:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.772+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef04011cd0 con 0x7fef0c0a9930 2022-07-20T02:29:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.772+0000 7feeff7fe700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 0x7feeec055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.772+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fef0406b950 con 0x7fef0c0a9930 2022-07-20T02:29:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.772+0000 7fef124b8700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 0x7feeec055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.772+0000 7fef124b8700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 0x7feeec055ff0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fef0801d000 tx=0x7fef08019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.780+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef04042710 con 0x7fef0c0a9930 2022-07-20T02:29:24.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.940+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef0c0574e0 con 0x7fef0c0a9930 2022-07-20T02:29:24.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7feeff7fe700 1 -- 172.21.15.26:0/3298534262 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7fef04019020 con 0x7fef0c0a9930 2022-07-20T02:29:24.944 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:24.944 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:24.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 msgr2=0x7feeec055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 0x7feeec055ff0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fef0801d000 tx=0x7fef08019000).stop 2022-07-20T02:29:24.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 msgr2=0x7fef0c0a9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:24.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fef0c06b470 tx=0x7fef0c067420).stop 2022-07-20T02:29:24.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 shutdown_connections 2022-07-20T02:29:24.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef0c0a9930 0x7fef0c0a9db0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feeec053b30 0x7feeec055ff0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef0c1503a0 0x7fef0c1541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 --2- 172.21.15.26:0/3298534262 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feef4003930 0x7fef0c141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:24.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 >> 172.21.15.26:0/3298534262 conn(0x7fef0c0b66e0 msgr2=0x7fef0c0b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 shutdown_connections 2022-07-20T02:29:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:24.944+0000 7fef1471c700 1 -- 172.21.15.26:0/3298534262 wait complete. 2022-07-20T02:29:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:24.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c cache drop 5 2022-07-20T02:29:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 Processor -- start 2022-07-20T02:29:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 -- start start 2022-07-20T02:29:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fad980aff40 0x7fad98066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad980621f0 0x7fad98067020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980b0f50 con 0x7fad98060250 2022-07-20T02:29:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad98060df0 con 0x7fad980aff40 2022-07-20T02:29:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad9ce9b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9813e7f0 con 0x7fad980621f0 2022-07-20T02:29:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad977fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fad980aff40 0x7fad98066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad97fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fad98060250 0x7fad980606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37320/0 (socket says 172.21.15.26:37320) 2022-07-20T02:29:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad97fff700 1 -- 172.21.15.26:0/2647639176 learned_addr learned my addr 172.21.15.26:0/2647639176 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.072+0000 7fad96ffd700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad980621f0 0x7fad98067020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad967fc700 1 -- 172.21.15.26:0/2647639176 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180562732 0 0) 0x7fad980b0f50 con 0x7fad98060250 2022-07-20T02:29:25.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad967fc700 1 -- 172.21.15.26:0/2647639176 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003260 con 0x7fad98060250 2022-07-20T02:29:25.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad977fe700 1 -- 172.21.15.26:0/2647639176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad980621f0 msgr2=0x7fad98067020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad977fe700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad980621f0 0x7fad98067020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad977fe700 1 -- 172.21.15.26:0/2647639176 >> v1:172.21.15.26:6789/0 conn(0x7fad98060250 legacy=0x7fad980606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad977fe700 1 -- 172.21.15.26:0/2647639176 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9805f8e0 con 0x7fad980aff40 2022-07-20T02:29:25.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad977fe700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980aff40 0x7fad98066b30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fad88002f70 tx=0x7fad88006df0).ready entity=mon.0 client_cookie=c49b3eb345f912e6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad967fc700 1 -- 172.21.15.26:0/2647639176 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8800d4a0 con 0x7fad980aff40 2022-07-20T02:29:25.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad967fc700 1 -- 172.21.15.26:0/2647639176 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad88000e90 con 0x7fad980aff40 2022-07-20T02:29:25.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad967fc700 1 -- 172.21.15.26:0/2647639176 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8800d7f0 con 0x7fad980aff40 2022-07-20T02:29:25.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2647639176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980aff40 msgr2=0x7fad98066b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980aff40 0x7fad98066b30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fad88002f70 tx=0x7fad88006df0).stop 2022-07-20T02:29:25.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2647639176 shutdown_connections 2022-07-20T02:29:25.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad980621f0 0x7fad98067020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2647639176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980aff40 0x7fad98066b30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2647639176 >> 172.21.15.26:0/2647639176 conn(0x7fad980b3050 msgr2=0x7fad980b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:25.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2647639176 shutdown_connections 2022-07-20T02:29:25.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.076+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2647639176 wait complete. 2022-07-20T02:29:25.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 Processor -- start 2022-07-20T02:29:25.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- start start 2022-07-20T02:29:25.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad98060250 0x7fad9813f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 0x7fad98065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980aca00 con 0x7fad980621f0 2022-07-20T02:29:25.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980acc30 con 0x7fad980ac220 2022-07-20T02:29:25.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad981541f0 con 0x7fad98060250 2022-07-20T02:29:25.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38042/0 (socket says 172.21.15.26:38042) 2022-07-20T02:29:25.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 -- 172.21.15.26:0/2370908616 learned_addr learned my addr 172.21.15.26:0/2370908616 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:25.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad97fff700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 0x7fad98065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad977fe700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad98060250 0x7fad9813f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 -- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad98060250 msgr2=0x7fad9813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad98060250 0x7fad9813f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 -- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 msgr2=0x7fad98065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 0x7fad98065ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 -- 172.21.15.26:0/2370908616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8800b180 con 0x7fad980621f0 2022-07-20T02:29:25.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad97fff700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 0x7fad98065ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:25.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad96ffd700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fad8000c000 tx=0x7fad80006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad94ff9700 1 -- 172.21.15.26:0/2370908616 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad80010780 con 0x7fad980621f0 2022-07-20T02:29:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad94ff9700 1 -- 172.21.15.26:0/2370908616 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad800108e0 con 0x7fad980621f0 2022-07-20T02:29:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98142200 con 0x7fad980621f0 2022-07-20T02:29:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad94ff9700 1 -- 172.21.15.26:0/2370908616 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad80010c40 con 0x7fad980621f0 2022-07-20T02:29:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.080+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad98066680 con 0x7fad980621f0 2022-07-20T02:29:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fad94ff9700 1 -- 172.21.15.26:0/2370908616 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad8001c460 con 0x7fad980621f0 2022-07-20T02:29:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fad94ff9700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 0x7fad7c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fad977fe700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 0x7fad7c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fad94ff9700 1 -- 172.21.15.26:0/2370908616 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad8006c5f0 con 0x7fad980621f0 2022-07-20T02:29:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fad977fe700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 0x7fad7c055e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fad88018000 tx=0x7fad88014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fada8361740 1 Processor -- start 2022-07-20T02:29:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fada8361740 1 -- start start 2022-07-20T02:29:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.084+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2b3ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b42010 0x2b42490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2ab1210 con 0x2b76a90 2022-07-20T02:29:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a91450 con 0x2b3e6f0 2022-07-20T02:29:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2b3e390 con 0x2b42010 2022-07-20T02:29:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad977fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2b3ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38054/0 (socket says 172.21.15.26:38054) 2022-07-20T02:29:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 -- 172.21.15.26:0/30025443 learned_addr learned my addr 172.21.15.26:0/30025443 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad97fff700 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b42010 0x2b42490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 -- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b42010 msgr2=0x2b42490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b42010 0x2b42490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 -- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 msgr2=0x2b3ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2b3ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 -- 172.21.15.26:0/30025443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2ae8ad0 con 0x2b76a90 2022-07-20T02:29:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad96ffd700 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fad8006c800 tx=0x7fad80014000).ready entity=mon.0 client_cookie=fde973e05bcb36fb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad9ce9b700 1 -- 172.21.15.26:0/30025443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad80045110 con 0x2b76a90 2022-07-20T02:29:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad9ce9b700 1 -- 172.21.15.26:0/30025443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8000fd30 con 0x2b76a90 2022-07-20T02:29:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fad9ce9b700 1 -- 172.21.15.26:0/30025443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8000fea0 con 0x2b76a90 2022-07-20T02:29:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- 172.21.15.26:0/30025443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 msgr2=0x2b76ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fad8006c800 tx=0x7fad80014000).stop 2022-07-20T02:29:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- 172.21.15.26:0/30025443 shutdown_connections 2022-07-20T02:29:25.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b76ef0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b42010 0x2b42490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 --2- 172.21.15.26:0/30025443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2b3ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- 172.21.15.26:0/30025443 >> 172.21.15.26:0/30025443 conn(0x2ac88f0 msgr2=0x2b3f470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- 172.21.15.26:0/30025443 shutdown_connections 2022-07-20T02:29:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- 172.21.15.26:0/30025443 wait complete. 2022-07-20T02:29:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.088+0000 7fada8361740 1 -- start start 2022-07-20T02:29:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 Processor -- start 2022-07-20T02:29:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b3e6f0 0x2b853f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b85000 con 0x2b76a90 2022-07-20T02:29:25.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b46f10 con 0x2b42010 2022-07-20T02:29:25.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2b471b0 con 0x2b3e6f0 2022-07-20T02:29:25.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 10 client.0 ll_register_callbacks cb 0x2af80b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:29:25.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38058/0 (socket says 172.21.15.26:38058) 2022-07-20T02:29:25.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 -- 172.21.15.26:0/3670467063 learned_addr learned my addr 172.21.15.26:0/3670467063 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:25.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad96ffd700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad977fe700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b3e6f0 0x2b853f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 -- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b3e6f0 msgr2=0x2b853f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b3e6f0 0x2b853f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 -- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 msgr2=0x2b858e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b858e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 -- 172.21.15.26:0/3670467063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad98066680 con 0x2b76a90 2022-07-20T02:29:25.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad97fff700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fad9000d000 tx=0x7fad90009040).ready entity=mon.0 client_cookie=cf6b4c225cfd0eb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:29:25.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad90005d70 con 0x2b76a90 2022-07-20T02:29:25.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad90016e00 con 0x2b76a90 2022-07-20T02:29:25.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2aa71c0 con 0x2b76a90 2022-07-20T02:29:25.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad900070d0 con 0x2b76a90 2022-07-20T02:29:25.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 10 client.5820 fetch_fsmap learned FSMap version 24 2022-07-20T02:29:25.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x2aa07b0 con 0x2b76a90 2022-07-20T02:29:25.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2aa71c0 con 0x2b76a90 2022-07-20T02:29:25.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fad900143d0 con 0x2b76a90 2022-07-20T02:29:25.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 10 client.5820 fetch_fsmap finished waiting for FSMap version 24 2022-07-20T02:29:25.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 10 client.5820 resolve_mds: resolved ID 'c' to GID 5616 2022-07-20T02:29:25.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 20 client.5820 populate_metadata read hostname 'smithi026' 2022-07-20T02:29:25.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b79860 0x2b713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 4 client.5820 mds_command: new command op to 5616 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T02:29:25.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 --> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2a9f240 con 0x2b79860 2022-07-20T02:29:25.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.092+0000 7fad977fe700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b79860 0x2b713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fad767fc700 10 client.5820 ms_handle_connect on v2:172.21.15.26:6836/3215220955 2022-07-20T02:29:25.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fad977fe700 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b79860 0x2b713f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=900e2cc66504e430 server_cookie=760d11e1cfef92aa in_seq=0 out_seq=0 2022-07-20T02:29:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fad767fc700 1 -- 172.21.15.26:0/3670467063 <== mds.0 v2:172.21.15.26:6836/3215220955 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x2a9f240 con 0x2b79860 2022-07-20T02:29:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fad767fc700 10 client.5820 handle_command_reply: tid=0 2022-07-20T02:29:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 client.5820 shutdown 2022-07-20T02:29:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 msgr2=0x2b78ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fad9000d000 tx=0x7fad90009040).stop 2022-07-20T02:29:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 shutdown_connections 2022-07-20T02:29:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b79860 0x2b713f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b76a90 0x2b78ea0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 --2- 172.21.15.26:0/3670467063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b3e6f0 0x2b853f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 >> 172.21.15.26:0/3670467063 conn(0x2ac88f0 msgr2=0x2b76160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fad767fc700 0 client.5820 ms_handle_reset on v2:172.21.15.26:6836/3215220955 2022-07-20T02:29:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 shutdown_connections 2022-07-20T02:29:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 1 -- 172.21.15.26:0/3670467063 wait complete. 2022-07-20T02:29:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.096+0000 7fada8361740 20 client.5820 trim_cache size 0 max 16384 2022-07-20T02:29:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 Processor -- start 2022-07-20T02:29:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 -- start start 2022-07-20T02:29:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b3d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b71360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b2c690 con 0x2b3e6f0 2022-07-20T02:29:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b749a0 con 0x2b42010 2022-07-20T02:29:25.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2b77c00 con 0x2b76a90 2022-07-20T02:29:25.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38064/0 (socket says 172.21.15.26:38064) 2022-07-20T02:29:25.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 -- 172.21.15.26:0/1550378049 learned_addr learned my addr 172.21.15.26:0/1550378049 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:25.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad97fff700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b71360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 -- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 msgr2=0x2b71360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad96ffd700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b3d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b71360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 -- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 msgr2=0x2b3d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b3d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 -- 172.21.15.26:0/1550378049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2aa93a0 con 0x2b3e6f0 2022-07-20T02:29:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad96ffd700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b3d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:29:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad97fff700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b71360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:29:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad977fe700 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x2b3ff10 tx=0x7fad88012000).ready entity=mon.0 client_cookie=6b0465642f2930ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad8cff9700 1 -- 172.21.15.26:0/1550378049 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad88016040 con 0x2b3e6f0 2022-07-20T02:29:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad8cff9700 1 -- 172.21.15.26:0/1550378049 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad880300e0 con 0x2b3e6f0 2022-07-20T02:29:25.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.112+0000 7fad8cff9700 1 -- 172.21.15.26:0/1550378049 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad880305e0 con 0x2b3e6f0 2022-07-20T02:29:25.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- 172.21.15.26:0/1550378049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 msgr2=0x2b6eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x2b3ff10 tx=0x7fad88012000).stop 2022-07-20T02:29:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- 172.21.15.26:0/1550378049 shutdown_connections 2022-07-20T02:29:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b71360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b42010 0x2b3d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- 172.21.15.26:0/1550378049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b3e6f0 0x2b6eb60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- 172.21.15.26:0/1550378049 >> 172.21.15.26:0/1550378049 conn(0x2ac88f0 msgr2=0x2b75f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- 172.21.15.26:0/1550378049 shutdown_connections 2022-07-20T02:29:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- 172.21.15.26:0/1550378049 wait complete. 2022-07-20T02:29:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- start start 2022-07-20T02:29:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 Processor -- start 2022-07-20T02:29:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2bdb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b77e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b85d20 con 0x2b42010 2022-07-20T02:29:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b94690 con 0x2b3e6f0 2022-07-20T02:29:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2b85f10 con 0x2b76a90 2022-07-20T02:29:25.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.116+0000 7fada8361740 10 client.0 ll_register_callbacks cb 0x2b46e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:29:25.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38072/0 (socket says 172.21.15.26:38072) 2022-07-20T02:29:25.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 -- 172.21.15.26:0/3827946204 learned_addr learned my addr 172.21.15.26:0/3827946204 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:25.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad97fff700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b77e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad977fe700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2bdb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 -- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 msgr2=0x2b77e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b77e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 -- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 msgr2=0x2bdb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:25.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2bdb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 -- 172.21.15.26:0/3827946204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9805f8e0 con 0x2b42010 2022-07-20T02:29:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:29:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad96ffd700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fad8001e000 tx=0x7fad8001d000).ready entity=mon.0 client_cookie=821111f655226411 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad800463f0 con 0x2b42010 2022-07-20T02:29:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad80046550 con 0x2b42010 2022-07-20T02:29:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad8000fd30 con 0x2b42010 2022-07-20T02:29:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad8006ab70 con 0x2b42010 2022-07-20T02:29:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 10 client.5826 fetch_fsmap learned FSMap version 24 2022-07-20T02:29:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x2aa5d90 con 0x2b42010 2022-07-20T02:29:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8000fd30 con 0x2b42010 2022-07-20T02:29:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fad8006acd0 con 0x2b42010 2022-07-20T02:29:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 10 client.5826 fetch_fsmap finished waiting for FSMap version 24 2022-07-20T02:29:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 10 client.5826 resolve_mds: resolved ID 'c' to GID 5616 2022-07-20T02:29:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 20 client.5826 populate_metadata read hostname 'smithi026' 2022-07-20T02:29:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b94ac0 0x2b78330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 4 client.5826 mds_command: new command op to 5616 tid=0[{"prefix": "cache drop", "timeout": 5}] 2022-07-20T02:29:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 --> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] -- command(tid 0: {"prefix": "cache drop", "timeout": 5}) v1 -- 0x29945e0 con 0x2b94ac0 2022-07-20T02:29:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad977fe700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b94ac0 0x2b78330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad777fe700 10 client.5826 ms_handle_connect on v2:172.21.15.26:6836/3215220955 2022-07-20T02:29:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:25.120+0000 7fad977fe700 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b94ac0 0x2b78330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=824428880873a07a server_cookie=aca4643086cc85cc in_seq=0 out_seq=0 2022-07-20T02:29:30.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fad777fe700 1 -- 172.21.15.26:0/3827946204 <== mds.0 v2:172.21.15.26:6836/3215220955 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+404 (crc 0 0 0) 0x29945e0 con 0x2b94ac0 2022-07-20T02:29:30.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fad777fe700 10 client.5826 handle_command_reply: tid=0 2022-07-20T02:29:30.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 client.5826 shutdown 2022-07-20T02:29:30.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 msgr2=0x2bdbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fad8001e000 tx=0x7fad8001d000).stop 2022-07-20T02:29:30.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 shutdown_connections 2022-07-20T02:29:30.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:6836/3215220955,v1:172.21.15.26:6837/3215220955] conn(0x2b94ac0 0x2b78330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x2b42010 0x2bdbea0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x2b76a90 0x2b77e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fad777fe700 0 client.5826 ms_handle_reset on v2:172.21.15.26:6836/3215220955 2022-07-20T02:29:30.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 --2- 172.21.15.26:0/3827946204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x2b3e6f0 0x2bdb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 >> 172.21.15.26:0/3827946204 conn(0x2ac88f0 msgr2=0x2b3d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:30.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 shutdown_connections 2022-07-20T02:29:30.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 1 -- 172.21.15.26:0/3827946204 wait complete. 2022-07-20T02:29:30.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.324+0000 7fada8361740 20 client.5826 trim_cache size 0 max 16384 2022-07-20T02:29:30.351 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:29:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_recall": { 2022-07-20T02:29:30.352 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": -110, 2022-07-20T02:29:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "(110) Connection timed out", 2022-07-20T02:29:30.353 INFO:teuthology.orchestra.run.smithi026.stdout: "recalled": 120 2022-07-20T02:29:30.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "flush_journal": { 2022-07-20T02:29:30.355 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:29:30.356 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "" 2022-07-20T02:29:30.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:30.360 INFO:teuthology.orchestra.run.smithi026.stdout: "trim_cache": { 2022-07-20T02:29:30.367 INFO:teuthology.orchestra.run.smithi026.stdout: "trimmed": 0 2022-07-20T02:29:30.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2022-07-20T02:29:30.369 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": { 2022-07-20T02:29:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "items": 19993, 2022-07-20T02:29:30.370 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes": 3099872 2022-07-20T02:29:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:29:30.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:29:30.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2040125399999999 2022-07-20T02:29:30.372 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:29:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 msgr2=0x7fad7c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 0x7fad7c055e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fad88018000 tx=0x7fad88014000).stop 2022-07-20T02:29:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 msgr2=0x7fad981500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fad8000c000 tx=0x7fad80006df0).stop 2022-07-20T02:29:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 shutdown_connections 2022-07-20T02:29:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad980621f0 0x7fad981500f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad7c053950 0x7fad7c055e10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad980ac220 0x7fad98065ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 --2- 172.21.15.26:0/2370908616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad98060250 0x7fad9813f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 >> 172.21.15.26:0/2370908616 conn(0x7fad980b3050 msgr2=0x7fad980b5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 shutdown_connections 2022-07-20T02:29:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.328+0000 7fad9ce9b700 1 -- 172.21.15.26:0/2370908616 wait complete. 2022-07-20T02:29:30.384 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T02:29:30.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:30.386 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:30.387 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:29:30.397 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T02:29:30.397 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T02:29:30.398 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:29:30.398 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:29:30.399 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:30.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:30.473 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:30.491 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:30.503 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:29:30.503 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:29:30.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:29:30.505 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:30.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:29:30.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 Processor -- start 2022-07-20T02:29:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 -- start start 2022-07-20T02:29:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba4147be0 0x7f5ba40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5ba40a7fc0 0x7f5ba40af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba40b62d0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba414aac0 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baad71700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba414ad60 con 0x7f5ba4147be0 2022-07-20T02:29:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5ba40a7fc0 0x7f5ba40af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba9d6f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ba40b5ef0 0x7f5ba41474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37366/0 (socket says 172.21.15.26:37366) 2022-07-20T02:29:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba9d6f700 1 -- 172.21.15.26:0/4000044645 learned_addr learned my addr 172.21.15.26:0/4000044645 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba956e700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba4147be0 0x7f5ba40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 -- 172.21.15.26:0/4000044645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba4147be0 msgr2=0x7f5ba40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba4147be0 0x7f5ba40aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 -- 172.21.15.26:0/4000044645 >> v1:172.21.15.26:6789/0 conn(0x7f5ba40b5ef0 legacy=0x7f5ba41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba8d6d700 1 -- 172.21.15.26:0/4000044645 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631730361 0 0) 0x7f5ba40b62d0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 -- 172.21.15.26:0/4000044645 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba414b000 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5baa570700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40a7fc0 0x7f5ba40af330 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5b9c0032f0 tx=0x7f5b9c0091d0).ready entity=mon.0 client_cookie=54b55cf8cb01a877 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba8d6d700 1 -- 172.21.15.26:0/4000044645 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c00b6c0 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba8d6d700 1 -- 172.21.15.26:0/4000044645 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c00b820 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.680+0000 7f5ba8d6d700 1 -- 172.21.15.26:0/4000044645 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c00b990 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- 172.21.15.26:0/4000044645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40a7fc0 msgr2=0x7f5ba40af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40a7fc0 0x7f5ba40af330 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5b9c0032f0 tx=0x7f5b9c0091d0).stop 2022-07-20T02:29:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- 172.21.15.26:0/4000044645 shutdown_connections 2022-07-20T02:29:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40a7fc0 0x7f5ba40af330 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 --2- 172.21.15.26:0/4000044645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba4147be0 0x7f5ba40aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- 172.21.15.26:0/4000044645 >> 172.21.15.26:0/4000044645 conn(0x7f5ba4060310 msgr2=0x7f5ba40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- 172.21.15.26:0/4000044645 shutdown_connections 2022-07-20T02:29:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- 172.21.15.26:0/4000044645 wait complete. 2022-07-20T02:29:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 Processor -- start 2022-07-20T02:29:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.684+0000 7f5baad71700 1 -- start start 2022-07-20T02:29:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba40a7fc0 0x7f5ba40aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 0x7f5ba40aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40629f0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4062790 con 0x7f5ba4147be0 2022-07-20T02:29:30.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba41565a0 con 0x7f5ba40a7fc0 2022-07-20T02:29:30.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38092/0 (socket says 172.21.15.26:38092) 2022-07-20T02:29:30.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 -- 172.21.15.26:0/3677519456 learned_addr learned my addr 172.21.15.26:0/3677519456 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:30.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba9d6f700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba40a7fc0 0x7f5ba40aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baa570700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 0x7f5ba40aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 -- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba40a7fc0 msgr2=0x7f5ba40aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba40a7fc0 0x7f5ba40aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 -- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 msgr2=0x7f5ba40aaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 0x7f5ba40aaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 -- 172.21.15.26:0/3677519456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c009470 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baa570700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 0x7f5ba40aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:30.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5ba956e700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f5ba000c000 tx=0x7f5ba0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:30.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba0010780 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba00108e0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba4156840 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba0010c40 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.688+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba40ab9d0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba41569d0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5ba001c460 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5b92ffd700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 0x7f5b84055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5ba006c3f0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5ba9d6f700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 0x7f5b84055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.692+0000 7f5ba9d6f700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 0x7f5b84055fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5b9400d000 tx=0x7f5b94009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:30.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.700+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ba0043150 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.828+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba414b2a0 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.828+0000 7f5b92ffd700 1 -- 172.21.15.26:0/3677519456 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16321 (secure 0 0 0) 0x7f5ba0046770 con 0x7f5ba40b5ef0 2022-07-20T02:29:30.832 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:30.833 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:29:07.622520+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:29:05.266754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T02:29:05.807643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:29:30.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 msgr2=0x7f5b84055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 0x7f5b84055fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5b9400d000 tx=0x7f5b94009040).stop 2022-07-20T02:29:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 msgr2=0x7f5ba4156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f5ba000c000 tx=0x7f5ba0006df0).stop 2022-07-20T02:29:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 shutdown_connections 2022-07-20T02:29:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ba40b5ef0 0x7f5ba4156ce0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5b84053b00 0x7f5b84055fc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ba4147be0 0x7f5ba40aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 --2- 172.21.15.26:0/3677519456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ba40a7fc0 0x7f5ba40aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 >> 172.21.15.26:0/3677519456 conn(0x7f5ba4060310 msgr2=0x7f5ba40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 shutdown_connections 2022-07-20T02:29:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.832+0000 7f5baad71700 1 -- 172.21.15.26:0/3677519456 wait complete. 2022-07-20T02:29:30.843 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:29:05.266754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:29:05.807643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:29:30.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:29:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 Processor -- start 2022-07-20T02:29:30.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 -- start start 2022-07-20T02:29:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 0x7f945c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f945c060950 0x7f945c060dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0aa1e0 con 0x7f945c0612b0 2022-07-20T02:29:30.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0ae110 con 0x7f945c060950 2022-07-20T02:29:30.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.956+0000 7f94610e9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0a9f40 con 0x7f945c0add30 2022-07-20T02:29:30.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f945c060950 0x7f945c060dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f945c060950 0x7f945c060dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38100/0 (socket says 172.21.15.26:38100) 2022-07-20T02:29:30.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 -- 172.21.15.26:0/3806490006 learned_addr learned my addr 172.21.15.26:0/3806490006 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:30.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94608e8700 1 --1- 172.21.15.26:0/3806490006 >> v1:172.21.15.120:6789/0 conn(0x7f945c0612b0 0x7f945c143450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51594/0 (socket says 172.21.15.26:51594) 2022-07-20T02:29:30.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945bfff700 1 --2- 172.21.15.26:0/3806490006 >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 0x7f945c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945affd700 1 -- 172.21.15.26:0/3806490006 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739810635 0 0) 0x7f945c0aa1e0 con 0x7f945c0612b0 2022-07-20T02:29:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945affd700 1 -- 172.21.15.26:0/3806490006 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003260 con 0x7f945c0612b0 2022-07-20T02:29:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 -- 172.21.15.26:0/3806490006 >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 msgr2=0x7f945c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 --2- 172.21.15.26:0/3806490006 >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 0x7f945c0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 -- 172.21.15.26:0/3806490006 >> v1:172.21.15.120:6789/0 conn(0x7f945c0612b0 legacy=0x7f945c143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 -- 172.21.15.26:0/3806490006 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0b1110 con 0x7f945c060950 2022-07-20T02:29:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945bfff700 1 --2- 172.21.15.26:0/3806490006 >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 0x7f945c0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945b7fe700 1 --2- 172.21.15.26:0/3806490006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c060dc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f9450002f70 tx=0x7f9450006f40).ready entity=mon.0 client_cookie=843b20c0fadc8df8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945affd700 1 -- 172.21.15.26:0/3806490006 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f945000d9a0 con 0x7f945c060950 2022-07-20T02:29:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945affd700 1 -- 172.21.15.26:0/3806490006 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450000e90 con 0x7f945c060950 2022-07-20T02:29:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f945affd700 1 -- 172.21.15.26:0/3806490006 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f945000dd30 con 0x7f945c060950 2022-07-20T02:29:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 -- 172.21.15.26:0/3806490006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 msgr2=0x7f945c060dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 --2- 172.21.15.26:0/3806490006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c060dc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f9450002f70 tx=0x7f9450006f40).stop 2022-07-20T02:29:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 -- 172.21.15.26:0/3806490006 shutdown_connections 2022-07-20T02:29:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 --2- 172.21.15.26:0/3806490006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c060dc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 --2- 172.21.15.26:0/3806490006 >> v2:172.21.15.120:3300/0 conn(0x7f945c0add30 0x7f945c0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 -- 172.21.15.26:0/3806490006 >> 172.21.15.26:0/3806490006 conn(0x7f945c0b3050 msgr2=0x7f945c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 -- 172.21.15.26:0/3806490006 shutdown_connections 2022-07-20T02:29:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.960+0000 7f94610e9700 1 -- 172.21.15.26:0/3806490006 wait complete. 2022-07-20T02:29:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 Processor -- start 2022-07-20T02:29:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- start start 2022-07-20T02:29:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 0x7f945c0ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f945c0add30 0x7f945c13b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c066250 con 0x7f945c060950 2022-07-20T02:29:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c066480 con 0x7f945c0add30 2022-07-20T02:29:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1564c0 con 0x7f945c0612b0 2022-07-20T02:29:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38104/0 (socket says 172.21.15.26:38104) 2022-07-20T02:29:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 -- 172.21.15.26:0/2992266511 learned_addr learned my addr 172.21.15.26:0/2992266511 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945b7fe700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 0x7f945c0ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94608e8700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f945c0add30 0x7f945c13b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 -- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 msgr2=0x7f945c0ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 0x7f945c0ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 -- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f945c0add30 msgr2=0x7f945c13b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f945c0add30 0x7f945c13b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 -- 172.21.15.26:0/2992266511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c00b180 con 0x7f945c060950 2022-07-20T02:29:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945b7fe700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 0x7f945c0ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f945bfff700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f944c00c000 tx=0x7f944c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f944c010780 con 0x7f945c060950 2022-07-20T02:29:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c0108e0 con 0x7f945c060950 2022-07-20T02:29:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945000b460 con 0x7f945c060950 2022-07-20T02:29:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f944c010c40 con 0x7f945c060950 2022-07-20T02:29:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.964+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c143fd0 con 0x7f945c060950 2022-07-20T02:29:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c156960 con 0x7f945c060950 2022-07-20T02:29:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f944c01c460 con 0x7f945c060950 2022-07-20T02:29:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f94597fa700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 0x7f943c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f944c06c7a0 con 0x7f945c060950 2022-07-20T02:29:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f945b7fe700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 0x7f943c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.968+0000 7f945b7fe700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 0x7f943c055ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9450019000 tx=0x7f945c0ae860).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:30.976+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f944c043370 con 0x7f945c060950 2022-07-20T02:29:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f945c0b13b0 con 0x7f945c060950 2022-07-20T02:29:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94597fa700 1 -- 172.21.15.26:0/2992266511 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v55) v1 ==== 110+0+10 (secure 0 0 0) 0x7f944c046990 con 0x7f945c060950 2022-07-20T02:29:31.109 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:29:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 msgr2=0x7f943c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 0x7f943c055ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9450019000 tx=0x7f945c0ae860).stop 2022-07-20T02:29:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 msgr2=0x7f945c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f944c00c000 tx=0x7f944c006df0).stop 2022-07-20T02:29:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 shutdown_connections 2022-07-20T02:29:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f943c0539e0 0x7f943c055ea0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f945c0add30 0x7f945c13b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f945c0612b0 0x7f945c0ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 --2- 172.21.15.26:0/2992266511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f945c060950 0x7f945c141090 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 >> 172.21.15.26:0/2992266511 conn(0x7f945c0b3050 msgr2=0x7f945c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 shutdown_connections 2022-07-20T02:29:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.100+0000 7f94610e9700 1 -- 172.21.15.26:0/2992266511 wait complete. 2022-07-20T02:29:31.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:29:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.224+0000 7f552991d700 1 Processor -- start 2022-07-20T02:29:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.224+0000 7f552991d700 1 -- start start 2022-07-20T02:29:31.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55240b9c80 0x7f55241431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5524062710 0x7f55240b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55240b6fe0 0x7f55241403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552412bee0 con 0x7f5524062710 2022-07-20T02:29:31.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55240ba060 con 0x7f55240b6fe0 2022-07-20T02:29:31.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5524061f80 con 0x7f55240b9c80 2022-07-20T02:29:31.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5524062710 0x7f55240b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5524062710 0x7f55240b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38112/0 (socket says 172.21.15.26:38112) 2022-07-20T02:29:31.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 -- 172.21.15.26:0/2567072042 learned_addr learned my addr 172.21.15.26:0/2567072042 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55237fe700 1 --2- 172.21.15.26:0/2567072042 >> v2:172.21.15.120:3300/0 conn(0x7f55240b6fe0 0x7f55241403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f5522ffd700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55240b9c80 0x7f55241431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 -- 172.21.15.26:0/2567072042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55240b9c80 msgr2=0x7f55241431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:31.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55240b9c80 0x7f55241431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 -- 172.21.15.26:0/2567072042 >> v2:172.21.15.120:3300/0 conn(0x7f55240b6fe0 msgr2=0x7f55241403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:31.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 --2- 172.21.15.26:0/2567072042 >> v2:172.21.15.120:3300/0 conn(0x7f55240b6fe0 0x7f55241403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 -- 172.21.15.26:0/2567072042 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524062220 con 0x7f5524062710 2022-07-20T02:29:31.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f55227fc700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5524062710 0x7f55240b6af0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f5518002f70 tx=0x7f5518006eb0).ready entity=mon.0 client_cookie=aad482e3206abb09 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f5521ffb700 1 -- 172.21.15.26:0/2567072042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f551800d570 con 0x7f5524062710 2022-07-20T02:29:31.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f5521ffb700 1 -- 172.21.15.26:0/2567072042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518000e90 con 0x7f5524062710 2022-07-20T02:29:31.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f5521ffb700 1 -- 172.21.15.26:0/2567072042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f551800d900 con 0x7f5524062710 2022-07-20T02:29:31.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- 172.21.15.26:0/2567072042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5524062710 msgr2=0x7f55240b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5524062710 0x7f55240b6af0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f5518002f70 tx=0x7f5518006eb0).stop 2022-07-20T02:29:31.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- 172.21.15.26:0/2567072042 shutdown_connections 2022-07-20T02:29:31.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- 172.21.15.26:0/2567072042 >> v2:172.21.15.120:3300/0 conn(0x7f55240b6fe0 0x7f55241403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5524062710 0x7f55240b6af0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 --2- 172.21.15.26:0/2567072042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55240b9c80 0x7f55241431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- 172.21.15.26:0/2567072042 >> 172.21.15.26:0/2567072042 conn(0x7f5524066f70 msgr2=0x7f552405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- 172.21.15.26:0/2567072042 shutdown_connections 2022-07-20T02:29:31.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.228+0000 7f552991d700 1 -- 172.21.15.26:0/2567072042 wait complete. 2022-07-20T02:29:31.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 Processor -- start 2022-07-20T02:29:31.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- start start 2022-07-20T02:29:31.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5524062710 0x7f55240b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55240b6fe0 0x7f5524149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5524146560 con 0x7f55240b9c80 2022-07-20T02:29:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55241470f0 con 0x7f55240b6fe0 2022-07-20T02:29:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5524146100 con 0x7f5524062710 2022-07-20T02:29:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38122/0 (socket says 172.21.15.26:38122) 2022-07-20T02:29:31.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 -- 172.21.15.26:0/3897247660 learned_addr learned my addr 172.21.15.26:0/3897247660 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55227fc700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55240b6fe0 0x7f5524149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f5522ffd700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5524062710 0x7f55240b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 -- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5524062710 msgr2=0x7f55240b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5524062710 0x7f55240b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 -- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55240b6fe0 msgr2=0x7f5524149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55240b6fe0 0x7f5524149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 -- 172.21.15.26:0/3897247660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551800b1e0 con 0x7f55240b9c80 2022-07-20T02:29:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f55237fe700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f551c006e00 tx=0x7f551c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f551c00eeb0 con 0x7f55240b9c80 2022-07-20T02:29:31.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c00c850 con 0x7f55240b9c80 2022-07-20T02:29:31.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55240b3ba0 con 0x7f55240b9c80 2022-07-20T02:29:31.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f551c00cbb0 con 0x7f55240b9c80 2022-07-20T02:29:31.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.232+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55241524b0 con 0x7f55240b9c80 2022-07-20T02:29:31.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f551c00cd10 con 0x7f55240b9c80 2022-07-20T02:29:31.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55240b6650 con 0x7f55240b9c80 2022-07-20T02:29:31.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f5513fff700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 0x7f5500055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f5522ffd700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 0x7f5500055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f551c06c400 con 0x7f55240b9c80 2022-07-20T02:29:31.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.236+0000 7f5522ffd700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 0x7f5500055e10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f551400b000 tx=0x7f5514006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.240+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f551c0430b0 con 0x7f55240b9c80 2022-07-20T02:29:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.364+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f55240b1720 con 0x7f55240b9c80 2022-07-20T02:29:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f5513fff700 1 -- 172.21.15.26:0/3897247660 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7f551c0466d0 con 0x7f55240b9c80 2022-07-20T02:29:31.375 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 msgr2=0x7f5500055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 0x7f5500055e10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f551400b000 tx=0x7f5514006df0).stop 2022-07-20T02:29:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 msgr2=0x7f55240b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f551c006e00 tx=0x7f551c0091d0).stop 2022-07-20T02:29:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 shutdown_connections 2022-07-20T02:29:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55240b9c80 0x7f55240b0ef0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5500053950 0x7f5500055e10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55240b6fe0 0x7f5524149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 --2- 172.21.15.26:0/3897247660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5524062710 0x7f55240b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 >> 172.21.15.26:0/3897247660 conn(0x7f5524066f70 msgr2=0x7f55240b9170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 shutdown_connections 2022-07-20T02:29:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.368+0000 7f552991d700 1 -- 172.21.15.26:0/3897247660 wait complete. 2022-07-20T02:29:31.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:29:31.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 Processor -- start 2022-07-20T02:29:31.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 -- start start 2022-07-20T02:29:31.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f024405cfd0 0x7f02441407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02440b1160 0x7f0244140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 0x7f02441472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02441435b0 con 0x7f02440b1160 2022-07-20T02:29:31.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024405d3b0 con 0x7f0244146e80 2022-07-20T02:29:31.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f024b77b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02440b4bc0 con 0x7f024405cfd0 2022-07-20T02:29:31.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f0248d16700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02440b1160 0x7f0244140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f0248d16700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02440b1160 0x7f0244140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38126/0 (socket says 172.21.15.26:38126) 2022-07-20T02:29:31.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f0248d16700 1 -- 172.21.15.26:0/3237342232 learned_addr learned my addr 172.21.15.26:0/3237342232 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.492+0000 7f0249517700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f024405cfd0 0x7f02441407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0249d18700 1 --2- 172.21.15.26:0/3237342232 >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 0x7f02441472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 -- 172.21.15.26:0/3237342232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f024405cfd0 msgr2=0x7f02441407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:31.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f024405cfd0 0x7f02441407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 -- 172.21.15.26:0/3237342232 >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 msgr2=0x7f02441472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:31.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 --2- 172.21.15.26:0/3237342232 >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 0x7f02441472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 -- 172.21.15.26:0/3237342232 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02440b4e60 con 0x7f02440b1160 2022-07-20T02:29:31.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0249d18700 1 --2- 172.21.15.26:0/3237342232 >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 0x7f02441472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:31.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f0248d16700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02440b1160 0x7f0244140cb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f0240002f70 tx=0x7f0240006eb0).ready entity=mon.0 client_cookie=e2de272b42460e46 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f022ffff700 1 -- 172.21.15.26:0/3237342232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024000d570 con 0x7f02440b1160 2022-07-20T02:29:31.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f022ffff700 1 -- 172.21.15.26:0/3237342232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240000e90 con 0x7f02440b1160 2022-07-20T02:29:31.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f022ffff700 1 -- 172.21.15.26:0/3237342232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024000d900 con 0x7f02440b1160 2022-07-20T02:29:31.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- 172.21.15.26:0/3237342232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02440b1160 msgr2=0x7f0244140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02440b1160 0x7f0244140cb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f0240002f70 tx=0x7f0240006eb0).stop 2022-07-20T02:29:31.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- 172.21.15.26:0/3237342232 shutdown_connections 2022-07-20T02:29:31.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 --2- 172.21.15.26:0/3237342232 >> v2:172.21.15.120:3300/0 conn(0x7f0244146e80 0x7f02441472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02440b1160 0x7f0244140cb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 --2- 172.21.15.26:0/3237342232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f024405cfd0 0x7f02441407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- 172.21.15.26:0/3237342232 >> 172.21.15.26:0/3237342232 conn(0x7f0244066f70 msgr2=0x7f024405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- 172.21.15.26:0/3237342232 shutdown_connections 2022-07-20T02:29:31.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- 172.21.15.26:0/3237342232 wait complete. 2022-07-20T02:29:31.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 Processor -- start 2022-07-20T02:29:31.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.496+0000 7f024b77b700 1 -- start start 2022-07-20T02:29:31.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f024405cfd0 0x7f02441464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02440b1160 0x7f02440af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0244144d10 con 0x7f0244146e80 2022-07-20T02:29:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0244145ba0 con 0x7f024405cfd0 2022-07-20T02:29:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02441448b0 con 0x7f02440b1160 2022-07-20T02:29:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249517700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f024405cfd0 0x7f02441464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38136/0 (socket says 172.21.15.26:38136) 2022-07-20T02:29:31.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 -- 172.21.15.26:0/2477591964 learned_addr learned my addr 172.21.15.26:0/2477591964 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0248d16700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02440b1160 0x7f02440af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 -- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02440b1160 msgr2=0x7f02440af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02440b1160 0x7f02440af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 -- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f024405cfd0 msgr2=0x7f02441464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f024405cfd0 0x7f02441464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 -- 172.21.15.26:0/2477591964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024000b1e0 con 0x7f0244146e80 2022-07-20T02:29:31.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0249d18700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f023c006cc0 tx=0x7f023c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023c013620 con 0x7f0244146e80 2022-07-20T02:29:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c013780 con 0x7f0244146e80 2022-07-20T02:29:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02441450e0 con 0x7f0244146e80 2022-07-20T02:29:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023c013ae0 con 0x7f0244146e80 2022-07-20T02:29:31.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.500+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024413ff20 con 0x7f0244146e80 2022-07-20T02:29:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02440574e0 con 0x7f0244146e80 2022-07-20T02:29:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f023c013c40 con 0x7f0244146e80 2022-07-20T02:29:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f0236ffd700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 0x7f0220055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f023c06c470 con 0x7f0244146e80 2022-07-20T02:29:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f0249517700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 0x7f0220055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.504+0000 7f0249517700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 0x7f0220055e40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f023800c000 tx=0x7f0238008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.508+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f023c043150 con 0x7f0244146e80 2022-07-20T02:29:31.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0244060030 con 0x7f0244146e80 2022-07-20T02:29:31.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f0236ffd700 1 -- 172.21.15.26:0/2477591964 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7f023c046770 con 0x7f0244146e80 2022-07-20T02:29:31.639 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:31.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 msgr2=0x7f0220055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 0x7f0220055e40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f023800c000 tx=0x7f0238008040).stop 2022-07-20T02:29:31.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 msgr2=0x7f02440afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f023c006cc0 tx=0x7f023c008040).stop 2022-07-20T02:29:31.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 shutdown_connections 2022-07-20T02:29:31.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0244146e80 0x7f02440afde0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0220053980 0x7f0220055e40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02440b1160 0x7f02440af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 --2- 172.21.15.26:0/2477591964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f024405cfd0 0x7f02441464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.636+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 >> 172.21.15.26:0/2477591964 conn(0x7f0244066f70 msgr2=0x7f0244143700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.640+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 shutdown_connections 2022-07-20T02:29:31.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.640+0000 7f024b77b700 1 -- 172.21.15.26:0/2477591964 wait complete. 2022-07-20T02:29:31.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:29:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.764+0000 7fd4c3fff700 1 Processor -- start 2022-07-20T02:29:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.764+0000 7fd4c3fff700 1 -- start start 2022-07-20T02:29:31.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 0x7fd4c4147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 0x7fd4c40b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c4061c80 con 0x7fd4c40618a0 2022-07-20T02:29:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c40b2810 con 0x7fd4c4143f80 2022-07-20T02:29:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40b2ab0 con 0x7fd4c40b22d0 2022-07-20T02:29:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c2ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd4c40618a0 0x7fd4c4143870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37420/0 (socket says 172.21.15.26:37420) 2022-07-20T02:29:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c2ffd700 1 -- 172.21.15.26:0/3227668820 learned_addr learned my addr 172.21.15.26:0/3227668820 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c27fc700 1 --2- 172.21.15.26:0/3227668820 >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 0x7fd4c4147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 --2- 172.21.15.26:0/3227668820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 0x7fd4c40b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628176849 0 0) 0x7fd4c4061c80 con 0x7fd4c40618a0 2022-07-20T02:29:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8003260 con 0x7fd4c40618a0 2022-07-20T02:29:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146038863 0 0) 0x7fd4a8003260 con 0x7fd4c40618a0 2022-07-20T02:29:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 -- 172.21.15.26:0/3227668820 >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 msgr2=0x7fd4c4147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 --2- 172.21.15.26:0/3227668820 >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 0x7fd4c4147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 -- 172.21.15.26:0/3227668820 >> v1:172.21.15.26:6789/0 conn(0x7fd4c40618a0 legacy=0x7fd4c4143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 -- 172.21.15.26:0/3227668820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40b2d50 con 0x7fd4c40b22d0 2022-07-20T02:29:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c27fc700 1 --2- 172.21.15.26:0/3227668820 >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 0x7fd4c4147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:29:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c37fe700 1 --2- 172.21.15.26:0/3227668820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 0x7fd4c40b35b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd4b4001800 tx=0x7fd4bc0091d0).ready entity=mon.2 client_cookie=fcdc6688e031941c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4b4001660 con 0x7fd4c40b22d0 2022-07-20T02:29:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc00f510 con 0x7fd4c40b22d0 2022-07-20T02:29:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c1ffb700 1 -- 172.21.15.26:0/3227668820 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4bc00f680 con 0x7fd4c40b22d0 2022-07-20T02:29:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- 172.21.15.26:0/3227668820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 msgr2=0x7fd4c40b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/3227668820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 0x7fd4c40b35b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd4b4001800 tx=0x7fd4bc0091d0).stop 2022-07-20T02:29:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- 172.21.15.26:0/3227668820 shutdown_connections 2022-07-20T02:29:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/3227668820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c40b22d0 0x7fd4c40b35b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/3227668820 >> v2:172.21.15.120:3300/0 conn(0x7fd4c4143f80 0x7fd4c4147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- 172.21.15.26:0/3227668820 >> 172.21.15.26:0/3227668820 conn(0x7fd4c4066f70 msgr2=0x7fd4c4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- 172.21.15.26:0/3227668820 shutdown_connections 2022-07-20T02:29:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.768+0000 7fd4c3fff700 1 -- 172.21.15.26:0/3227668820 wait complete. 2022-07-20T02:29:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 Processor -- start 2022-07-20T02:29:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 -- start start 2022-07-20T02:29:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4c40b22d0 0x7fd4c4144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c4143f80 0x7fd4c4140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c4144d50 con 0x7fd4c40618a0 2022-07-20T02:29:31.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c413ec70 con 0x7fd4c40b22d0 2022-07-20T02:29:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c4140650 con 0x7fd4c4143f80 2022-07-20T02:29:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38146/0 (socket says 172.21.15.26:38146) 2022-07-20T02:29:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 -- 172.21.15.26:0/732040411 learned_addr learned my addr 172.21.15.26:0/732040411 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c27fc700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4c40b22d0 0x7fd4c4144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c37fe700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c4143f80 0x7fd4c4140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 -- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c4143f80 msgr2=0x7fd4c4140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c4143f80 0x7fd4c4140160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 -- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4c40b22d0 msgr2=0x7fd4c4144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4c40b22d0 0x7fd4c4144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 -- 172.21.15.26:0/732040411 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b800b180 con 0x7fd4c40618a0 2022-07-20T02:29:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.772+0000 7fd4c2ffd700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fd4c4140bb0 tx=0x7fd4b4001b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4b4007170 con 0x7fd4c40618a0 2022-07-20T02:29:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4bc009470 con 0x7fd4c40618a0 2022-07-20T02:29:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b40072d0 con 0x7fd4c40618a0 2022-07-20T02:29:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4b4004920 con 0x7fd4c40618a0 2022-07-20T02:29:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c413f160 con 0x7fd4c40618a0 2022-07-20T02:29:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c413ea50 con 0x7fd4c40618a0 2022-07-20T02:29:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd4b4004a80 con 0x7fd4c40618a0 2022-07-20T02:29:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 0x7fd4a0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd4b406bc90 con 0x7fd4c40618a0 2022-07-20T02:29:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4c27fc700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 0x7fd4a0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.776+0000 7fd4c27fc700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 0x7fd4a0055ed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fd4b800b000 tx=0x7fd4b8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.784+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4b40429b0 con 0x7fd4c40618a0 2022-07-20T02:29:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.904+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4c4140a00 con 0x7fd4c40618a0 2022-07-20T02:29:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.904+0000 7fd4b3fff700 1 -- 172.21.15.26:0/732040411 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd4b4045fd0 con 0x7fd4c40618a0 2022-07-20T02:29:31.910 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:29:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 msgr2=0x7fd4a0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 0x7fd4a0055ed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fd4b800b000 tx=0x7fd4b8006df0).stop 2022-07-20T02:29:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 msgr2=0x7fd4c413f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fd4c4140bb0 tx=0x7fd4b4001b40).stop 2022-07-20T02:29:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 shutdown_connections 2022-07-20T02:29:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4a0053a10 0x7fd4a0055ed0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4c4143f80 0x7fd4c4140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4c40b22d0 0x7fd4c4144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 --2- 172.21.15.26:0/732040411 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4c40618a0 0x7fd4c413f600 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 >> 172.21.15.26:0/732040411 conn(0x7fd4c4066f70 msgr2=0x7fd4c40b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 shutdown_connections 2022-07-20T02:29:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:31.908+0000 7fd4c3fff700 1 -- 172.21.15.26:0/732040411 wait complete. 2022-07-20T02:29:31.922 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:29:31.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:29:32.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 Processor -- start 2022-07-20T02:29:32.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 -- start start 2022-07-20T02:29:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a1405cfd0 0x7f8a14143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 0x7f8a140b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a141474e0 con 0x7f8a14060f00 2022-07-20T02:29:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1405d3b0 con 0x7f8a1405cfd0 2022-07-20T02:29:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a1987a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a141431e0 con 0x7f8a14147000 2022-07-20T02:29:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a19079700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8a14060f00 0x7f8a1413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37436/0 (socket says 172.21.15.26:37436) 2022-07-20T02:29:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a1405cfd0 0x7f8a14143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a19079700 1 -- 172.21.15.26:0/2176395831 learned_addr learned my addr 172.21.15.26:0/2176395831 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:29:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a13fff700 1 --2- 172.21.15.26:0/2176395831 >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 0x7f8a140b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a137fe700 1 -- 172.21.15.26:0/2176395831 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031084181 0 0) 0x7f8a141474e0 con 0x7f8a14060f00 2022-07-20T02:29:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a137fe700 1 -- 172.21.15.26:0/2176395831 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003260 con 0x7f8a14060f00 2022-07-20T02:29:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 -- 172.21.15.26:0/2176395831 >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 msgr2=0x7f8a140b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:29:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 --2- 172.21.15.26:0/2176395831 >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 0x7f8a140b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 -- 172.21.15.26:0/2176395831 >> v1:172.21.15.26:6789/0 conn(0x7f8a14060f00 legacy=0x7f8a1413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 -- 172.21.15.26:0/2176395831 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a14143480 con 0x7f8a1405cfd0 2022-07-20T02:29:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a13fff700 1 --2- 172.21.15.26:0/2176395831 >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 0x7f8a140b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:29:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a18878700 1 --2- 172.21.15.26:0/2176395831 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a1405cfd0 0x7f8a14143bc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f8a0c0032f0 tx=0x7f8a0c0091d0).ready entity=mon.0 client_cookie=e68d37c1685de10f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a137fe700 1 -- 172.21.15.26:0/2176395831 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c00b680 con 0x7f8a1405cfd0 2022-07-20T02:29:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a137fe700 1 -- 172.21.15.26:0/2176395831 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c00b7e0 con 0x7f8a1405cfd0 2022-07-20T02:29:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.028+0000 7f8a137fe700 1 -- 172.21.15.26:0/2176395831 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c00b950 con 0x7f8a1405cfd0 2022-07-20T02:29:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- 172.21.15.26:0/2176395831 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a1405cfd0 msgr2=0x7f8a14143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 --2- 172.21.15.26:0/2176395831 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a1405cfd0 0x7f8a14143bc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f8a0c0032f0 tx=0x7f8a0c0091d0).stop 2022-07-20T02:29:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- 172.21.15.26:0/2176395831 shutdown_connections 2022-07-20T02:29:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 --2- 172.21.15.26:0/2176395831 >> v2:172.21.15.120:3300/0 conn(0x7f8a14147000 0x7f8a140b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 --2- 172.21.15.26:0/2176395831 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a1405cfd0 0x7f8a14143bc0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- 172.21.15.26:0/2176395831 >> 172.21.15.26:0/2176395831 conn(0x7f8a140620c0 msgr2=0x7f8a1405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- 172.21.15.26:0/2176395831 shutdown_connections 2022-07-20T02:29:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- 172.21.15.26:0/2176395831 wait complete. 2022-07-20T02:29:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 Processor -- start 2022-07-20T02:29:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.032+0000 7f8a1987a700 1 -- start start 2022-07-20T02:29:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a14147000 0x7f8a14142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a14141360 0x7f8a14137170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14141960 con 0x7f8a14060f00 2022-07-20T02:29:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14156190 con 0x7f8a14141360 2022-07-20T02:29:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14156430 con 0x7f8a14147000 2022-07-20T02:29:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38160/0 (socket says 172.21.15.26:38160) 2022-07-20T02:29:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 -- 172.21.15.26:0/1659150704 learned_addr learned my addr 172.21.15.26:0/1659150704 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:29:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a13fff700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a14147000 0x7f8a14142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a19079700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a14141360 0x7f8a14137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 -- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a14147000 msgr2=0x7f8a14142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a14147000 0x7f8a14142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 -- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a14141360 msgr2=0x7f8a14137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a14141360 0x7f8a14137170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 -- 172.21.15.26:0/1659150704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c009470 con 0x7f8a14060f00 2022-07-20T02:29:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a18878700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f8a0c0032f0 tx=0x7f8a0c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c006ea0 con 0x7f8a14060f00 2022-07-20T02:29:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c011d70 con 0x7f8a14060f00 2022-07-20T02:29:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a140b1c10 con 0x7f8a14060f00 2022-07-20T02:29:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c021670 con 0x7f8a14060f00 2022-07-20T02:29:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.036+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a14141d90 con 0x7f8a14060f00 2022-07-20T02:29:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a0c00f070 con 0x7f8a14060f00 2022-07-20T02:29:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a14143720 con 0x7f8a14060f00 2022-07-20T02:29:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a11ffb700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 0x7f89f4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:29:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8a0c06ccc0 con 0x7f8a14060f00 2022-07-20T02:29:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a13fff700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 0x7f89f4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:29:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.040+0000 7f8a13fff700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 0x7f89f4055f30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8a0800c000 tx=0x7f8a08008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:29:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.044+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a0c0439e0 con 0x7f8a14060f00 2022-07-20T02:29:32.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.204+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a140574e0 con 0x7f8a14060f00 2022-07-20T02:29:32.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a11ffb700 1 -- 172.21.15.26:0/1659150704 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f8a0c015070 con 0x7f8a14060f00 2022-07-20T02:29:32.212 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:29:32.212 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:29:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 msgr2=0x7f89f4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 0x7f89f4055f30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8a0800c000 tx=0x7f8a08008040).stop 2022-07-20T02:29:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 msgr2=0x7f8a140b8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:29:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f8a0c0032f0 tx=0x7f8a0c016000).stop 2022-07-20T02:29:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 shutdown_connections 2022-07-20T02:29:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89f4053a70 0x7f89f4055f30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a14141360 0x7f8a14137170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a14147000 0x7f8a14142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 --2- 172.21.15.26:0/1659150704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a14060f00 0x7f8a140b8c60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:29:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 >> 172.21.15.26:0/1659150704 conn(0x7f8a1405cfd0 msgr2=0x7f8a14138fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:29:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 shutdown_connections 2022-07-20T02:29:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:29:32.208+0000 7f8a1987a700 1 -- 172.21.15.26:0/1659150704 wait complete. 2022-07-20T02:29:32.236 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:29:32.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:29:06.619736+0000', 'modified': '2022-07-20T02:29:07.623517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5616}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5616': {'gid': 5616, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3215220955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3215220955}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3215220955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:29:32.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:29:32.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:29:32.664 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5616,"want_state":"up:active","state":"up:active","rank_uptime":26.032748304999998,"mdsmap_epoch":24,"osdmap_epoch":55,"osdmap_epoch_barrier":55,"uptime":30.328168161000001} 2022-07-20T02:29:32.665 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5616, "mdsmap_epoch": 24, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "rank_uptime": 26.032748305, "state": "up:active", "uptime": 30.328168161, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:29:32.666 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:29:32.666 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:29:32.667 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:29:32.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:32.669 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:32.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:29:32.745 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:29:32.746 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:29:32.768 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:29:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:29:32.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:32.772 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:29:32.824 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:29:32.825 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:32.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:29:32.879 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:29:32.880 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:29:32.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:29:32.900+0000 7f7617ac4080 -1 init, newargv = 0x5579a741f850 newargc=9 2022-07-20T02:29:32.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22643]: starting ceph client 2022-07-20T02:29:32.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22643]: starting fuse 2022-07-20T02:29:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:29:32.933 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:29:32.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:32.935 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:29:33.007 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:29:33.007 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:29:33.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:33.009 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:29:33.010 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:29:33.011 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:29:33.011 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:29:33.012 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:29:33.013 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:33.015 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:29:33.016 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:29:33.016 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:29:33.017 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:33.018 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:29:33.020 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:29:33.021 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:29:33.022 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:29:33.023 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:33.023 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:29:33.025 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:29:33.026 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:29:33.027 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:29:33.027 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:29:33.028 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:29:33.029 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:29:33.030 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:29:33.087 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22643.asok 2022-07-20T02:29:33.093 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22643.asok 2022-07-20T02:29:33.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:33.094 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22643.asok status 2022-07-20T02:29:33.231 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:29:33.232 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:29:33.233 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:29:33.233 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:29:33.234 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:29:33.236 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:29:33.236 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:29:33.237 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22643", 2022-07-20T02:29:33.237 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:29:33.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:33.239 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:29:33.239 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:29:33.240 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 5874, 2022-07-20T02:29:33.240 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:29:33.241 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:29:33.241 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:29:33.245 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 5874 2022-07-20T02:29:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:33.246 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:29:33.248 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:29:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:29:33.249 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 539028249 2022-07-20T02:29:33.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:29:33.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:33.251 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:29:33.252 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:29:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:29:33.253 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 539028249 2022-07-20T02:29:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:29:33.254 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.5874 172.21.15.159:0/539028249", 2022-07-20T02:29:33.263 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/539028249", 2022-07-20T02:29:33.263 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:29:33.264 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 24, 2022-07-20T02:29:33.264 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 55, 2022-07-20T02:29:33.265 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:29:33.265 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:29:33.266 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:29:33.267 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:29:33.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:33.270 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:29:33.271 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:33.283 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:29:33.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:29:33.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:29:33.287 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:29:46.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:46.734 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:46.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:29:46.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:16.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:16.810 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:16.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:16.818 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:27.340 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:30:27.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:27.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:27.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:27.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:27.486 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:30:27.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:27.488 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:27.488 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:27.513 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:30:27.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:30:27.516 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:30:27.517 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:30:27.518 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:27.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22643]: fuse finished with error 0 and tester_r 0 2022-07-20T02:30:27.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:27.540 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:27.541 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:27.631 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:30:27.632 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:30:27.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:30:27.633 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:27.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:27.644 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:27.645 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:30:27.652 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:30:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:30:27.654 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:30:27.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:30:27.655 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:30:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:30:27.716 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:30:27.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:30:27.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 Processor -- start 2022-07-20T02:30:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 -- start start 2022-07-20T02:30:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc44060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc44095b10 0x7fdc44095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc44067dc0 con 0x7fdc440b3260 2022-07-20T02:30:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc440b3640 con 0x7fdc44095b10 2022-07-20T02:30:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc48a0a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440615c0 con 0x7fdc440a95a0 2022-07-20T02:30:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc44095b10 0x7fdc44095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc42d9d700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdc440b3260 0x7fdc440676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51660/0 (socket says 172.21.15.26:51660) 2022-07-20T02:30:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc42d9d700 1 -- 172.21.15.26:0/292011909 learned_addr learned my addr 172.21.15.26:0/292011909 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc3a59c700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc44060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 -- 172.21.15.26:0/292011909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 msgr2=0x7fdc44060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc44060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:27.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4259c700 1 -- 172.21.15.26:0/292011909 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052865191 0 0) 0x7fdc44067dc0 con 0x7fdc440b3260 2022-07-20T02:30:27.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 -- 172.21.15.26:0/292011909 >> v1:172.21.15.120:6789/0 conn(0x7fdc440b3260 legacy=0x7fdc440676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:27.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 -- 172.21.15.26:0/292011909 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44061860 con 0x7fdc44095b10 2022-07-20T02:30:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4359e700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc44095b10 0x7fdc44095f50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fdc3c0032f0 tx=0x7fdc3c0091d0).ready entity=mon.0 client_cookie=fd155db197230131 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4259c700 1 -- 172.21.15.26:0/292011909 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c00b6c0 con 0x7fdc44095b10 2022-07-20T02:30:27.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4259c700 1 -- 172.21.15.26:0/292011909 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c00b820 con 0x7fdc44095b10 2022-07-20T02:30:27.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.832+0000 7fdc4259c700 1 -- 172.21.15.26:0/292011909 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c00b990 con 0x7fdc44095b10 2022-07-20T02:30:27.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- 172.21.15.26:0/292011909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc44095b10 msgr2=0x7fdc44095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:27.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc44095b10 0x7fdc44095f50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fdc3c0032f0 tx=0x7fdc3c0091d0).stop 2022-07-20T02:30:27.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- 172.21.15.26:0/292011909 shutdown_connections 2022-07-20T02:30:27.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc44095b10 0x7fdc44095f50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:27.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/292011909 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc44060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:27.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- 172.21.15.26:0/292011909 >> 172.21.15.26:0/292011909 conn(0x7fdc440b67f0 msgr2=0x7fdc440b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:27.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- 172.21.15.26:0/292011909 shutdown_connections 2022-07-20T02:30:27.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- 172.21.15.26:0/292011909 wait complete. 2022-07-20T02:30:27.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 Processor -- start 2022-07-20T02:30:27.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.836+0000 7fdc48a0a700 1 -- start start 2022-07-20T02:30:27.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc44095b10 0x7fdc44143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc441557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc44140ca0 con 0x7fdc440b3260 2022-07-20T02:30:27.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc44142cb0 con 0x7fdc44095b10 2022-07-20T02:30:27.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc44152d10 con 0x7fdc440a95a0 2022-07-20T02:30:27.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38178/0 (socket says 172.21.15.26:38178) 2022-07-20T02:30:27.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 -- 172.21.15.26:0/1732386038 learned_addr learned my addr 172.21.15.26:0/1732386038 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:27.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc42d9d700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc44095b10 0x7fdc44143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc3a59c700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc441557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 -- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 msgr2=0x7fdc441557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:27.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc441557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:27.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 -- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc44095b10 msgr2=0x7fdc44143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:27.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc44095b10 0x7fdc44143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:27.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 -- 172.21.15.26:0/1732386038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c009470 con 0x7fdc440b3260 2022-07-20T02:30:27.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc4359e700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fdc3c0032f0 tx=0x7fdc3c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:27.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c00b6c0 con 0x7fdc440b3260 2022-07-20T02:30:27.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c021630 con 0x7fdc440b3260 2022-07-20T02:30:27.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc44152fb0 con 0x7fdc440b3260 2022-07-20T02:30:27.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c021990 con 0x7fdc440b3260 2022-07-20T02:30:27.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc441438f0 con 0x7fdc440b3260 2022-07-20T02:30:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc44061b00 con 0x7fdc440b3260 2022-07-20T02:30:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.840+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc3c027020 con 0x7fdc440b3260 2022-07-20T02:30:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.844+0000 7fdc40d99700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 0x7fdc28055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.844+0000 7fdc42d9d700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 0x7fdc28055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.844+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdc3c06d580 con 0x7fdc440b3260 2022-07-20T02:30:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.844+0000 7fdc42d9d700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 0x7fdc28055ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fdc3400d000 tx=0x7fdc34009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.848+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc3c044230 con 0x7fdc440b3260 2022-07-20T02:30:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.972+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdc440574e0 con 0x7fdc440b3260 2022-07-20T02:30:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.972+0000 7fdc40d99700 1 -- 172.21.15.26:0/1732386038 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v55) v1 ==== 81+0+34087 (secure 0 0 0) 0x7fdc3c015090 con 0x7fdc440b3260 2022-07-20T02:30:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:27.975 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:30:27.975 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 55, 2022-07-20T02:30:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:30:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:30:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:29:07.622520+0000", 2022-07-20T02:30:27.976 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:30:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:30:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:30:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:30:27.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:30:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:30:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:30:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:30:27.978 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:30:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:30:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:30:27.979 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:30:27.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:30:27.980 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:30:27.980 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 10, 2022-07-20T02:30:27.980 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:30:27.980 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:30:27.981 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:30:27.981 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:30:27.981 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:30:27.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:27.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:30:27.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:30:27.982 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:30:27.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:30:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:30:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:30:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:30:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:30:27.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:27.991 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:27.991 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:27.992 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:27.992 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:27.992 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:27.993 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:27.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:27.993 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:30:27.994 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:27.994 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:27.994 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:27.994 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:27.996 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:27.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:27.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:28.002 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:28.002 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:28.003 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:28.003 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:28.003 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:28.004 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:28.004 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:28.004 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:28.004 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:28.004 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:28.005 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:28.005 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:28.005 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:28.005 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:28.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:28.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.006 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:28.006 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:28.006 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:28.007 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:28.007 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:28.007 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:28.007 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:28.008 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:30:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:30:28.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:30:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:30:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:30:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:30:28.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:28.021 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:28.022 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:30:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:30:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:30:28.032 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:30:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:30:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:28.033 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:28.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:30:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:28.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:28.036 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:28.037 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:28.038 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:28.041 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:28.042 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:30:28.043 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:30:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 9, 2022-07-20T02:30:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:30:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:29:05.266754+0000", 2022-07-20T02:30:28.045 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:28.064 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:30:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:30:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:30:28.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:30:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:30:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:28.086 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:28.087 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "54", 2022-07-20T02:30:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:28.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:28.089 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:28.090 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:28.091 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:28.092 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:28.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:28.094 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:28.096 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:28.097 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:28.098 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:28.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:28.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.101 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:28.102 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:28.103 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:28.103 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:28.104 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:28.104 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:28.104 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:28.105 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:28.105 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:28.106 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:28.108 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:30:28.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:30:28.110 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:30:28.111 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:30:28.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.111 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:28.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:30:28.112 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:30:28.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.116 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 10, 2022-07-20T02:30:28.116 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:30:28.117 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:29:05.807643+0000", 2022-07-20T02:30:28.118 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:28.118 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:28.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:28.119 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:28.120 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:28.120 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:28.120 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:28.121 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:28.121 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:30:28.122 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:30:28.122 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:30:28.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:30:28.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:30:28.124 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:28.124 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:28.125 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:28.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:28.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:28.126 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:28.126 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:28.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.128 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "55", 2022-07-20T02:30:28.128 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:28.129 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:28.129 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:28.130 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:28.130 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:28.131 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:28.131 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:28.132 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:28.132 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:28.133 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:28.133 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:28.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:28.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:28.139 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:28.139 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:28.139 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:28.140 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:28.140 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:28.141 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:28.141 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:28.142 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:28.142 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:28.143 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:28.143 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:28.144 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:28.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:28.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.145 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:28.145 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:28.146 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:28.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:28.147 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:28.147 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:28.149 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:28.150 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:28.150 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:28.151 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:28.151 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:28.152 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:30:28.152 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:28.153 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:30:28.153 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:30:28.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:30:28.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.156 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:30:28.157 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:30:28.157 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.158 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.158 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.159 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.159 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.161 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.161 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:28.161 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.162 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:30:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:30:28.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.169 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:30:28.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:30:28.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.176 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:30:28.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:30:28.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.187 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:30:28.190 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:30:28.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:28.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.207 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:30:28.207 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:30:28.207 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:30:28.208 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:30:28.208 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.208 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.208 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.209 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:30:28.210 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:30:28.210 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.210 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.211 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.211 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.211 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:28.213 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.213 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.213 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:30:28.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:30:28.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:30:28.218 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:30:28.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.221 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:30:28.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:30:28.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.225 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:30:28.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:30:28.229 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:28.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.230 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:30:28.230 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:30:28.230 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:30:28.230 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:30:28.231 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.231 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.231 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.232 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:30:28.232 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:30:28.232 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.233 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.233 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.234 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.234 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.234 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.234 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.235 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 53, 2022-07-20T02:30:28.235 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.235 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:30:28.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:30:28.238 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.239 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:30:28.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:30:28.241 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.242 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:30:28.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:30:28.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.246 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:30:28.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:30:28.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:28.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:30:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:30:28.250 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:30:28.251 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:30:28.251 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.251 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.251 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.252 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:30:28.252 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:30:28.253 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.253 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.253 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.254 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.254 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.255 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.255 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:30:28.255 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.256 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:30:28.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:30:28.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.260 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:30:28.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:30:28.262 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.264 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.265 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:30:28.266 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:30:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:30:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:30:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:28.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.271 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:30:28.272 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:30:28.272 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:30:28.272 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:30:28.272 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.273 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.273 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.274 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:30:28.274 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:30:28.274 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.275 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.275 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.276 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.276 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.276 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.276 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.277 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:28.277 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.277 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.277 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:30:28.278 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:30:28.280 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.280 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.281 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:30:28.282 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:30:28.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.284 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:30:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:30:28.287 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.288 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:30:28.289 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.290 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:30:28.290 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:28.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.291 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:30:28.291 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:30:28.292 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:30:28.292 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:30:28.292 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.292 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.293 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.294 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:30:28.294 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:30:28.294 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.294 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.295 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.295 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.295 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.295 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.296 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.296 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:28.297 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.297 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.297 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.297 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.298 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:30:28.298 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:30:28.300 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.301 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:30:28.302 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:30:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:30:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:30:28.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.308 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:30:28.310 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.310 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:30:28.311 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:28.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:30:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:30:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:30:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:30:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.313 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.313 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.315 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:30:28.315 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:30:28.316 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.317 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.318 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.319 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.319 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.320 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.320 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.321 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:28.321 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.321 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.322 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:30:28.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:30:28.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.330 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:30:28.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:30:28.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.335 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:30:28.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:30:28.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:30:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:30:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:30:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:30:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:30:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:30:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.375 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:30:28.375 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:30:28.376 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:28.376 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:28.377 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:28.377 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:28.378 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:28.378 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:28.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:28.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:30:28.379 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:28.380 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:28.381 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:28.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:30:28.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:30:28.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.387 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:28.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:30:28.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:30:28.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.394 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.395 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:28.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:30:28.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:30:28.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.401 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:28.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:28.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:28.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:30:28.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:28.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:30:28.406 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:28.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.408 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:30:28.409 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:30:28.409 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:30:28.410 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:30:28.410 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:28.411 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:28.411 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:28.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:28.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.412 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:28.413 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:30:28.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.414 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:30:28.414 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.415 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.415 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.416 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.416 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:30:28.418 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.419 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:30:28.419 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.420 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.420 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.421 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.421 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.422 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:30:28.422 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.423 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:30:28.424 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.425 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.425 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.425 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.426 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.426 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:30:28.427 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.429 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:30:28.429 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.430 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.430 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.431 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.431 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.432 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:30:28.432 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.434 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:30:28.434 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.434 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.435 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.435 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.436 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.436 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:30:28.437 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.439 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:30:28.439 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.439 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.440 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.440 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.441 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.441 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:30:28.442 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.443 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:30:28.444 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.444 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.444 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.445 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.445 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.446 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:30:28.446 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:28.448 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:30:28.448 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:28.452 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:28.452 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:28.452 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:28.452 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:28.453 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:30:28.453 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:28.453 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.453 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:28.454 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:30:28.454 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:30:28.454 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:30:28.454 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:30:28.454 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:30:28.455 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:30:28.455 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:30:28.455 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:30:28.455 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:30:28.456 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:30:28.456 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:30:28.456 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:30:28.456 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:30:28.457 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:30:28.457 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:30:28.457 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:30:28.457 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:30:28.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.458 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:30:28.458 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:30:28.458 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:30:28.458 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:30:28.459 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:30:28.459 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:30:28.459 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:30:28.459 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:28.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:28.460 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:30:28.460 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:30:28.460 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:30:28.460 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:30:28.460 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:30:28.461 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:30:28.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 msgr2=0x7fdc28055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 0x7fdc28055ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fdc3400d000 tx=0x7fdc34009040).stop 2022-07-20T02:30:28.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 msgr2=0x7fdc440ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fdc3c0032f0 tx=0x7fdc3c016000).stop 2022-07-20T02:30:28.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 shutdown_connections 2022-07-20T02:30:28.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc440b3260 0x7fdc440ada60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc280539e0 0x7fdc28055ea0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc440a95a0 0x7fdc441557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.976+0000 7fdc48a0a700 1 --2- 172.21.15.26:0/1732386038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc44095b10 0x7fdc44143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.980+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 >> 172.21.15.26:0/1732386038 conn(0x7fdc440b67f0 msgr2=0x7fdc440b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:28.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.980+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 shutdown_connections 2022-07-20T02:30:28.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:27.980+0000 7fdc48a0a700 1 -- 172.21.15.26:0/1732386038 wait complete. 2022-07-20T02:30:28.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:28.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 Processor -- start 2022-07-20T02:30:28.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 -- start start 2022-07-20T02:30:28.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b7d80 0x7f62dc0aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62dc0b4b80 0x7f62dc14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62dc0608d0 0x7f62dc060d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc0b4fc0 con 0x7f62dc0b4b80 2022-07-20T02:30:28.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc064650 con 0x7f62dc0608d0 2022-07-20T02:30:28.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e313c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0b41a0 con 0x7f62dc0b7d80 2022-07-20T02:30:28.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.580+0000 7f62e1939700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62dc0b4b80 0x7f62dc14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62dc0b4b80 0x7f62dc14ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38184/0 (socket says 172.21.15.26:38184) 2022-07-20T02:30:28.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 -- 172.21.15.26:0/989937011 learned_addr learned my addr 172.21.15.26:0/989937011 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:28.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e213a700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b7d80 0x7f62dc0aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e293b700 1 --2- 172.21.15.26:0/989937011 >> v2:172.21.15.120:3300/0 conn(0x7f62dc0608d0 0x7f62dc060d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 -- 172.21.15.26:0/989937011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b7d80 msgr2=0x7f62dc0aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b7d80 0x7f62dc0aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 -- 172.21.15.26:0/989937011 >> v2:172.21.15.120:3300/0 conn(0x7f62dc0608d0 msgr2=0x7f62dc060d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 --2- 172.21.15.26:0/989937011 >> v2:172.21.15.120:3300/0 conn(0x7f62dc0608d0 0x7f62dc060d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 -- 172.21.15.26:0/989937011 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc0b4440 con 0x7f62dc0b4b80 2022-07-20T02:30:28.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1939700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b4b80 0x7f62dc14ad60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f62d0002f70 tx=0x7f62d0006eb0).ready entity=mon.0 client_cookie=85102c713d6cffb2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1138700 1 -- 172.21.15.26:0/989937011 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d000d570 con 0x7f62dc0b4b80 2022-07-20T02:30:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1138700 1 -- 172.21.15.26:0/989937011 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0000e90 con 0x7f62dc0b4b80 2022-07-20T02:30:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e1138700 1 -- 172.21.15.26:0/989937011 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d000d900 con 0x7f62dc0b4b80 2022-07-20T02:30:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- 172.21.15.26:0/989937011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b4b80 msgr2=0x7f62dc14ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b4b80 0x7f62dc14ad60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f62d0002f70 tx=0x7f62d0006eb0).stop 2022-07-20T02:30:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- 172.21.15.26:0/989937011 shutdown_connections 2022-07-20T02:30:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 --2- 172.21.15.26:0/989937011 >> v2:172.21.15.120:3300/0 conn(0x7f62dc0608d0 0x7f62dc060d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b4b80 0x7f62dc14ad60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 --2- 172.21.15.26:0/989937011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b7d80 0x7f62dc0aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- 172.21.15.26:0/989937011 >> 172.21.15.26:0/989937011 conn(0x7f62dc05f840 msgr2=0x7f62dc05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- 172.21.15.26:0/989937011 shutdown_connections 2022-07-20T02:30:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- 172.21.15.26:0/989937011 wait complete. 2022-07-20T02:30:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 Processor -- start 2022-07-20T02:30:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.584+0000 7f62e313c700 1 -- start start 2022-07-20T02:30:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62dc0608d0 0x7f62dc0b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b4b80 0x7f62dc14bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc156540 con 0x7f62dc0b7d80 2022-07-20T02:30:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0b62d0 con 0x7f62dc0608d0 2022-07-20T02:30:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0b67d0 con 0x7f62dc0b4b80 2022-07-20T02:30:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38192/0 (socket says 172.21.15.26:38192) 2022-07-20T02:30:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 -- 172.21.15.26:0/749461603 learned_addr learned my addr 172.21.15.26:0/749461603 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e213a700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62dc0608d0 0x7f62dc0b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e1939700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b4b80 0x7f62dc14bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 -- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b4b80 msgr2=0x7f62dc14bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b4b80 0x7f62dc14bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 -- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62dc0608d0 msgr2=0x7f62dc0b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62dc0608d0 0x7f62dc0b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 -- 172.21.15.26:0/749461603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d000b1e0 con 0x7f62dc0b7d80 2022-07-20T02:30:28.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e293b700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f62dc0650a0 tx=0x7f62d4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d4010780 con 0x7f62dc0b7d80 2022-07-20T02:30:28.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d40108e0 con 0x7f62dc0b7d80 2022-07-20T02:30:28.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc0b5750 con 0x7f62dc0b7d80 2022-07-20T02:30:28.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d4010c40 con 0x7f62dc0b7d80 2022-07-20T02:30:28.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.588+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc156730 con 0x7f62dc0b7d80 2022-07-20T02:30:28.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc0b46e0 con 0x7f62dc0b7d80 2022-07-20T02:30:28.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f62d401c460 con 0x7f62dc0b7d80 2022-07-20T02:30:28.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62db7fe700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 0x7f62bc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62e213a700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 0x7f62bc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f62d406c300 con 0x7f62dc0b7d80 2022-07-20T02:30:28.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.592+0000 7f62e213a700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 0x7f62bc055e10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f62cc00b000 tx=0x7f62cc0091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.600+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62d4043060 con 0x7f62dc0b7d80 2022-07-20T02:30:28.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.756+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc156b70 con 0x7f62dc0b7d80 2022-07-20T02:30:28.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62db7fe700 1 -- 172.21.15.26:0/749461603 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f62d4046680 con 0x7f62dc0b7d80 2022-07-20T02:30:28.763 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:28.764 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:29:07.623517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5616":{"gid":5616,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3215220955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3215220955},{"type":"v1","addr":"172.21.15.26:6837","nonce":3215220955}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:30:28.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 msgr2=0x7f62bc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 0x7f62bc055e10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f62cc00b000 tx=0x7f62cc0091d0).stop 2022-07-20T02:30:28.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 msgr2=0x7f62dc0b5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f62dc0650a0 tx=0x7f62d4006df0).stop 2022-07-20T02:30:28.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 shutdown_connections 2022-07-20T02:30:28.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62dc0b7d80 0x7f62dc0b5260 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f62bc053950 0x7f62bc055e10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62dc0b4b80 0x7f62dc14bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 --2- 172.21.15.26:0/749461603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62dc0608d0 0x7f62dc0b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 >> 172.21.15.26:0/749461603 conn(0x7f62dc05f840 msgr2=0x7f62dc1495a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:28.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 shutdown_connections 2022-07-20T02:30:28.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.760+0000 7f62e313c700 1 -- 172.21.15.26:0/749461603 wait complete. 2022-07-20T02:30:28.773 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2022-07-20T02:30:28.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:30:28.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 Processor -- start 2022-07-20T02:30:28.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 -- start start 2022-07-20T02:30:28.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77e80b2b30 0x7f77e80a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e80b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e80b2f10 con 0x7f77e8060f10 2022-07-20T02:30:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80a8e30 con 0x7f77e80b2b30 2022-07-20T02:30:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77eec0b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80a7310 con 0x7f77e8147d70 2022-07-20T02:30:28.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f77e8060f10 0x7f77e8147660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37476/0 (socket says 172.21.15.26:37476) 2022-07-20T02:30:28.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e7fff700 1 -- 172.21.15.26:0/923667203 learned_addr learned my addr 172.21.15.26:0/923667203 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:28.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/923667203 >> v2:172.21.15.26:3300/0 conn(0x7f77e80b2b30 0x7f77e80a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ed1a8700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e80b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e77fe700 1 -- 172.21.15.26:0/923667203 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566182080 0 0) 0x7f77e80b2f10 con 0x7f77e8060f10 2022-07-20T02:30:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e77fe700 1 -- 172.21.15.26:0/923667203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4003260 con 0x7f77e8060f10 2022-07-20T02:30:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 -- 172.21.15.26:0/923667203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 msgr2=0x7f77e80b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e80b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 -- 172.21.15.26:0/923667203 >> v1:172.21.15.26:6789/0 conn(0x7f77e8060f10 legacy=0x7f77e8147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 -- 172.21.15.26:0/923667203 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80a7500 con 0x7f77e80b2b30 2022-07-20T02:30:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80b2b30 0x7f77e80a7ee0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f77dc002f70 tx=0x7f77dc006df0).ready entity=mon.0 client_cookie=b8bf2797b88704a2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e77fe700 1 -- 172.21.15.26:0/923667203 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc00d4a0 con 0x7f77e80b2b30 2022-07-20T02:30:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e77fe700 1 -- 172.21.15.26:0/923667203 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc000e90 con 0x7f77e80b2b30 2022-07-20T02:30:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.900+0000 7f77e77fe700 1 -- 172.21.15.26:0/923667203 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc00d7f0 con 0x7f77e80b2b30 2022-07-20T02:30:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- 172.21.15.26:0/923667203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80b2b30 msgr2=0x7f77e80a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80b2b30 0x7f77e80a7ee0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f77dc002f70 tx=0x7f77dc006df0).stop 2022-07-20T02:30:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- 172.21.15.26:0/923667203 shutdown_connections 2022-07-20T02:30:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e80b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- 172.21.15.26:0/923667203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e80b2b30 0x7f77e80a7ee0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- 172.21.15.26:0/923667203 >> 172.21.15.26:0/923667203 conn(0x7f77e80b6110 msgr2=0x7f77e80b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- 172.21.15.26:0/923667203 shutdown_connections 2022-07-20T02:30:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- 172.21.15.26:0/923667203 wait complete. 2022-07-20T02:30:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 Processor -- start 2022-07-20T02:30:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- start start 2022-07-20T02:30:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 0x7f77e8064fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e8064180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e81572a0 con 0x7f77e8060f10 2022-07-20T02:30:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8149fb0 con 0x7f77e80b2b30 2022-07-20T02:30:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.904+0000 7f77eec0b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80657b0 con 0x7f77e8147d70 2022-07-20T02:30:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38202/0 (socket says 172.21.15.26:38202) 2022-07-20T02:30:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ed1a8700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e8064180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 -- 172.21.15.26:0/2126370089 learned_addr learned my addr 172.21.15.26:0/2126370089 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77e7fff700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 0x7f77e8064fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 -- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 msgr2=0x7f77e8064180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e8064180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 -- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 msgr2=0x7f77e8064fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 0x7f77e8064fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 -- 172.21.15.26:0/2126370089 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc00b180 con 0x7f77e8060f10 2022-07-20T02:30:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ed1a8700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e8064180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77e7fff700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 0x7f77e8064fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77ec9a7700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f77dc002f70 tx=0x7f77dc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc012930 con 0x7f77e8060f10 2022-07-20T02:30:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc012a90 con 0x7f77e8060f10 2022-07-20T02:30:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e8157500 con 0x7f77e8060f10 2022-07-20T02:30:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc012df0 con 0x7f77e8060f10 2022-07-20T02:30:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e8064670 con 0x7f77e8060f10 2022-07-20T02:30:28.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.908+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e80a77a0 con 0x7f77e8060f10 2022-07-20T02:30:28.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.912+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f77dc01c070 con 0x7f77e8060f10 2022-07-20T02:30:28.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.912+0000 7f77e5ffb700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 0x7f77c8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:28.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.912+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f77dc06cbb0 con 0x7f77e8060f10 2022-07-20T02:30:28.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.912+0000 7f77e7fff700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 0x7f77c8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:28.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.912+0000 7f77e7fff700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 0x7f77c8055e10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f77d800d000 tx=0x7f77d8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:28.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:28.920+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77dc043890 con 0x7f77e8060f10 2022-07-20T02:30:29.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.084+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f77e80574e0 con 0x7f77e8060f10 2022-07-20T02:30:29.533 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:30:29.532+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:30:29.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.544+0000 7f77e5ffb700 1 -- 172.21.15.26:0/2126370089 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f77dc019030 con 0x7f77e8060f10 2022-07-20T02:30:29.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 msgr2=0x7f77c8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 0x7f77c8055e10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f77d800d000 tx=0x7f77d8009040).stop 2022-07-20T02:30:29.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 msgr2=0x7f77e8149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f77dc002f70 tx=0x7f77dc014000).stop 2022-07-20T02:30:29.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 shutdown_connections 2022-07-20T02:30:29.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77c8053950 0x7f77c8055e10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77e8147d70 0x7f77e8064180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77e80b2b30 0x7f77e8064fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 --2- 172.21.15.26:0/2126370089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77e8060f10 0x7f77e8149ac0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 >> 172.21.15.26:0/2126370089 conn(0x7f77e80b6110 msgr2=0x7f77e80ac1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:29.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 shutdown_connections 2022-07-20T02:30:29.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.548+0000 7f77eec0b700 1 -- 172.21.15.26:0/2126370089 wait complete. 2022-07-20T02:30:29.559 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:30:29.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:29.589 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:30:29.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 Processor -- start 2022-07-20T02:30:29.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 -- start start 2022-07-20T02:30:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 0x7f9358067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93580980d0 0x7f9358142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935805ff70 con 0x7f9358061860 2022-07-20T02:30:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93580b3640 con 0x7f93580980d0 2022-07-20T02:30:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.668+0000 7f935e92c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93580601a0 con 0x7f93580b3260 2022-07-20T02:30:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93580980d0 0x7f9358142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f93580980d0 0x7f9358142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38226/0 (socket says 172.21.15.26:38226) 2022-07-20T02:30:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 -- 172.21.15.26:0/2393565642 learned_addr learned my addr 172.21.15.26:0/2393565642 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935d92a700 1 --2- 172.21.15.26:0/2393565642 >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 0x7f9358067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 -- 172.21.15.26:0/2393565642 >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 msgr2=0x7f9358067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:29.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 --2- 172.21.15.26:0/2393565642 >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 0x7f9358067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 -- 172.21.15.26:0/2393565642 >> v1:172.21.15.120:6789/0 conn(0x7f9358061860 legacy=0x7f93580979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 -- 172.21.15.26:0/2393565642 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580606e0 con 0x7f93580980d0 2022-07-20T02:30:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935d92a700 1 --2- 172.21.15.26:0/2393565642 >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 0x7f9358067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:29.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e12b700 1 --2- 172.21.15.26:0/2393565642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580980d0 0x7f9358142d30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9350002f70 tx=0x7f9350006eb0).ready entity=mon.0 client_cookie=9d784a0d12acbd79 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:29.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935c928700 1 -- 172.21.15.26:0/2393565642 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935000f9c0 con 0x7f93580980d0 2022-07-20T02:30:29.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935c928700 1 -- 172.21.15.26:0/2393565642 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9350000e90 con 0x7f93580980d0 2022-07-20T02:30:29.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935c928700 1 -- 172.21.15.26:0/2393565642 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935000fd50 con 0x7f93580980d0 2022-07-20T02:30:29.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 -- 172.21.15.26:0/2393565642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580980d0 msgr2=0x7f9358142d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 --2- 172.21.15.26:0/2393565642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580980d0 0x7f9358142d30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9350002f70 tx=0x7f9350006eb0).stop 2022-07-20T02:30:29.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 -- 172.21.15.26:0/2393565642 shutdown_connections 2022-07-20T02:30:29.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 --2- 172.21.15.26:0/2393565642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580980d0 0x7f9358142d30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 --2- 172.21.15.26:0/2393565642 >> v2:172.21.15.120:3300/0 conn(0x7f93580b3260 0x7f9358067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 -- 172.21.15.26:0/2393565642 >> 172.21.15.26:0/2393565642 conn(0x7f93580b67f0 msgr2=0x7f93580b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:29.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 -- 172.21.15.26:0/2393565642 shutdown_connections 2022-07-20T02:30:29.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.672+0000 7f935e92c700 1 -- 172.21.15.26:0/2393565642 wait complete. 2022-07-20T02:30:29.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 Processor -- start 2022-07-20T02:30:29.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- start start 2022-07-20T02:30:29.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9358061860 0x7f93581416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93580980d0 0x7f935814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358144250 con 0x7f93580b3260 2022-07-20T02:30:29.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358144480 con 0x7f93580980d0 2022-07-20T02:30:29.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935814e690 con 0x7f9358061860 2022-07-20T02:30:29.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38232/0 (socket says 172.21.15.26:38232) 2022-07-20T02:30:29.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 -- 172.21.15.26:0/2209087225 learned_addr learned my addr 172.21.15.26:0/2209087225 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:29.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935d92a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9358061860 0x7f93581416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935d129700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93580980d0 0x7f935814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 -- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9358061860 msgr2=0x7f93581416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9358061860 0x7f93581416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 -- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93580980d0 msgr2=0x7f935814eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93580980d0 0x7f935814eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 -- 172.21.15.26:0/2209087225 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935400a610 con 0x7f93580b3260 2022-07-20T02:30:29.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e12b700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9350002f70 tx=0x7f9350017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:29.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9350020070 con 0x7f93580b3260 2022-07-20T02:30:29.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9350010dc0 con 0x7f93580b3260 2022-07-20T02:30:29.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935000b1e0 con 0x7f93580b3260 2022-07-20T02:30:29.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935001a670 con 0x7f93580b3260 2022-07-20T02:30:29.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.676+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935814e3c0 con 0x7f93580b3260 2022-07-20T02:30:29.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9358144c10 con 0x7f93580b3260 2022-07-20T02:30:29.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f935001a7d0 con 0x7f93580b3260 2022-07-20T02:30:29.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f9346ffd700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 0x7f9338055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:29.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f935006f0d0 con 0x7f93580b3260 2022-07-20T02:30:29.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f935d92a700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 0x7f9338055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:29.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.680+0000 7f935d92a700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 0x7f9338055fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f935400c000 tx=0x7f935400a370).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:29.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.684+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9350045e40 con 0x7f93580b3260 2022-07-20T02:30:29.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f935814e850 con 0x7f93580b3260 2022-07-20T02:30:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f9346ffd700 1 -- 172.21.15.26:0/2209087225 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2131 (secure 0 0 0) 0x7f9350049460 con 0x7f93580b3260 2022-07-20T02:30:29.848 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:29.849 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:29:06.619736+0000","modified":"2022-07-20T02:30:29.534387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T02:30:29.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 msgr2=0x7f9338055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 0x7f9338055fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f935400c000 tx=0x7f935400a370).stop 2022-07-20T02:30:29.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 msgr2=0x7f9358151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:29.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9350002f70 tx=0x7f9350017000).stop 2022-07-20T02:30:29.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 shutdown_connections 2022-07-20T02:30:29.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93580b3260 0x7f9358151910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9338053b00 0x7f9338055fc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93580980d0 0x7f935814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 --2- 172.21.15.26:0/2209087225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9358061860 0x7f93581416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:29.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 >> 172.21.15.26:0/2209087225 conn(0x7f93580b67f0 msgr2=0x7f93580b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:29.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 shutdown_connections 2022-07-20T02:30:29.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.848+0000 7f935e92c700 1 -- 172.21.15.26:0/2209087225 wait complete. 2022-07-20T02:30:29.858 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 25 2022-07-20T02:30:29.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:30:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.968+0000 7fcd9748b700 1 Processor -- start 2022-07-20T02:30:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.968+0000 7fcd9748b700 1 -- start start 2022-07-20T02:30:30.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd900602c0 0x7fcd9005fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900aeec0 con 0x7fcd900aeae0 2022-07-20T02:30:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900af210 con 0x7fcd90067060 2022-07-20T02:30:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd900b2ba0 con 0x7fcd900602c0 2022-07-20T02:30:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd94a26700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd900602c0 0x7fcd9005fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd95227700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcd900aeae0 0x7fcd90066950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37516/0 (socket says 172.21.15.26:37516) 2022-07-20T02:30:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd95227700 1 -- 172.21.15.26:0/4013479828 learned_addr learned my addr 172.21.15.26:0/4013479828 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd94a26700 1 -- 172.21.15.26:0/4013479828 >> v1:172.21.15.120:6789/0 conn(0x7fcd90067060 legacy=0x7fcd900af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd7bfff700 1 -- 172.21.15.26:0/4013479828 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200849708 0 0) 0x7fcd900aeec0 con 0x7fcd900aeae0 2022-07-20T02:30:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd94a26700 1 -- 172.21.15.26:0/4013479828 >> v1:172.21.15.26:6789/0 conn(0x7fcd900aeae0 legacy=0x7fcd90066950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd94a26700 1 -- 172.21.15.26:0/4013479828 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90061630 con 0x7fcd900602c0 2022-07-20T02:30:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd94a26700 1 --2- 172.21.15.26:0/4013479828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd9005fa50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c006df0).ready entity=mon.0 client_cookie=f92ab7dc6272ccf5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd7bfff700 1 -- 172.21.15.26:0/4013479828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c00d610 con 0x7fcd900602c0 2022-07-20T02:30:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd7bfff700 1 -- 172.21.15.26:0/4013479828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c000e90 con 0x7fcd900602c0 2022-07-20T02:30:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd7bfff700 1 -- 172.21.15.26:0/4013479828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c00d960 con 0x7fcd900602c0 2022-07-20T02:30:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- 172.21.15.26:0/4013479828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 msgr2=0x7fcd9005fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 --2- 172.21.15.26:0/4013479828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd9005fa50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c006df0).stop 2022-07-20T02:30:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- 172.21.15.26:0/4013479828 shutdown_connections 2022-07-20T02:30:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 --2- 172.21.15.26:0/4013479828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd9005fa50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- 172.21.15.26:0/4013479828 >> 172.21.15.26:0/4013479828 conn(0x7fcd900b66e0 msgr2=0x7fcd900b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- 172.21.15.26:0/4013479828 shutdown_connections 2022-07-20T02:30:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.972+0000 7fcd9748b700 1 -- 172.21.15.26:0/4013479828 wait complete. 2022-07-20T02:30:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 Processor -- start 2022-07-20T02:30:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- start start 2022-07-20T02:30:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd90067060 0x7fcd900af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd900aeae0 0x7fcd90064aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90065e00 con 0x7fcd900602c0 2022-07-20T02:30:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90147f50 con 0x7fcd900aeae0 2022-07-20T02:30:30.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90145810 con 0x7fcd90067060 2022-07-20T02:30:30.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38242/0 (socket says 172.21.15.26:38242) 2022-07-20T02:30:30.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 -- 172.21.15.26:0/1100775382 learned_addr learned my addr 172.21.15.26:0/1100775382 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:30.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd94a26700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd90067060 0x7fcd900af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95a28700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd900aeae0 0x7fcd90064aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 -- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd90067060 msgr2=0x7fcd900af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd90067060 0x7fcd900af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 -- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd900aeae0 msgr2=0x7fcd90064aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd900aeae0 0x7fcd90064aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 -- 172.21.15.26:0/1100775382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c00b180 con 0x7fcd900602c0 2022-07-20T02:30:30.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd95227700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fcd901459a0 tx=0x7fcd84007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd840072e0 con 0x7fcd900602c0 2022-07-20T02:30:30.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84007440 con 0x7fcd900602c0 2022-07-20T02:30:30.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd90145570 con 0x7fcd900602c0 2022-07-20T02:30:30.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd840057e0 con 0x7fcd900602c0 2022-07-20T02:30:30.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.976+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd900b01a0 con 0x7fcd900602c0 2022-07-20T02:30:30.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd900618d0 con 0x7fcd900602c0 2022-07-20T02:30:30.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd8401a060 con 0x7fcd900602c0 2022-07-20T02:30:30.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd82ffd700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 0x7fcd6c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fcd8406ba80 con 0x7fcd900602c0 2022-07-20T02:30:30.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd94a26700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 0x7fcd6c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.980+0000 7fcd94a26700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 0x7fcd6c055ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fcd8c01a000 tx=0x7fcd8c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:29.984+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd840426e0 con 0x7fcd900602c0 2022-07-20T02:30:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.148+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcd900574e0 con 0x7fcd900602c0 2022-07-20T02:30:30.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.552+0000 7fcd82ffd700 1 -- 172.21.15.26:0/1100775382 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcd840180a0 con 0x7fcd900602c0 2022-07-20T02:30:30.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.552+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 msgr2=0x7fcd6c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.552+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 0x7fcd6c055ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fcd8c01a000 tx=0x7fcd8c015000).stop 2022-07-20T02:30:30.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.552+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 msgr2=0x7fcd90148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.552+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fcd901459a0 tx=0x7fcd84007af0).stop 2022-07-20T02:30:30.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 shutdown_connections 2022-07-20T02:30:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd6c0539e0 0x7fcd6c055ea0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd900aeae0 0x7fcd90064aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd90067060 0x7fcd900af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 --2- 172.21.15.26:0/1100775382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd900602c0 0x7fcd90148650 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 >> 172.21.15.26:0/1100775382 conn(0x7fcd900b66e0 msgr2=0x7fcd900b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:30.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 shutdown_connections 2022-07-20T02:30:30.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.556+0000 7fcd9748b700 1 -- 172.21.15.26:0/1100775382 wait complete. 2022-07-20T02:30:30.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:30:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 Processor -- start 2022-07-20T02:30:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 -- start start 2022-07-20T02:30:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3f780aff40 0x7f3f78066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f78060250 0x7f3f78060690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f780b0f50 con 0x7f3f780621f0 2022-07-20T02:30:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7813e7f0 con 0x7f3f780aff40 2022-07-20T02:30:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780b0320 con 0x7f3f78060250 2022-07-20T02:30:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f7659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3f780aff40 0x7f3f78066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f75d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f780621f0 0x7f3f78067020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37534/0 (socket says 172.21.15.26:37534) 2022-07-20T02:30:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f75d9b700 1 -- 172.21.15.26:0/3292136746 learned_addr learned my addr 172.21.15.26:0/3292136746 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.684+0000 7f3f76d9d700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f78060250 0x7f3f78060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 -- 172.21.15.26:0/3292136746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f78060250 msgr2=0x7f3f78060690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f78060250 0x7f3f78060690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 -- 172.21.15.26:0/3292136746 >> v1:172.21.15.26:6789/0 conn(0x7f3f780621f0 legacy=0x7f3f78067020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 -- 172.21.15.26:0/3292136746 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7805f8e0 con 0x7f3f780aff40 2022-07-20T02:30:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78066b30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f3f68002f70 tx=0x7f3f68006df0).ready entity=mon.0 client_cookie=880fabef59855731 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7559a700 1 -- 172.21.15.26:0/3292136746 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800d570 con 0x7f3f780aff40 2022-07-20T02:30:30.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7559a700 1 -- 172.21.15.26:0/3292136746 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f68000e90 con 0x7f3f780aff40 2022-07-20T02:30:30.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7559a700 1 -- 172.21.15.26:0/3292136746 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800d8c0 con 0x7f3f780aff40 2022-07-20T02:30:30.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- 172.21.15.26:0/3292136746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 msgr2=0x7f3f78066b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78066b30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f3f68002f70 tx=0x7f3f68006df0).stop 2022-07-20T02:30:30.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- 172.21.15.26:0/3292136746 shutdown_connections 2022-07-20T02:30:30.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f78060250 0x7f3f78060690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- 172.21.15.26:0/3292136746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78066b30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- 172.21.15.26:0/3292136746 >> 172.21.15.26:0/3292136746 conn(0x7f3f780b3050 msgr2=0x7f3f780b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- 172.21.15.26:0/3292136746 shutdown_connections 2022-07-20T02:30:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- 172.21.15.26:0/3292136746 wait complete. 2022-07-20T02:30:30.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 Processor -- start 2022-07-20T02:30:30.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- start start 2022-07-20T02:30:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f780621f0 0x7f3f780649f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f78064280 con 0x7f3f780aff40 2022-07-20T02:30:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780af2b0 con 0x7f3f78060250 2022-07-20T02:30:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780aece0 con 0x7f3f780621f0 2022-07-20T02:30:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f76d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:57950/0 (socket says 172.21.15.26:57950) 2022-07-20T02:30:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f75d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f780621f0 0x7f3f780649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f7659c700 1 -- 172.21.15.26:0/772621079 learned_addr learned my addr 172.21.15.26:0/772621079 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.688+0000 7f3f76d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38262/0 (socket says 172.21.15.26:38262) 2022-07-20T02:30:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 -- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f780621f0 msgr2=0x7f3f780649f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f780621f0 0x7f3f780649f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 -- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 msgr2=0x7f3f780b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 -- 172.21.15.26:0/772621079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6800b180 con 0x7f3f780aff40 2022-07-20T02:30:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f7659c700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f76d9d700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f3f7000c000 tx=0x7f3f700091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f70012600 con 0x7f3f780aff40 2022-07-20T02:30:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f780644b0 con 0x7f3f780aff40 2022-07-20T02:30:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70012760 con 0x7f3f780aff40 2022-07-20T02:30:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f70012ac0 con 0x7f3f780aff40 2022-07-20T02:30:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f78142ec0 con 0x7f3f780aff40 2022-07-20T02:30:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7805fb10 con 0x7f3f780aff40 2022-07-20T02:30:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f70022070 con 0x7f3f780aff40 2022-07-20T02:30:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 0x7f3f54055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f3f7006c740 con 0x7f3f780aff40 2022-07-20T02:30:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.692+0000 7f3f7659c700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 0x7f3f54055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.696+0000 7f3f7659c700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 0x7f3f54055ea0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f3f6801a000 tx=0x7f3f68014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.700+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f700433b0 con 0x7f3f780aff40 2022-07-20T02:30:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:30.828+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3f780574e0 con 0x7f3f780aff40 2022-07-20T02:30:31.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.556+0000 7f3f677fe700 1 -- 172.21.15.26:0/772621079 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3f70016020 con 0x7f3f780aff40 2022-07-20T02:30:31.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 msgr2=0x7f3f54055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:31.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 0x7f3f54055ea0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f3f6801a000 tx=0x7f3f68014000).stop 2022-07-20T02:30:31.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 msgr2=0x7f3f78064ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:31.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f3f7000c000 tx=0x7f3f700091d0).stop 2022-07-20T02:30:31.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 shutdown_connections 2022-07-20T02:30:31.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3f780aff40 0x7f3f78064ee0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3f540539e0 0x7f3f54055ea0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f780621f0 0x7f3f780649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 --2- 172.21.15.26:0/772621079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f78060250 0x7f3f780b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 >> 172.21.15.26:0/772621079 conn(0x7f3f780b3050 msgr2=0x7f3f7805cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:31.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 shutdown_connections 2022-07-20T02:30:31.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.564+0000 7f3f7759e700 1 -- 172.21.15.26:0/772621079 wait complete. 2022-07-20T02:30:31.572 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:30:31.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:30:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.688+0000 7fa3da535700 1 Processor -- start 2022-07-20T02:30:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.688+0000 7fa3da535700 1 -- start start 2022-07-20T02:30:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa3d41465b0 0x7fa3d4066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa3d40b24c0 0x7fa3d40604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40672a0 0x7fa3d40609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40b2e20 con 0x7fa3d40b24c0 2022-07-20T02:30:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40b1ae0 con 0x7fa3d41465b0 2022-07-20T02:30:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40b1d80 con 0x7fa3d40672a0 2022-07-20T02:30:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa3d40b24c0 0x7fa3d40604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa3d40b24c0 0x7fa3d40604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38268/0 (socket says 172.21.15.26:38268) 2022-07-20T02:30:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 -- 172.21.15.26:0/2679171951 learned_addr learned my addr 172.21.15.26:0/2679171951 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d9533700 1 --2- 172.21.15.26:0/2679171951 >> v2:172.21.15.120:3300/0 conn(0x7fa3d41465b0 0x7fa3d4066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d9d34700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40672a0 0x7fa3d40609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 -- 172.21.15.26:0/2679171951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40672a0 msgr2=0x7fa3d40609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40672a0 0x7fa3d40609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 -- 172.21.15.26:0/2679171951 >> v2:172.21.15.120:3300/0 conn(0x7fa3d41465b0 msgr2=0x7fa3d4066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 --2- 172.21.15.26:0/2679171951 >> v2:172.21.15.120:3300/0 conn(0x7fa3d41465b0 0x7fa3d4066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 -- 172.21.15.26:0/2679171951 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40b2020 con 0x7fa3d40b24c0 2022-07-20T02:30:31.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3d8d32700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40b24c0 0x7fa3d40604e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fa3c4002f70 tx=0x7fa3c4006eb0).ready entity=mon.0 client_cookie=9717aad2c943c085 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:31.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3c3fff700 1 -- 172.21.15.26:0/2679171951 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c400d570 con 0x7fa3d40b24c0 2022-07-20T02:30:31.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3c3fff700 1 -- 172.21.15.26:0/2679171951 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4000e90 con 0x7fa3d40b24c0 2022-07-20T02:30:31.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3c3fff700 1 -- 172.21.15.26:0/2679171951 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c400d900 con 0x7fa3d40b24c0 2022-07-20T02:30:31.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 -- 172.21.15.26:0/2679171951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40b24c0 msgr2=0x7fa3d40604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:31.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.692+0000 7fa3da535700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40b24c0 0x7fa3d40604e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fa3c4002f70 tx=0x7fa3c4006eb0).stop 2022-07-20T02:30:31.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/2679171951 shutdown_connections 2022-07-20T02:30:31.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d40672a0 0x7fa3d40609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- 172.21.15.26:0/2679171951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40b24c0 0x7fa3d40604e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- 172.21.15.26:0/2679171951 >> v2:172.21.15.120:3300/0 conn(0x7fa3d41465b0 0x7fa3d4066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/2679171951 >> 172.21.15.26:0/2679171951 conn(0x7fa3d40b3050 msgr2=0x7fa3d40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/2679171951 shutdown_connections 2022-07-20T02:30:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/2679171951 wait complete. 2022-07-20T02:30:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 Processor -- start 2022-07-20T02:30:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- start start 2022-07-20T02:30:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40672a0 0x7fa3d40abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d41465b0 0x7fa3d4063b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4064170 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4064990 con 0x7fa3d40672a0 2022-07-20T02:30:31.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40acd60 con 0x7fa3d41465b0 2022-07-20T02:30:31.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38276/0 (socket says 172.21.15.26:38276) 2022-07-20T02:30:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 -- 172.21.15.26:0/3736707739 learned_addr learned my addr 172.21.15.26:0/3736707739 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9533700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40672a0 0x7fa3d40abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d8d32700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d41465b0 0x7fa3d4063b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 -- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d41465b0 msgr2=0x7fa3d4063b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d41465b0 0x7fa3d4063b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 -- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40672a0 msgr2=0x7fa3d40abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40672a0 0x7fa3d40abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 -- 172.21.15.26:0/3736707739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c400b1e0 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3d9d34700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fa3cc00c000 tx=0x7fa3cc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3cc00c670 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc00c7d0 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3cc00cb30 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d4066050 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.696+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d4066790 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d405f840 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa3cc022070 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3c27fc700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 0x7fa3b4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fa3cc06c530 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3d9533700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 0x7fa3b4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.700+0000 7fa3d9533700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 0x7fa3b4055f30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa3d000b000 tx=0x7fa3d0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.708+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3cc043450 con 0x7fa3d40ac8a0 2022-07-20T02:30:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:31.832+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3d40574e0 con 0x7fa3d40ac8a0 2022-07-20T02:30:32.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.560+0000 7fa3c27fc700 1 -- 172.21.15.26:0/3736707739 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa3cc0160d0 con 0x7fa3d40ac8a0 2022-07-20T02:30:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 msgr2=0x7fa3b4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 0x7fa3b4055f30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa3d000b000 tx=0x7fa3d0006df0).stop 2022-07-20T02:30:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 msgr2=0x7fa3d40651d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fa3cc00c000 tx=0x7fa3cc0091d0).stop 2022-07-20T02:30:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 shutdown_connections 2022-07-20T02:30:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa3d40ac8a0 0x7fa3d40651d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa3b4053a70 0x7fa3b4055f30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3d41465b0 0x7fa3d4063b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.564+0000 7fa3da535700 1 --2- 172.21.15.26:0/3736707739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3d40672a0 0x7fa3d40abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.568+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 >> 172.21.15.26:0/3736707739 conn(0x7fa3d40b3050 msgr2=0x7fa3d40b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:32.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.568+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 shutdown_connections 2022-07-20T02:30:32.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.568+0000 7fa3da535700 1 -- 172.21.15.26:0/3736707739 wait complete. 2022-07-20T02:30:32.578 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:30:32.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:30:32.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 Processor -- start 2022-07-20T02:30:32.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 -- start start 2022-07-20T02:30:32.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f700b5ef0 0x7f4f701474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f700af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f700b62d0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7014aac0 con 0x7f4f700b5ef0 2022-07-20T02:30:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f765cd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7014ad60 con 0x7f4f700a8c40 2022-07-20T02:30:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f700b5ef0 0x7f4f701474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f700b5ef0 0x7f4f701474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38282/0 (socket says 172.21.15.26:38282) 2022-07-20T02:30:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f74dca700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4f700a7fc0 0x7f4f700aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51772/0 (socket says 172.21.15.26:51772) 2022-07-20T02:30:32.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 -- 172.21.15.26:0/3616547446 learned_addr learned my addr 172.21.15.26:0/3616547446 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:32.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f75dcc700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f700af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 -- 172.21.15.26:0/3616547446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 msgr2=0x7f4f700af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:32.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f700af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 -- 172.21.15.26:0/3616547446 >> v1:172.21.15.120:6789/0 conn(0x7f4f700a7fc0 legacy=0x7f4f700aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 -- 172.21.15.26:0/3616547446 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7014b000 con 0x7f4f700b5ef0 2022-07-20T02:30:32.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f755cb700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700b5ef0 0x7f4f701474d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f4f60002f70 tx=0x7f4f60006eb0).ready entity=mon.0 client_cookie=497dbf119720818a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:32.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f67fff700 1 -- 172.21.15.26:0/3616547446 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f6000d5c0 con 0x7f4f700b5ef0 2022-07-20T02:30:32.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f67fff700 1 -- 172.21.15.26:0/3616547446 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60000e90 con 0x7f4f700b5ef0 2022-07-20T02:30:32.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.688+0000 7f4f67fff700 1 -- 172.21.15.26:0/3616547446 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f6000d950 con 0x7f4f700b5ef0 2022-07-20T02:30:32.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- 172.21.15.26:0/3616547446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700b5ef0 msgr2=0x7f4f701474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700b5ef0 0x7f4f701474d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f4f60002f70 tx=0x7f4f60006eb0).stop 2022-07-20T02:30:32.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- 172.21.15.26:0/3616547446 shutdown_connections 2022-07-20T02:30:32.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f700af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 --2- 172.21.15.26:0/3616547446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700b5ef0 0x7f4f701474d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- 172.21.15.26:0/3616547446 >> 172.21.15.26:0/3616547446 conn(0x7f4f70060310 msgr2=0x7f4f700b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- 172.21.15.26:0/3616547446 shutdown_connections 2022-07-20T02:30:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- 172.21.15.26:0/3616547446 wait complete. 2022-07-20T02:30:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 Processor -- start 2022-07-20T02:30:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.692+0000 7f4f765cd700 1 -- start start 2022-07-20T02:30:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f70145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f700b5ef0 0x7f4f70145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700aa9e0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700aac80 con 0x7f4f700b5ef0 2022-07-20T02:30:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701474b0 con 0x7f4f700a8c40 2022-07-20T02:30:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38286/0 (socket says 172.21.15.26:38286) 2022-07-20T02:30:32.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 -- 172.21.15.26:0/265454665 learned_addr learned my addr 172.21.15.26:0/265454665 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:32.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f74dca700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f70145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f75dcc700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f700b5ef0 0x7f4f70145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 -- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 msgr2=0x7f4f70145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f70145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 -- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f700b5ef0 msgr2=0x7f4f70145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f700b5ef0 0x7f4f70145860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 -- 172.21.15.26:0/265454665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6000b1e0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f74dca700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f70145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f755cb700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f4f60002f70 tx=0x7f4f60015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f60011860 con 0x7f4f700a7fc0 2022-07-20T02:30:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f600119c0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f70147750 con 0x7f4f700a7fc0 2022-07-20T02:30:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f60011d20 con 0x7f4f700a7fc0 2022-07-20T02:30:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f70147e90 con 0x7f4f700a7fc0 2022-07-20T02:30:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7014b2a0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f60025030 con 0x7f4f700a7fc0 2022-07-20T02:30:32.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.696+0000 7f4f667fc700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 0x7f4f50055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.700+0000 7f4f74dca700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 0x7f4f50055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.700+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f4f6006d290 con 0x7f4f700a7fc0 2022-07-20T02:30:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.700+0000 7f4f74dca700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 0x7f4f50055f30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f4f6c00b000 tx=0x7f4f6c0070e0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:32.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.700+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f60044220 con 0x7f4f700a7fc0 2022-07-20T02:30:32.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.824+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4f700574e0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.832+0000 7f4f667fc700 1 -- 172.21.15.26:0/265454665 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4f600230a0 con 0x7f4f700a7fc0 2022-07-20T02:30:32.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 msgr2=0x7f4f50055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 0x7f4f50055f30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f4f6c00b000 tx=0x7f4f6c0070e0).stop 2022-07-20T02:30:32.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 msgr2=0x7f4f700b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:32.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f4f60002f70 tx=0x7f4f60015000).stop 2022-07-20T02:30:32.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 shutdown_connections 2022-07-20T02:30:32.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f50053a70 0x7f4f50055f30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f700b5ef0 0x7f4f70145860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f700a8c40 0x7f4f70145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 --2- 172.21.15.26:0/265454665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f700a7fc0 0x7f4f700b4370 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:32.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 >> 172.21.15.26:0/265454665 conn(0x7f4f70060310 msgr2=0x7f4f701498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:32.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 shutdown_connections 2022-07-20T02:30:32.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.836+0000 7f4f765cd700 1 -- 172.21.15.26:0/265454665 wait complete. 2022-07-20T02:30:32.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-20T02:30:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 Processor -- start 2022-07-20T02:30:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 -- start start 2022-07-20T02:30:33.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54380b5ef0 0x7f54380b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380b62d0 con 0x7f54380a97e0 2022-07-20T02:30:33.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380a9cf0 con 0x7f54380b1850 2022-07-20T02:30:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543e20f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54380a9f90 con 0x7f54380b5ef0 2022-07-20T02:30:33.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543ca0c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f54380a97e0 0x7f54381426a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37576/0 (socket says 172.21.15.26:37576) 2022-07-20T02:30:33.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543ca0c700 1 -- 172.21.15.26:0/1252992502 learned_addr learned my addr 172.21.15.26:0/1252992502 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:33.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f543d20d700 1 --2- 172.21.15.26:0/1252992502 >> v2:172.21.15.120:3300/0 conn(0x7f54380b5ef0 0x7f54380b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621741215 0 0) 0x7f54380b62d0 con 0x7f54380a97e0 2022-07-20T02:30:33.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.956+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5418003260 con 0x7f54380a97e0 2022-07-20T02:30:33.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245127662 0 0) 0x7f54380a9cf0 con 0x7f54380b1850 2022-07-20T02:30:33.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54380b62d0 con 0x7f54380b1850 2022-07-20T02:30:33.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543d20d700 1 -- 172.21.15.26:0/1252992502 >> v1:172.21.15.120:6789/0 conn(0x7f54380b1850 legacy=0x7f543814a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543d20d700 1 -- 172.21.15.26:0/1252992502 >> v1:172.21.15.26:6789/0 conn(0x7f54380a97e0 legacy=0x7f54381426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543d20d700 1 -- 172.21.15.26:0/1252992502 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380aa230 con 0x7f54380b5ef0 2022-07-20T02:30:33.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543d20d700 1 --2- 172.21.15.26:0/1252992502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b5ef0 0x7f54380b0380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f54200032f0 tx=0x7f54200091d0).ready entity=mon.1 client_cookie=6fffb429391cc9fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f542000b600 con 0x7f54380b5ef0 2022-07-20T02:30:33.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542000b760 con 0x7f54380b5ef0 2022-07-20T02:30:33.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f5437fff700 1 -- 172.21.15.26:0/1252992502 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f542000b8d0 con 0x7f54380b5ef0 2022-07-20T02:30:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- 172.21.15.26:0/1252992502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b5ef0 msgr2=0x7f54380b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 --2- 172.21.15.26:0/1252992502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b5ef0 0x7f54380b0380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f54200032f0 tx=0x7f54200091d0).stop 2022-07-20T02:30:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- 172.21.15.26:0/1252992502 shutdown_connections 2022-07-20T02:30:33.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 --2- 172.21.15.26:0/1252992502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b5ef0 0x7f54380b0380 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- 172.21.15.26:0/1252992502 >> 172.21.15.26:0/1252992502 conn(0x7f5438060310 msgr2=0x7f54380b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:33.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- 172.21.15.26:0/1252992502 shutdown_connections 2022-07-20T02:30:33.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- 172.21.15.26:0/1252992502 wait complete. 2022-07-20T02:30:33.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 Processor -- start 2022-07-20T02:30:33.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- start start 2022-07-20T02:30:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54380a97e0 0x7f54380b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b1850 0x7f543814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438065af0 con 0x7f54380b5ef0 2022-07-20T02:30:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54380662a0 con 0x7f54380b1850 2022-07-20T02:30:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.960+0000 7f543e20f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5438065d20 con 0x7f54380a97e0 2022-07-20T02:30:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38304/0 (socket says 172.21.15.26:38304) 2022-07-20T02:30:33.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 -- 172.21.15.26:0/1820187784 learned_addr learned my addr 172.21.15.26:0/1820187784 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:33.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543d20d700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54380a97e0 0x7f54380b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543ca0c700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b1850 0x7f543814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 -- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54380a97e0 msgr2=0x7f54380b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54380a97e0 0x7f54380b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 -- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b1850 msgr2=0x7f543814f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b1850 0x7f543814f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 -- 172.21.15.26:0/1820187784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420009470 con 0x7f54380b5ef0 2022-07-20T02:30:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543da0e700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f5430001110 tx=0x7f5430007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5430012400 con 0x7f54380b5ef0 2022-07-20T02:30:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5430012560 con 0x7f54380b5ef0 2022-07-20T02:30:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5438066500 con 0x7f54380b5ef0 2022-07-20T02:30:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54300128c0 con 0x7f54380b5ef0 2022-07-20T02:30:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543809ebe0 con 0x7f54380b5ef0 2022-07-20T02:30:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5438066880 con 0x7f54380b5ef0 2022-07-20T02:30:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.964+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5430010070 con 0x7f54380b5ef0 2022-07-20T02:30:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.968+0000 7f54367fc700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 0x7f541c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.968+0000 7f543d20d700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 0x7f541c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.968+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f543006afe0 con 0x7f54380b5ef0 2022-07-20T02:30:33.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.968+0000 7f543d20d700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 0x7f541c055f30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5420019000 tx=0x7f5420016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:32.972+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5430041f50 con 0x7f54380b5ef0 2022-07-20T02:30:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.136+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f54380afee0 con 0x7f54380b5ef0 2022-07-20T02:30:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f54367fc700 1 -- 172.21.15.26:0/1820187784 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v221) v1 ==== 152+0+0 (secure 0 0 0) 0x7f5430045570 con 0x7f54380b5ef0 2022-07-20T02:30:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 msgr2=0x7f541c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 0x7f541c055f30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5420019000 tx=0x7f5420016000).stop 2022-07-20T02:30:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 msgr2=0x7f543814fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f5430001110 tx=0x7f5430007ae0).stop 2022-07-20T02:30:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 shutdown_connections 2022-07-20T02:30:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54380b5ef0 0x7f543814fb90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f541c053a70 0x7f541c055f30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54380b1850 0x7f543814f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 --2- 172.21.15.26:0/1820187784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54380a97e0 0x7f54380b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.576+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 >> 172.21.15.26:0/1820187784 conn(0x7f5438060310 msgr2=0x7f54380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.580+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 shutdown_connections 2022-07-20T02:30:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.580+0000 7f543e20f700 1 -- 172.21.15.26:0/1820187784 wait complete. 2022-07-20T02:30:33.602 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T02:30:33.603 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T02:30:33.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-20T02:30:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 Processor -- start 2022-07-20T02:30:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 -- start start 2022-07-20T02:30:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 0x7f1e440b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e440b2e40 con 0x7f1e44064220 2022-07-20T02:30:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e440617a0 con 0x7f1e440b3060 2022-07-20T02:30:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e4a3f6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e44060b20 con 0x7f1e440b7d80 2022-07-20T02:30:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e48bf3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 0x7f1e440b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e49bf5700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1e44064220 0x7f1e44146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:51804/0 (socket says 172.21.15.26:51804) 2022-07-20T02:30:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:59824/0 (socket says 172.21.15.26:59824) 2022-07-20T02:30:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e49bf5700 1 -- 172.21.15.26:0/352641557 learned_addr learned my addr 172.21.15.26:0/352641557 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e3bfff700 1 -- 172.21.15.26:0/352641557 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324347626 0 0) 0x7f1e440b2e40 con 0x7f1e44064220 2022-07-20T02:30:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e3bfff700 1 -- 172.21.15.26:0/352641557 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28003260 con 0x7f1e44064220 2022-07-20T02:30:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 -- 172.21.15.26:0/352641557 >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 msgr2=0x7f1e440b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 --2- 172.21.15.26:0/352641557 >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 0x7f1e440b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 -- 172.21.15.26:0/352641557 >> v1:172.21.15.120:6789/0 conn(0x7f1e44064220 legacy=0x7f1e44146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e493f4700 1 -- 172.21.15.26:0/352641557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44060dc0 con 0x7f1e440b7d80 2022-07-20T02:30:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.712+0000 7f1e48bf3700 1 --2- 172.21.15.26:0/352641557 >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 0x7f1e440b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e493f4700 1 --2- 172.21.15.26:0/352641557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1e40002f70 tx=0x7f1e40006df0).ready entity=mon.2 client_cookie=c357f8f5fbd731f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e3bfff700 1 -- 172.21.15.26:0/352641557 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e4000d4a0 con 0x7f1e440b7d80 2022-07-20T02:30:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e3bfff700 1 -- 172.21.15.26:0/352641557 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e40000e90 con 0x7f1e440b7d80 2022-07-20T02:30:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e3bfff700 1 -- 172.21.15.26:0/352641557 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e4000d7f0 con 0x7f1e440b7d80 2022-07-20T02:30:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/352641557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 msgr2=0x7f1e440b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/352641557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1e40002f70 tx=0x7f1e40006df0).stop 2022-07-20T02:30:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/352641557 shutdown_connections 2022-07-20T02:30:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/352641557 >> v2:172.21.15.26:3300/0 conn(0x7f1e440b3060 0x7f1e440b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/352641557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e440b4020 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/352641557 >> 172.21.15.26:0/352641557 conn(0x7f1e4405f840 msgr2=0x7f1e4405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:33.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/352641557 shutdown_connections 2022-07-20T02:30:33.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.716+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/352641557 wait complete. 2022-07-20T02:30:33.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 Processor -- start 2022-07-20T02:30:33.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- start start 2022-07-20T02:30:33.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 0x7f1e441526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e44152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414fc00 con 0x7f1e44064220 2022-07-20T02:30:33.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e44151fc0 con 0x7f1e440b3060 2022-07-20T02:30:33.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440b0990 con 0x7f1e440b7d80 2022-07-20T02:30:33.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38314/0 (socket says 172.21.15.26:38314) 2022-07-20T02:30:33.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 -- 172.21.15.26:0/978779714 learned_addr learned my addr 172.21.15.26:0/978779714 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:33.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e49bf5700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e44152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e48bf3700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 0x7f1e441526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 -- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 msgr2=0x7f1e44152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e44152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 -- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 msgr2=0x7f1e441526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 0x7f1e441526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 -- 172.21.15.26:0/978779714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3400b180 con 0x7f1e44064220 2022-07-20T02:30:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e49bf5700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e44152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e48bf3700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 0x7f1e441526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e493f4700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f1e40002f70 tx=0x7f1e40014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e40013ac0 con 0x7f1e44064220 2022-07-20T02:30:33.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e40013c20 con 0x7f1e44064220 2022-07-20T02:30:33.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4000b180 con 0x7f1e44064220 2022-07-20T02:30:33.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e4001e400 con 0x7f1e44064220 2022-07-20T02:30:33.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.720+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e440a6010 con 0x7f1e44064220 2022-07-20T02:30:33.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44061060 con 0x7f1e44064220 2022-07-20T02:30:33.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e40025030 con 0x7f1e44064220 2022-07-20T02:30:33.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e3a7fc700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 0x7f1e2c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:33.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1e4006cc80 con 0x7f1e44064220 2022-07-20T02:30:33.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e48bf3700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 0x7f1e2c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:33.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.724+0000 7f1e48bf3700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 0x7f1e2c055fc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f1e3400b000 tx=0x7f1e34008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.732+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e40043c10 con 0x7f1e44064220 2022-07-20T02:30:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:33.908+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f1e440574e0 con 0x7f1e44064220 2022-07-20T02:30:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.576+0000 7f1e3a7fc700 1 -- 172.21.15.26:0/978779714 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v222) v1 ==== 150+0+0 (secure 0 0 0) 0x7f1e40023070 con 0x7f1e44064220 2022-07-20T02:30:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 msgr2=0x7f1e2c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 0x7f1e2c055fc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f1e3400b000 tx=0x7f1e34008040).stop 2022-07-20T02:30:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 msgr2=0x7f1e44064bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f1e40002f70 tx=0x7f1e40014000).stop 2022-07-20T02:30:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 shutdown_connections 2022-07-20T02:30:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e2c053b00 0x7f1e2c055fc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e440b7d80 0x7f1e44152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e440b3060 0x7f1e441526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 --2- 172.21.15.26:0/978779714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e44064220 0x7f1e44064bc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 >> 172.21.15.26:0/978779714 conn(0x7f1e4405f840 msgr2=0x7f1e44065610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 shutdown_connections 2022-07-20T02:30:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.580+0000 7f1e4a3f6700 1 -- 172.21.15.26:0/978779714 wait complete. 2022-07-20T02:30:34.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:30:34.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 Processor -- start 2022-07-20T02:30:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 -- start start 2022-07-20T02:30:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 0x7fcb60147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6005d4d0 con 0x7fcb600b1200 2022-07-20T02:30:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb600a9ff0 con 0x7fcb600afe40 2022-07-20T02:30:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60067e60 con 0x7fcb6005c720 2022-07-20T02:30:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38324/0 (socket says 172.21.15.26:38324) 2022-07-20T02:30:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5e7fc700 1 -- 172.21.15.26:0/3828916789 learned_addr learned my addr 172.21.15.26:0/3828916789 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 --2- 172.21.15.26:0/3828916789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 0x7fcb60147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5dffb700 1 -- 172.21.15.26:0/3828916789 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826595685 0 0) 0x7fcb6005d4d0 con 0x7fcb600b1200 2022-07-20T02:30:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 -- 172.21.15.26:0/3828916789 >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 msgr2=0x7fcb60147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 --2- 172.21.15.26:0/3828916789 >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 -- 172.21.15.26:0/3828916789 >> v1:172.21.15.120:6789/0 conn(0x7fcb600b1200 legacy=0x7fcb600abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 -- 172.21.15.26:0/3828916789 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb60067bc0 con 0x7fcb6005c720 2022-07-20T02:30:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5e7fc700 1 --2- 172.21.15.26:0/3828916789 >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5effd700 1 --2- 172.21.15.26:0/3828916789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 0x7fcb60147550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcb54002f70 tx=0x7fcb54006df0).ready entity=mon.2 client_cookie=2e7ece97545ff149 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5dffb700 1 -- 172.21.15.26:0/3828916789 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5400d570 con 0x7fcb6005c720 2022-07-20T02:30:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5dffb700 1 -- 172.21.15.26:0/3828916789 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54000e90 con 0x7fcb6005c720 2022-07-20T02:30:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.720+0000 7fcb5dffb700 1 -- 172.21.15.26:0/3828916789 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5400d8c0 con 0x7fcb6005c720 2022-07-20T02:30:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- 172.21.15.26:0/3828916789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 msgr2=0x7fcb60147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/3828916789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 0x7fcb60147550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcb54002f70 tx=0x7fcb54006df0).stop 2022-07-20T02:30:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- 172.21.15.26:0/3828916789 shutdown_connections 2022-07-20T02:30:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/3828916789 >> v2:172.21.15.26:3300/0 conn(0x7fcb600afe40 0x7fcb60147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/3828916789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6005c720 0x7fcb60147550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- 172.21.15.26:0/3828916789 >> 172.21.15.26:0/3828916789 conn(0x7fcb60060310 msgr2=0x7fcb600b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- 172.21.15.26:0/3828916789 shutdown_connections 2022-07-20T02:30:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- 172.21.15.26:0/3828916789 wait complete. 2022-07-20T02:30:34.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 Processor -- start 2022-07-20T02:30:34.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.724+0000 7fcb5ffff700 1 -- start start 2022-07-20T02:30:34.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb6005c720 0x7fcb60143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb600b1200 0x7fcb601415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600b4880 con 0x7fcb600afe40 2022-07-20T02:30:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb601445c0 con 0x7fcb6005c720 2022-07-20T02:30:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60144860 con 0x7fcb600b1200 2022-07-20T02:30:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38330/0 (socket says 172.21.15.26:38330) 2022-07-20T02:30:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 -- 172.21.15.26:0/1548507095 learned_addr learned my addr 172.21.15.26:0/1548507095 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5effd700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb6005c720 0x7fcb60143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5f7fe700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb600b1200 0x7fcb601415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 -- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb600b1200 msgr2=0x7fcb601415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb600b1200 0x7fcb601415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 -- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb6005c720 msgr2=0x7fcb60143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb6005c720 0x7fcb60143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 -- 172.21.15.26:0/1548507095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5000b1e0 con 0x7fcb600afe40 2022-07-20T02:30:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5e7fc700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcb5000c000 tx=0x7fcb50006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb50012600 con 0x7fcb600afe40 2022-07-20T02:30:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb50012760 con 0x7fcb600afe40 2022-07-20T02:30:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5400b180 con 0x7fcb600afe40 2022-07-20T02:30:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb50012ac0 con 0x7fcb600afe40 2022-07-20T02:30:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.728+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb600b44c0 con 0x7fcb600afe40 2022-07-20T02:30:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb600aa250 con 0x7fcb600afe40 2022-07-20T02:30:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb50022070 con 0x7fcb600afe40 2022-07-20T02:30:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb4ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 0x7fcb3c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb5effd700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 0x7fcb3c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fcb5006c200 con 0x7fcb600afe40 2022-07-20T02:30:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.732+0000 7fcb5effd700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 0x7fcb3c055ea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fcb5401a000 tx=0x7fcb54014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.736+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb500432a0 con 0x7fcb600afe40 2022-07-20T02:30:34.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.860+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcb60144d00 con 0x7fcb600afe40 2022-07-20T02:30:34.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.864+0000 7fcb4ffff700 1 -- 172.21.15.26:0/1548507095 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcb500468c0 con 0x7fcb600afe40 2022-07-20T02:30:34.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 msgr2=0x7fcb3c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 0x7fcb3c055ea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fcb5401a000 tx=0x7fcb54014000).stop 2022-07-20T02:30:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 msgr2=0x7fcb601410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcb5000c000 tx=0x7fcb50006eb0).stop 2022-07-20T02:30:34.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 shutdown_connections 2022-07-20T02:30:34.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb600afe40 0x7fcb601410e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb3c0539e0 0x7fcb3c055ea0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb600b1200 0x7fcb601415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 --2- 172.21.15.26:0/1548507095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb6005c720 0x7fcb60143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:34.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 >> 172.21.15.26:0/1548507095 conn(0x7fcb60060310 msgr2=0x7fcb60149600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:34.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 shutdown_connections 2022-07-20T02:30:34.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:34.868+0000 7fcb5ffff700 1 -- 172.21.15.26:0/1548507095 wait complete. 2022-07-20T02:30:34.881 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:30:34.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:34.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:34.936 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:34.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:30:35.016 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:30:35.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:35.018 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:35.019 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:30:35.043 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:30:35.044 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:30:35.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:35.045 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:35.046 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:30:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:30:35.073 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:30:35.077 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:30:35.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:30:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.212+0000 7fca31172700 1 Processor -- start 2022-07-20T02:30:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.212+0000 7fca31172700 1 -- start start 2022-07-20T02:30:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 0x7fca2c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 0x7fca2c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c1474e0 con 0x7fca2c060f00 2022-07-20T02:30:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c05d3b0 con 0x7fca2c05cfd0 2022-07-20T02:30:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1431e0 con 0x7fca2c147000 2022-07-20T02:30:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca30971700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fca2c060f00 0x7fca2c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37620/0 (socket says 172.21.15.26:37620) 2022-07-20T02:30:35.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca30971700 1 -- 172.21.15.26:0/4170740408 learned_addr learned my addr 172.21.15.26:0/4170740408 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:35.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/4170740408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 0x7fca2c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2bfff700 1 --2- 172.21.15.26:0/4170740408 >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 0x7fca2c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2affd700 1 -- 172.21.15.26:0/4170740408 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694437496 0 0) 0x7fca2c1474e0 con 0x7fca2c060f00 2022-07-20T02:30:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2affd700 1 -- 172.21.15.26:0/4170740408 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c003260 con 0x7fca2c060f00 2022-07-20T02:30:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 -- 172.21.15.26:0/4170740408 >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 msgr2=0x7fca2c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/4170740408 >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 0x7fca2c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 -- 172.21.15.26:0/4170740408 >> v1:172.21.15.26:6789/0 conn(0x7fca2c060f00 legacy=0x7fca2c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 -- 172.21.15.26:0/4170740408 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c143480 con 0x7fca2c147000 2022-07-20T02:30:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2bfff700 1 --2- 172.21.15.26:0/4170740408 >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 0x7fca2c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/4170740408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 0x7fca2c0b4730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fca1c002f70 tx=0x7fca1c006df0).ready entity=mon.2 client_cookie=7a86e3b4f185373f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2affd700 1 -- 172.21.15.26:0/4170740408 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c00d4a0 con 0x7fca2c147000 2022-07-20T02:30:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2affd700 1 -- 172.21.15.26:0/4170740408 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1c000e90 con 0x7fca2c147000 2022-07-20T02:30:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca2affd700 1 -- 172.21.15.26:0/4170740408 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c00d7f0 con 0x7fca2c147000 2022-07-20T02:30:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- 172.21.15.26:0/4170740408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 msgr2=0x7fca2c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 --2- 172.21.15.26:0/4170740408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 0x7fca2c0b4730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fca1c002f70 tx=0x7fca1c006df0).stop 2022-07-20T02:30:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- 172.21.15.26:0/4170740408 shutdown_connections 2022-07-20T02:30:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 --2- 172.21.15.26:0/4170740408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c147000 0x7fca2c0b4730 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 --2- 172.21.15.26:0/4170740408 >> v2:172.21.15.120:3300/0 conn(0x7fca2c05cfd0 0x7fca2c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.216+0000 7fca31172700 1 -- 172.21.15.26:0/4170740408 >> 172.21.15.26:0/4170740408 conn(0x7fca2c0620c0 msgr2=0x7fca2c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- 172.21.15.26:0/4170740408 shutdown_connections 2022-07-20T02:30:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- 172.21.15.26:0/4170740408 wait complete. 2022-07-20T02:30:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 Processor -- start 2022-07-20T02:30:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- start start 2022-07-20T02:30:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c060f00 0x7fca2c0b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0b04a0 0x7fca2c1425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1412b0 con 0x7fca2c147000 2022-07-20T02:30:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0b7fb0 con 0x7fca2c060f00 2022-07-20T02:30:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca31172700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c152430 con 0x7fca2c0b04a0 2022-07-20T02:30:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38344/0 (socket says 172.21.15.26:38344) 2022-07-20T02:30:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 -- 172.21.15.26:0/3723396842 learned_addr learned my addr 172.21.15.26:0/3723396842 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2bfff700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c060f00 0x7fca2c0b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca30971700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0b04a0 0x7fca2c1425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 -- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0b04a0 msgr2=0x7fca2c1425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0b04a0 0x7fca2c1425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 -- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c060f00 msgr2=0x7fca2c0b90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c060f00 0x7fca2c0b90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.220+0000 7fca2b7fe700 1 -- 172.21.15.26:0/3723396842 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1400b180 con 0x7fca2c147000 2022-07-20T02:30:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca2b7fe700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fca2c0b80f0 tx=0x7fca1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c012930 con 0x7fca2c147000 2022-07-20T02:30:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1c012a90 con 0x7fca2c147000 2022-07-20T02:30:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c012df0 con 0x7fca2c147000 2022-07-20T02:30:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c00b180 con 0x7fca2c147000 2022-07-20T02:30:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c141010 con 0x7fca2c147000 2022-07-20T02:30:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c1422e0 con 0x7fca2c147000 2022-07-20T02:30:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fca1c01c070 con 0x7fca2c147000 2022-07-20T02:30:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 0x7fca10055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fca1c06cad0 con 0x7fca2c147000 2022-07-20T02:30:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.224+0000 7fca2bfff700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 0x7fca10055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.228+0000 7fca2bfff700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 0x7fca10055fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fca1400b000 tx=0x7fca14006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.236+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca1c0439e0 con 0x7fca2c147000 2022-07-20T02:30:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.368+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fca2c143720 con 0x7fca2c147000 2022-07-20T02:30:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.368+0000 7fca297fa700 1 -- 172.21.15.26:0/3723396842 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v58) v1 ==== 81+0+29268 (secure 0 0 0) 0x7fca1c033270 con 0x7fca2c147000 2022-07-20T02:30:35.370 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:35.370 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:30:35.371 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 58, 2022-07-20T02:30:35.371 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:30:35.372 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:30:35.372 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:30:32.552094+0000", 2022-07-20T02:30:35.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:30:35.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:30:35.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:30:35.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:30:35.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:30:35.375 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:30:35.375 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:30:35.376 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:30:35.376 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:30:35.376 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:35.377 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:30:35.377 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:30:35.387 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:30:35.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:30:35.405 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:30:35.408 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 10, 2022-07-20T02:30:35.409 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:30:35.409 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:30:35.409 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:30:35.409 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:30:35.440 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:30:35.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.446 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:30:35.446 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:30:35.458 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:30:35.458 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:35.459 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:35.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:35.460 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:35.460 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:35.461 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:35.461 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:35.472 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:35.473 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:30:35.473 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:30:35.473 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:30:35.474 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:30:35.474 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:30:35.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:35.475 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:35.476 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:35.476 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:35.477 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:35.477 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:35.477 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:35.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.478 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:30:35.479 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:35.479 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:35.480 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:35.480 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:35.480 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:35.481 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:35.481 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:35.516 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:35.516 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:35.517 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:35.517 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:30:35.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:30:35.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.518 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:35.518 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:30:35.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:30:35.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:30:35.519 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:30:35.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:30:35.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:30:35.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:30:35.520 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:30:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:30:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:30:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:30:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:30:35.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:30:35.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:30:35.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:30:35.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:30:35.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:30:35.523 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:30:35.523 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:30:35.523 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:30:35.523 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:30:35.524 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:30:35.524 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:30:35.524 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:30:35.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.524 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:30:35.525 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:30:35.525 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:30:35.525 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:30:35.526 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:30:35.526 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:30:35.527 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:30:35.527 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:30:35.527 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:30:35.527 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:30:35.527 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:30:35.528 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:30:35.528 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:30:35.528 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:30:35.528 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:30:35.529 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:30:35.529 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:30:35.536 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:30:35.537 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:30:35.537 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:30:35.538 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:30:35.549 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:30:35.550 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:30:35.550 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:30:35.552 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:30:35.552 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:30:35.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:30:35.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.553 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:30:35.554 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:30:35.554 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:30:35.554 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:30:35.555 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:30:35.555 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:30:35.555 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:30:35.555 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:30:35.555 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:30:35.556 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:30:35.556 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:30:35.556 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:30:35.556 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:30:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:30:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:30:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:30:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:30:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:35.571 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.571 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:30:35.574 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:30:35.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.577 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.578 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:30:35.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:30:35.582 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.584 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:30:35.594 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:30:35.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.599 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:30:35.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:30:35.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:30:35.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:30:35.608 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:30:35.608 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:30:35.608 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:30:35.609 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.609 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.610 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.612 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:30:35.613 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:30:35.613 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.614 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.614 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.615 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.616 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.617 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:35.617 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.618 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.618 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:30:35.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:30:35.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:30:35.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:30:35.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:30:35.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:30:35.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:30:35.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:30:35.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:30:35.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:30:35.653 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:30:35.653 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:30:35.654 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:30:35.655 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.655 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.693 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:30:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:30:35.706 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.706 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.707 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.707 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.708 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.708 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.709 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.709 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 53, 2022-07-20T02:30:35.710 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.710 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:30:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:30:35.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.721 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:30:35.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:30:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.731 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:30:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:30:35.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:30:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:30:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:30:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:30:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:30:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:30:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:30:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:30:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:30:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:30:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:30:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:30:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:30:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:30:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:30:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:30:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:30:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:30:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:30:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:30:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:30:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:30:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:30:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:30:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:30:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:30:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:30:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:30:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:30:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:30:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:30:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:30:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:30:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:30:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:30:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:30:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:30:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:30:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:30:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:30:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:30:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:30:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:30:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:30:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:30:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:30:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:30:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:30:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:30:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:30:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:30:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:30:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:30:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:30:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:30:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 52, 2022-07-20T02:30:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:30:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:30:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:30:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:30:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:30:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:30:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:30:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:30:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:30:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:30:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:30:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:30:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:30:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:30:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:30:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:30:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:30:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:30:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:30:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:30:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:30:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:30:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:30:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:30:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:30:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:30:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:30:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:30:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:30:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:30:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:30:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:30:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:30:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:30:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:30:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:30:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:30:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:30:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:30:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:30:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:30:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:30:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:30:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:30:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:30:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:30:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:30:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:30:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:30:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:30:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:30:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:30:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:30:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:30:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:30:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:30:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:30:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:30:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:30:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:30:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:30:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:30:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:30:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:30:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:30:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:30:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:30:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:30:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:30:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:30:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:30:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:30:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:30:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:30:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:30:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:30:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:30:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/3215220955": "2022-07-21T02:30:29.085434+0000", 2022-07-20T02:30:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/3215220955": "2022-07-21T02:30:29.085434+0000" 2022-07-20T02:30:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:30:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:30:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:30:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:30:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:30:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:30:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:30:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:30:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:30:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:30:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:30:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:30:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:30:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:30:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:30:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:30:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:30:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:30:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:30:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:30:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:30:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:30:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:30:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:30:36.001 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:30:36.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 msgr2=0x7fca10055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 0x7fca10055fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fca1400b000 tx=0x7fca14006df0).stop 2022-07-20T02:30:36.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 msgr2=0x7fca2c151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fca2c0b80f0 tx=0x7fca1c014000).stop 2022-07-20T02:30:36.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 shutdown_connections 2022-07-20T02:30:36.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca2c147000 0x7fca2c151f40 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fca10053b00 0x7fca10055fc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca2c0b04a0 0x7fca2c1425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 --2- 172.21.15.26:0/3723396842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca2c060f00 0x7fca2c0b90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 >> 172.21.15.26:0/3723396842 conn(0x7fca2c05cfd0 msgr2=0x7fca2c0b97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 shutdown_connections 2022-07-20T02:30:36.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:35.372+0000 7fca31172700 1 -- 172.21.15.26:0/3723396842 wait complete. 2022-07-20T02:30:36.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.116+0000 7fb016836700 1 Processor -- start 2022-07-20T02:30:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.116+0000 7fb016836700 1 -- start start 2022-07-20T02:30:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb01005cfd0 0x7fb010143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb010147000 0x7fb0100b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 0x7fb01013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0101474e0 con 0x7fb010147000 2022-07-20T02:30:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01005d3b0 con 0x7fb010060f00 2022-07-20T02:30:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0101431e0 con 0x7fb01005cfd0 2022-07-20T02:30:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb010147000 0x7fb0100b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb010147000 0x7fb0100b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38352/0 (socket says 172.21.15.26:38352) 2022-07-20T02:30:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015834700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb01005cfd0 0x7fb010143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 -- 172.21.15.26:0/2357096645 learned_addr learned my addr 172.21.15.26:0/2357096645 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016035700 1 --2- 172.21.15.26:0/2357096645 >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 0x7fb01013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 -- 172.21.15.26:0/2357096645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb01005cfd0 msgr2=0x7fb010143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 --2- 172.21.15.26:0/2357096645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb01005cfd0 0x7fb010143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 -- 172.21.15.26:0/2357096645 >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 msgr2=0x7fb01013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 --2- 172.21.15.26:0/2357096645 >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 0x7fb01013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 -- 172.21.15.26:0/2357096645 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010143480 con 0x7fb010147000 2022-07-20T02:30:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016035700 1 --2- 172.21.15.26:0/2357096645 >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 0x7fb01013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb015033700 1 --2- 172.21.15.26:0/2357096645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b4730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fb004002f70 tx=0x7fb004006eb0).ready entity=mon.0 client_cookie=e9b22a5e39321b05 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb014832700 1 -- 172.21.15.26:0/2357096645 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb00400d570 con 0x7fb010147000 2022-07-20T02:30:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb014832700 1 -- 172.21.15.26:0/2357096645 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004000e90 con 0x7fb010147000 2022-07-20T02:30:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb014832700 1 -- 172.21.15.26:0/2357096645 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb00400d900 con 0x7fb010147000 2022-07-20T02:30:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- 172.21.15.26:0/2357096645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 msgr2=0x7fb0100b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- 172.21.15.26:0/2357096645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b4730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fb004002f70 tx=0x7fb004006eb0).stop 2022-07-20T02:30:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- 172.21.15.26:0/2357096645 shutdown_connections 2022-07-20T02:30:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- 172.21.15.26:0/2357096645 >> v2:172.21.15.120:3300/0 conn(0x7fb010060f00 0x7fb01013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- 172.21.15.26:0/2357096645 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b4730 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 --2- 172.21.15.26:0/2357096645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb01005cfd0 0x7fb010143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- 172.21.15.26:0/2357096645 >> 172.21.15.26:0/2357096645 conn(0x7fb0100620c0 msgr2=0x7fb01005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- 172.21.15.26:0/2357096645 shutdown_connections 2022-07-20T02:30:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.120+0000 7fb016836700 1 -- 172.21.15.26:0/2357096645 wait complete. 2022-07-20T02:30:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 Processor -- start 2022-07-20T02:30:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- start start 2022-07-20T02:30:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb010060f00 0x7fb0101558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0100b7850 0x7fb010148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0100ba260 con 0x7fb010147000 2022-07-20T02:30:36.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb010156770 con 0x7fb0100b7850 2022-07-20T02:30:36.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0100ba490 con 0x7fb010060f00 2022-07-20T02:30:36.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38356/0 (socket says 172.21.15.26:38356) 2022-07-20T02:30:36.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 -- 172.21.15.26:0/1338399622 learned_addr learned my addr 172.21.15.26:0/1338399622 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:36.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015834700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb010060f00 0x7fb0101558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016035700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0100b7850 0x7fb010148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 -- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb010060f00 msgr2=0x7fb0101558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb010060f00 0x7fb0101558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 -- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0100b7850 msgr2=0x7fb010148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0100b7850 0x7fb010148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 -- 172.21.15.26:0/1338399622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00800b180 con 0x7fb010147000 2022-07-20T02:30:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb015033700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fb004002f70 tx=0x7fb004015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb004011760 con 0x7fb010147000 2022-07-20T02:30:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0040118c0 con 0x7fb010147000 2022-07-20T02:30:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00400b1e0 con 0x7fb010147000 2022-07-20T02:30:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb004011c20 con 0x7fb010147000 2022-07-20T02:30:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.124+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb010149940 con 0x7fb010147000 2022-07-20T02:30:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0100ba620 con 0x7fb010147000 2022-07-20T02:30:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb004025030 con 0x7fb010147000 2022-07-20T02:30:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb00effd700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 0x7faff0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb00406ce10 con 0x7fb010147000 2022-07-20T02:30:36.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb015834700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 0x7faff0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.128+0000 7fb015834700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 0x7faff0055e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fb00000b000 tx=0x7fb0000091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.136+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb004043eb0 con 0x7fb010147000 2022-07-20T02:30:36.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.300+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb010143720 con 0x7fb010147000 2022-07-20T02:30:36.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.300+0000 7fb00effd700 1 -- 172.21.15.26:0/1338399622 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+1562 (secure 0 0 0) 0x7fb004023070 con 0x7fb010147000 2022-07-20T02:30:36.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:36.304 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5369,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/4159318767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4159318767},{"type":"v1","addr":"172.21.15.26:6837","nonce":4159318767}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26},{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[]} 2022-07-20T02:30:36.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 msgr2=0x7faff0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 0x7faff0055e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fb00000b000 tx=0x7fb0000091d0).stop 2022-07-20T02:30:36.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 msgr2=0x7fb0100b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fb004002f70 tx=0x7fb004015000).stop 2022-07-20T02:30:36.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 shutdown_connections 2022-07-20T02:30:36.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb010147000 0x7fb0100b8660 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faff0053950 0x7faff0055e10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0100b7850 0x7fb010148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 --2- 172.21.15.26:0/1338399622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb010060f00 0x7fb0101558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 >> 172.21.15.26:0/1338399622 conn(0x7fb01005cfd0 msgr2=0x7fb010142a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 shutdown_connections 2022-07-20T02:30:36.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.304+0000 7fb016836700 1 -- 172.21.15.26:0/1338399622 wait complete. 2022-07-20T02:30:36.324 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 26 2022-07-20T02:30:36.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.432+0000 7f0c0dd1e700 1 Processor -- start 2022-07-20T02:30:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.432+0000 7f0c0dd1e700 1 -- start start 2022-07-20T02:30:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080b5ef0 0x7f0c080b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0c080b1850 0x7f0c08142db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080b62d0 con 0x7f0c080a97e0 2022-07-20T02:30:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c080a9cf0 con 0x7f0c080b1850 2022-07-20T02:30:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080a9f90 con 0x7f0c080b5ef0 2022-07-20T02:30:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c07fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0c080a97e0 0x7f0c081426a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37644/0 (socket says 172.21.15.26:37644) 2022-07-20T02:30:36.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c07fff700 1 -- 172.21.15.26:0/2540056732 learned_addr learned my addr 172.21.15.26:0/2540056732 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/2540056732 >> v2:172.21.15.26:3300/0 conn(0x7f0c080b1850 0x7f0c08142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0cd1c700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080b5ef0 0x7f0c080b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c077fe700 1 -- 172.21.15.26:0/2540056732 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174738914 0 0) 0x7f0c080b62d0 con 0x7f0c080a97e0 2022-07-20T02:30:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c077fe700 1 -- 172.21.15.26:0/2540056732 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bec003260 con 0x7f0c080a97e0 2022-07-20T02:30:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 -- 172.21.15.26:0/2540056732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080b5ef0 msgr2=0x7f0c080b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080b5ef0 0x7f0c080b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 -- 172.21.15.26:0/2540056732 >> v1:172.21.15.26:6789/0 conn(0x7f0c080a97e0 legacy=0x7f0c081426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 -- 172.21.15.26:0/2540056732 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c080aa230 con 0x7f0c080b1850 2022-07-20T02:30:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b1850 0x7f0c08142db0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f0c000032f0 tx=0x7f0c000091d0).ready entity=mon.0 client_cookie=b6ad485c7a2108cf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c077fe700 1 -- 172.21.15.26:0/2540056732 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c0000b6c0 con 0x7f0c080b1850 2022-07-20T02:30:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c077fe700 1 -- 172.21.15.26:0/2540056732 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0000b820 con 0x7f0c080b1850 2022-07-20T02:30:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c077fe700 1 -- 172.21.15.26:0/2540056732 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c0000b990 con 0x7f0c080b1850 2022-07-20T02:30:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/2540056732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b1850 msgr2=0x7f0c08142db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b1850 0x7f0c08142db0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f0c000032f0 tx=0x7f0c000091d0).stop 2022-07-20T02:30:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/2540056732 shutdown_connections 2022-07-20T02:30:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b1850 0x7f0c08142db0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/2540056732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080b5ef0 0x7f0c080b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/2540056732 >> 172.21.15.26:0/2540056732 conn(0x7f0c08060310 msgr2=0x7f0c080b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/2540056732 shutdown_connections 2022-07-20T02:30:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.436+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/2540056732 wait complete. 2022-07-20T02:30:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 Processor -- start 2022-07-20T02:30:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- start start 2022-07-20T02:30:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 0x7f0c080adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c080b1850 0x7f0c08067000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08066170 con 0x7f0c080b5ef0 2022-07-20T02:30:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080674f0 con 0x7f0c080b1850 2022-07-20T02:30:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08152430 con 0x7f0c080a97e0 2022-07-20T02:30:36.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38372/0 (socket says 172.21.15.26:38372) 2022-07-20T02:30:36.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 -- 172.21.15.26:0/187445074 learned_addr learned my addr 172.21.15.26:0/187445074 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:36.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c07fff700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c080b1850 0x7f0c08067000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0cd1c700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 0x7f0c080adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 -- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 msgr2=0x7f0c080adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 0x7f0c080adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 -- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c080b1850 msgr2=0x7f0c08067000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c080b1850 0x7f0c08067000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 -- 172.21.15.26:0/187445074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00009470 con 0x7f0c080b5ef0 2022-07-20T02:30:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0cd1c700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 0x7f0c080adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0d51d700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f0c000032f0 tx=0x7f0c00016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c00011e20 con 0x7f0c080b5ef0 2022-07-20T02:30:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00021440 con 0x7f0c080b5ef0 2022-07-20T02:30:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c080663a0 con 0x7f0c080b5ef0 2022-07-20T02:30:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c000217a0 con 0x7f0c080b5ef0 2022-07-20T02:30:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.440+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c08065b40 con 0x7f0c080b5ef0 2022-07-20T02:30:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0809eca0 con 0x7f0c080b5ef0 2022-07-20T02:30:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0c00027020 con 0x7f0c080b5ef0 2022-07-20T02:30:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c05ffb700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 0x7f0be8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0c0006d100 con 0x7f0c080b5ef0 2022-07-20T02:30:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c0cd1c700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 0x7f0be8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.444+0000 7f0c0cd1c700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 0x7f0be8055f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f0bfc00c000 tx=0x7f0bfc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.448+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c000441e0 con 0x7f0c080b5ef0 2022-07-20T02:30:36.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.612+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c080afee0 con 0x7f0c080b5ef0 2022-07-20T02:30:36.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.612+0000 7f0c05ffb700 1 -- 172.21.15.26:0/187445074 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f0c00015020 con 0x7f0c080b5ef0 2022-07-20T02:30:36.613 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:36.622 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5369,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/4159318767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":4159318767},{"type":"v1","addr":"172.21.15.26:6837","nonce":4159318767}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26},{"gid":5598,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/4218530195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4218530195},{"type":"v1","addr":"172.21.15.120:6833","nonce":4218530195}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/870919280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":870919280},{"type":"v1","addr":"172.21.15.26:6835","nonce":870919280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[]} 2022-07-20T02:30:36.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 msgr2=0x7f0be8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 0x7f0be8055f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f0bfc00c000 tx=0x7f0bfc008040).stop 2022-07-20T02:30:36.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 msgr2=0x7f0c08065650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f0c000032f0 tx=0x7f0c00016000).stop 2022-07-20T02:30:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 shutdown_connections 2022-07-20T02:30:36.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c080b5ef0 0x7f0c08065650 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0be8053a70 0x7f0be8055f30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c080b1850 0x7f0c08067000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 --2- 172.21.15.26:0/187445074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c080a97e0 0x7f0c080adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 >> 172.21.15.26:0/187445074 conn(0x7f0c08060310 msgr2=0x7f0c0805cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 shutdown_connections 2022-07-20T02:30:36.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.616+0000 7f0c0dd1e700 1 -- 172.21.15.26:0/187445074 wait complete. 2022-07-20T02:30:36.641 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 26 2022-07-20T02:30:36.644 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:30:36.645 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:30:36.645 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:30:36.676 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:30:36.677 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:30:36.677 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:30:36.678 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:30:36.679 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:30:36.679 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:30:36.701 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:30:36.702 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:30:36.702 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:30:36.724 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:30:36.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:30:36.751 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:30:36.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:30:36.756 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:30:36.756 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:30:36.770 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:30:36.772 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:30:36.772 DEBUG:teuthology.parallel:result is None 2022-07-20T02:30:36.773 DEBUG:teuthology.parallel:result is None 2022-07-20T02:30:36.775 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:30:36.776 DEBUG:teuthology.parallel:result is None 2022-07-20T02:30:36.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:30:36.842 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:30:36.845 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:30:36.865 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:30:36.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 Processor -- start 2022-07-20T02:30:36.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- start start 2022-07-20T02:30:36.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f22f0067520 0x7f22f0067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f22f00b22c0 0x7f22f0097350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f0096c80 con 0x7f22f00b5600 2022-07-20T02:30:36.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f0098070 con 0x7f22f00b22c0 2022-07-20T02:30:36.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f00b48a0 con 0x7f22f0067520 2022-07-20T02:30:36.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f22f00b22c0 0x7f22f0097350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22edd9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f22f00b5600 0x7f22f0097960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37682/0 (socket says 172.21.15.26:37682) 2022-07-20T02:30:36.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22edd9b700 1 -- 172.21.15.26:0/100734122 learned_addr learned my addr 172.21.15.26:0/100734122 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:36.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22ee59c700 1 --2- 172.21.15.26:0/100734122 >> v2:172.21.15.120:3300/0 conn(0x7f22f0067520 0x7f22f0067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 -- 172.21.15.26:0/100734122 >> v2:172.21.15.120:3300/0 conn(0x7f22f0067520 msgr2=0x7f22f0067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:36.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22ed59a700 1 -- 172.21.15.26:0/100734122 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323993329 0 0) 0x7f22f0096c80 con 0x7f22f00b5600 2022-07-20T02:30:36.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 --2- 172.21.15.26:0/100734122 >> v2:172.21.15.120:3300/0 conn(0x7f22f0067520 0x7f22f0067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 -- 172.21.15.26:0/100734122 >> v1:172.21.15.26:6789/0 conn(0x7f22f00b5600 legacy=0x7f22f0097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 -- 172.21.15.26:0/100734122 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f00b4b40 con 0x7f22f00b22c0 2022-07-20T02:30:36.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22eed9d700 1 --2- 172.21.15.26:0/100734122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b22c0 0x7f22f0097350 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f22e80032f0 tx=0x7f22e80091d0).ready entity=mon.0 client_cookie=1e4d51e0d3402ec6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22ed59a700 1 -- 172.21.15.26:0/100734122 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22e800b680 con 0x7f22f00b22c0 2022-07-20T02:30:36.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22ed59a700 1 -- 172.21.15.26:0/100734122 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800b7e0 con 0x7f22f00b22c0 2022-07-20T02:30:36.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22ed59a700 1 -- 172.21.15.26:0/100734122 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22e800b950 con 0x7f22f00b22c0 2022-07-20T02:30:36.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/100734122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b22c0 msgr2=0x7f22f0097350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/100734122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b22c0 0x7f22f0097350 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f22e80032f0 tx=0x7f22e80091d0).stop 2022-07-20T02:30:36.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/100734122 shutdown_connections 2022-07-20T02:30:36.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/100734122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b22c0 0x7f22f0097350 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/100734122 >> v2:172.21.15.120:3300/0 conn(0x7f22f0067520 0x7f22f0067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/100734122 >> 172.21.15.26:0/100734122 conn(0x7f22f00b5ea0 msgr2=0x7f22f00b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:36.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/100734122 shutdown_connections 2022-07-20T02:30:36.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/100734122 wait complete. 2022-07-20T02:30:36.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 Processor -- start 2022-07-20T02:30:36.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- start start 2022-07-20T02:30:36.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22f0067520 0x7f22f0093870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22f00b22c0 0x7f22f009aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0152770 con 0x7f22f00b5600 2022-07-20T02:30:36.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00931d0 con 0x7f22f0067520 2022-07-20T02:30:36.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0092c90 con 0x7f22f00b22c0 2022-07-20T02:30:36.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38408/0 (socket says 172.21.15.26:38408) 2022-07-20T02:30:36.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 -- 172.21.15.26:0/2635486920 learned_addr learned my addr 172.21.15.26:0/2635486920 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:36.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22edd9b700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22f00b22c0 0x7f22f009aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22ee59c700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22f0067520 0x7f22f0093870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 -- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22f00b22c0 msgr2=0x7f22f009aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22f00b22c0 0x7f22f009aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 -- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22f0067520 msgr2=0x7f22f0093870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:36.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22f0067520 0x7f22f0093870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:36.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 -- 172.21.15.26:0/2635486920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e8009470 con 0x7f22f00b5600 2022-07-20T02:30:36.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22eed9d700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f22e80032f0 tx=0x7f22e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22e800b680 con 0x7f22f00b5600 2022-07-20T02:30:36.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8021630 con 0x7f22f00b5600 2022-07-20T02:30:36.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f0092f30 con 0x7f22f00b5600 2022-07-20T02:30:36.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22e8021990 con 0x7f22f00b5600 2022-07-20T02:30:36.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f00a9cb0 con 0x7f22f00b5600 2022-07-20T02:30:36.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0096eb0 con 0x7f22f00b5600 2022-07-20T02:30:36.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f22e8027020 con 0x7f22f00b5600 2022-07-20T02:30:36.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22df7fe700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 0x7f22cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:36.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f22e806d500 con 0x7f22f00b5600 2022-07-20T02:30:36.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22ee59c700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 0x7f22cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:36.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.904+0000 7f22ee59c700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 0x7f22cc055f60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f22e000b000 tx=0x7f22e0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:36.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:36.912+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f22e80445a0 con 0x7f22f00b5600 2022-07-20T02:30:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.048+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f22f00574e0 con 0x7f22f00b5600 2022-07-20T02:30:37.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.896+0000 7f22df7fe700 1 -- 172.21.15.26:0/2635486920 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f22e8015020 con 0x7f22f00b5600 2022-07-20T02:30:37.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 msgr2=0x7f22cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:37.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.896+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 0x7f22cc055f60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f22e000b000 tx=0x7f22e0006df0).stop 2022-07-20T02:30:37.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.896+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 msgr2=0x7f22f0152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:37.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.896+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f22e80032f0 tx=0x7f22e8016000).stop 2022-07-20T02:30:37.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 shutdown_connections 2022-07-20T02:30:37.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f22f00b5600 0x7f22f0152bd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:37.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f22cc053aa0 0x7f22cc055f60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:37.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22f00b22c0 0x7f22f009aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:37.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 --2- 172.21.15.26:0/2635486920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22f0067520 0x7f22f0093870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:37.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 >> 172.21.15.26:0/2635486920 conn(0x7f22f00b5ea0 msgr2=0x7f22f00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:37.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 shutdown_connections 2022-07-20T02:30:37.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:37.900+0000 7f22f4ea3700 1 -- 172.21.15.26:0/2635486920 wait complete. 2022-07-20T02:30:37.906 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:30:37.917 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:30:37.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:30:38.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 Processor -- start 2022-07-20T02:30:38.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 -- start start 2022-07-20T02:30:38.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bd00a7fc0 0x7f8bd01479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00b62d0 con 0x7f8bd00a8c40 2022-07-20T02:30:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd0147eb0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd014aac0 con 0x7f8bd00b5ef0 2022-07-20T02:30:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bd00a7fc0 0x7f8bd01479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8bd00a7fc0 0x7f8bd01479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38416/0 (socket says 172.21.15.26:38416) 2022-07-20T02:30:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 -- 172.21.15.26:0/1002721913 learned_addr learned my addr 172.21.15.26:0/1002721913 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bce59c700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 -- 172.21.15.26:0/1002721913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 msgr2=0x7f8bd01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd01474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 -- 172.21.15.26:0/1002721913 >> v1:172.21.15.26:6789/0 conn(0x7f8bd00a8c40 legacy=0x7f8bd00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 -- 172.21.15.26:0/1002721913 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd014ad60 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bce59c700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:30:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcdd9b700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd01479c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f8bc00032f0 tx=0x7f8bc0009290).ready entity=mon.0 client_cookie=cf0074b791df085d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcd59a700 1 -- 172.21.15.26:0/1002721913 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc000b8a0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcd59a700 1 -- 172.21.15.26:0/1002721913 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc000ba00 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.036+0000 7f8bcd59a700 1 -- 172.21.15.26:0/1002721913 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc000bb70 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/1002721913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 msgr2=0x7f8bd01479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd01479c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f8bc00032f0 tx=0x7f8bc0009290).stop 2022-07-20T02:30:38.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/1002721913 shutdown_connections 2022-07-20T02:30:38.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd01479c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/1002721913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/1002721913 >> 172.21.15.26:0/1002721913 conn(0x7f8bd0060310 msgr2=0x7f8bd00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/1002721913 shutdown_connections 2022-07-20T02:30:38.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/1002721913 wait complete. 2022-07-20T02:30:38.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 Processor -- start 2022-07-20T02:30:38.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- start start 2022-07-20T02:30:38.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd00a8c40 0x7f8bd0149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0067ae0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0067180 con 0x7f8bd00a8c40 2022-07-20T02:30:38.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0067420 con 0x7f8bd00b5ef0 2022-07-20T02:30:38.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38422/0 (socket says 172.21.15.26:38422) 2022-07-20T02:30:38.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 -- 172.21.15.26:0/786748824 learned_addr learned my addr 172.21.15.26:0/786748824 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcdd9b700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd00a8c40 0x7f8bd0149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bced9d700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 -- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 msgr2=0x7f8bd0156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd0156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 -- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd00a8c40 msgr2=0x7f8bd0149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd00a8c40 0x7f8bd0149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 -- 172.21.15.26:0/786748824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc0009530 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bce59c700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8bc4007000 tx=0x7f8bc4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc400ce80 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4010580 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.040+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd00676c0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc40108e0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd00626f0 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd0066f40 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8bc4010a40 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bbf7fe700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 0x7f8bac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f8bc406ba50 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bcdd9b700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 0x7f8bac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.044+0000 7f8bcdd9b700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 0x7f8bac055ea0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8bc001b000 tx=0x7f8bc0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.052+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8bd0066f40 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8bd0066f40 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bbf7fe700 1 -- 172.21.15.26:0/786748824 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f8bd0066f40 con 0x7f8bd00a7fc0 2022-07-20T02:30:38.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:38.229 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:30:38.229 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:30:38.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.230 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:30:38.230 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:30:38.231 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:30:38.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.232 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.232 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.233 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:30:38.234 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:30:38.234 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:30:38.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.235 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.235 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.236 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:30:38.236 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:30:38.236 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.236 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:30:38.237 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.237 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.237 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.241 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:30:38.242 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:30:38.242 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.243 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.244 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:30:38.244 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:30:38.245 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.245 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.246 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.246 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.264 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:30:38.265 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:30:38.265 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.270 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.270 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.271 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.272 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:30:38.272 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:30:38.272 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.272 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.273 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.273 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.274 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:30:38.274 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:30:38.274 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.274 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.276 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:30:38.276 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:30:38.276 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.276 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.280 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.280 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.281 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:30:38.282 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:30:38.282 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.282 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.282 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.283 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.284 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:30:38.284 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:30:38.284 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.284 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.285 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:30:38.286 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:30:38.287 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.287 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:30:38.287 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:30:38.287 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:30:38.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.288 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:30:38.289 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:30:38.289 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.289 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:30:38.289 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:30:38.290 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:30:38.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.291 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:30:38.292 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:30:38.292 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.292 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:30:38.292 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:30:38.293 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:30:38.293 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.294 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:30:38.294 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:30:38.295 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:30:38.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:30:38.295 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:38.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:38.297 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:30:38.297 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:30:38.297 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:30:38.298 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:30:38.298 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:30:38.299 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:30:38.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:38.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:30:38.300 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:30:38.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 msgr2=0x7f8bac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 0x7f8bac055ea0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8bc001b000 tx=0x7f8bc0016000).stop 2022-07-20T02:30:38.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 msgr2=0x7f8bd0063c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8bc4007000 tx=0x7f8bc4008040).stop 2022-07-20T02:30:38.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 shutdown_connections 2022-07-20T02:30:38.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8bac0539e0 0x7f8bac055ea0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8bd00b5ef0 0x7f8bd0156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd00a8c40 0x7f8bd0149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 --2- 172.21.15.26:0/786748824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8bd00a7fc0 0x7f8bd0063c10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 >> 172.21.15.26:0/786748824 conn(0x7f8bd0060310 msgr2=0x7f8bd00af380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 shutdown_connections 2022-07-20T02:30:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.216+0000 7f8bcf59e700 1 -- 172.21.15.26:0/786748824 wait complete. 2022-07-20T02:30:38.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:30:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 Processor -- start 2022-07-20T02:30:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 -- start start 2022-07-20T02:30:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71d4147be0 0x7f71d40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f71d40a7fc0 0x7f71d40af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d40b62d0 con 0x7f71d40b5ef0 2022-07-20T02:30:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d414aac0 con 0x7f71d40a7fc0 2022-07-20T02:30:38.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d3fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d414ad60 con 0x7f71d4147be0 2022-07-20T02:30:38.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f71d40a7fc0 0x7f71d40af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f71d40a7fc0 0x7f71d40af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38434/0 (socket says 172.21.15.26:38434) 2022-07-20T02:30:38.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 -- 172.21.15.26:0/4269231848 learned_addr learned my addr 172.21.15.26:0/4269231848 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d27fc700 1 --2- 172.21.15.26:0/4269231848 >> v2:172.21.15.120:3300/0 conn(0x7f71d4147be0 0x7f71d40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 -- 172.21.15.26:0/4269231848 >> v2:172.21.15.120:3300/0 conn(0x7f71d4147be0 msgr2=0x7f71d40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 --2- 172.21.15.26:0/4269231848 >> v2:172.21.15.120:3300/0 conn(0x7f71d4147be0 0x7f71d40aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 -- 172.21.15.26:0/4269231848 >> v1:172.21.15.120:6789/0 conn(0x7f71d40b5ef0 legacy=0x7f71d41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 -- 172.21.15.26:0/4269231848 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d414b000 con 0x7f71d40a7fc0 2022-07-20T02:30:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d37fe700 1 --2- 172.21.15.26:0/4269231848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40af330 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f71cc0032f0 tx=0x7f71cc009290).ready entity=mon.0 client_cookie=6fbc556bac286e6e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d1ffb700 1 -- 172.21.15.26:0/4269231848 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc00b870 con 0x7f71d40a7fc0 2022-07-20T02:30:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d1ffb700 1 -- 172.21.15.26:0/4269231848 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc00b9d0 con 0x7f71d40a7fc0 2022-07-20T02:30:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.420+0000 7f71d1ffb700 1 -- 172.21.15.26:0/4269231848 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc00bb40 con 0x7f71d40a7fc0 2022-07-20T02:30:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- 172.21.15.26:0/4269231848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 msgr2=0x7f71d40af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- 172.21.15.26:0/4269231848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40af330 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f71cc0032f0 tx=0x7f71cc009290).stop 2022-07-20T02:30:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- 172.21.15.26:0/4269231848 shutdown_connections 2022-07-20T02:30:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- 172.21.15.26:0/4269231848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40af330 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- 172.21.15.26:0/4269231848 >> v2:172.21.15.120:3300/0 conn(0x7f71d4147be0 0x7f71d40aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- 172.21.15.26:0/4269231848 >> 172.21.15.26:0/4269231848 conn(0x7f71d4060310 msgr2=0x7f71d40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- 172.21.15.26:0/4269231848 shutdown_connections 2022-07-20T02:30:38.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- 172.21.15.26:0/4269231848 wait complete. 2022-07-20T02:30:38.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 Processor -- start 2022-07-20T02:30:38.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- start start 2022-07-20T02:30:38.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40b5ef0 0x7f71d4156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d4147be0 0x7f71d40b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40aacc0 con 0x7f71d40a7fc0 2022-07-20T02:30:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d41566a0 con 0x7f71d4147be0 2022-07-20T02:30:38.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40abbc0 con 0x7f71d40b5ef0 2022-07-20T02:30:38.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38436/0 (socket says 172.21.15.26:38436) 2022-07-20T02:30:38.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d2ffd700 1 -- 172.21.15.26:0/3697276591 learned_addr learned my addr 172.21.15.26:0/3697276591 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d27fc700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40b5ef0 0x7f71d4156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.424+0000 7f71d37fe700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d4147be0 0x7f71d40b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 -- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40b5ef0 msgr2=0x7f71d4156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40b5ef0 0x7f71d4156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 -- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d4147be0 msgr2=0x7f71d40b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d4147be0 0x7f71d40b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 -- 172.21.15.26:0/3697276591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc009530 con 0x7f71d40a7fc0 2022-07-20T02:30:38.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d2ffd700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f71d4060810 tx=0x7f71c8007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c8007030 con 0x7f71d40a7fc0 2022-07-20T02:30:38.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8007190 con 0x7f71d40a7fc0 2022-07-20T02:30:38.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d40abe60 con 0x7f71d40a7fc0 2022-07-20T02:30:38.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c8006510 con 0x7f71d40a7fc0 2022-07-20T02:30:38.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d4157460 con 0x7f71d40a7fc0 2022-07-20T02:30:38.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d4157680 con 0x7f71d40a7fc0 2022-07-20T02:30:38.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f71c8006670 con 0x7f71d40a7fc0 2022-07-20T02:30:38.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 0x7f71b0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f71c806b590 con 0x7f71d40a7fc0 2022-07-20T02:30:38.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.428+0000 7f71d27fc700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 0x7f71b0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.432+0000 7f71d27fc700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 0x7f71b0055f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f71c400b000 tx=0x7f71c4006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.436+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71c8042630 con 0x7f71d40a7fc0 2022-07-20T02:30:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.564+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71d40ac300 con 0x7f71d40a7fc0 2022-07-20T02:30:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.564+0000 7f71c3fff700 1 -- 172.21.15.26:0/3697276591 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13463 (secure 0 0 0) 0x7f71d40ac300 con 0x7f71d40a7fc0 2022-07-20T02:30:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:38.578 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":61,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:30:37.880161+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:30:38.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 msgr2=0x7f71b0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 0x7f71b0055f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f71c400b000 tx=0x7f71c4006df0).stop 2022-07-20T02:30:38.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 msgr2=0x7f71d40aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f71d4060810 tx=0x7f71c8007c00).stop 2022-07-20T02:30:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 shutdown_connections 2022-07-20T02:30:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b0053aa0 0x7f71b0055f60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d4147be0 0x7f71d40b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40b5ef0 0x7f71d4156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 --2- 172.21.15.26:0/3697276591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d40a7fc0 0x7f71d40aa620 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 >> 172.21.15.26:0/3697276591 conn(0x7f71d4060310 msgr2=0x7f71d4067870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 shutdown_connections 2022-07-20T02:30:38.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.568+0000 7f71d3fff700 1 -- 172.21.15.26:0/3697276591 wait complete. 2022-07-20T02:30:38.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:30:38.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:30:38.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 Processor -- start 2022-07-20T02:30:38.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 -- start start 2022-07-20T02:30:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 0x7f263c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f263c0a8c40 0x7f263c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0b62d0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c14aac0 con 0x7f263c0a8c40 2022-07-20T02:30:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642844700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c14ad60 con 0x7f263c0b5ef0 2022-07-20T02:30:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642043700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f263c0a8c40 0x7f263c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642043700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f263c0a8c40 0x7f263c0af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38446/0 (socket says 172.21.15.26:38446) 2022-07-20T02:30:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2642043700 1 -- 172.21.15.26:0/435654135 learned_addr learned my addr 172.21.15.26:0/435654135 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.688+0000 7f2641842700 1 --2- 172.21.15.26:0/435654135 >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 0x7f263c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642043700 1 -- 172.21.15.26:0/435654135 >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 msgr2=0x7f263c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642043700 1 --2- 172.21.15.26:0/435654135 >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 0x7f263c1474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642043700 1 -- 172.21.15.26:0/435654135 >> v1:172.21.15.120:6789/0 conn(0x7f263c0a7fc0 legacy=0x7f263c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642043700 1 -- 172.21.15.26:0/435654135 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c14b000 con 0x7f263c0a8c40 2022-07-20T02:30:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2641842700 1 --2- 172.21.15.26:0/435654135 >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 0x7f263c1474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642043700 1 --2- 172.21.15.26:0/435654135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a8c40 0x7f263c0af550 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f263c060810 tx=0x7f262c006eb0).ready entity=mon.0 client_cookie=b747b671ad19dc25 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2640840700 1 -- 172.21.15.26:0/435654135 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f262c00d5c0 con 0x7f263c0a8c40 2022-07-20T02:30:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2640840700 1 -- 172.21.15.26:0/435654135 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c000e90 con 0x7f263c0a8c40 2022-07-20T02:30:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2640840700 1 -- 172.21.15.26:0/435654135 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f262c00d910 con 0x7f263c0a8c40 2022-07-20T02:30:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- 172.21.15.26:0/435654135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a8c40 msgr2=0x7f263c0af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 --2- 172.21.15.26:0/435654135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a8c40 0x7f263c0af550 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f263c060810 tx=0x7f262c006eb0).stop 2022-07-20T02:30:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- 172.21.15.26:0/435654135 shutdown_connections 2022-07-20T02:30:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 --2- 172.21.15.26:0/435654135 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a8c40 0x7f263c0af550 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 --2- 172.21.15.26:0/435654135 >> v2:172.21.15.120:3300/0 conn(0x7f263c0b5ef0 0x7f263c1474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- 172.21.15.26:0/435654135 >> 172.21.15.26:0/435654135 conn(0x7f263c060310 msgr2=0x7f263c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- 172.21.15.26:0/435654135 shutdown_connections 2022-07-20T02:30:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- 172.21.15.26:0/435654135 wait complete. 2022-07-20T02:30:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 Processor -- start 2022-07-20T02:30:38.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 -- start start 2022-07-20T02:30:38.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.692+0000 7f2642844700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 0x7f263c0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 0x7f263c0aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0ac3e0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c144c80 con 0x7f263c0a8c40 2022-07-20T02:30:38.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c148250 con 0x7f263c0b5ef0 2022-07-20T02:30:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38450/0 (socket says 172.21.15.26:38450) 2022-07-20T02:30:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 -- 172.21.15.26:0/800660311 learned_addr learned my addr 172.21.15.26:0/800660311 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642043700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 0x7f263c0aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641041700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 0x7f263c0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 -- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 msgr2=0x7f263c0aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 0x7f263c0aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 -- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 msgr2=0x7f263c0aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 0x7f263c0aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 -- 172.21.15.26:0/800660311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c00b1e0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642043700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 0x7f263c0aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:30:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641041700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 0x7f263c0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:30:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2641842700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f2634006cc0 tx=0x7f2634008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f263400b7f0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263400b950 con 0x7f263c0a7fc0 2022-07-20T02:30:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c15a7e0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f263400bcb0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c0ab6f0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c14b2a0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.696+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2634012420 con 0x7f263c0a7fc0 2022-07-20T02:30:38.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.700+0000 7f263affd700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 0x7f261c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.700+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f263406bf80 con 0x7f263c0a7fc0 2022-07-20T02:30:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.700+0000 7f2641041700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 0x7f261c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.700+0000 7f2641041700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 0x7f261c055f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f2630008000 tx=0x7f2630009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.704+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2634043020 con 0x7f263c0a7fc0 2022-07-20T02:30:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f263c0574e0 con 0x7f263c0a7fc0 2022-07-20T02:30:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f263affd700 1 -- 172.21.15.26:0/800660311 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v61) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2634046640 con 0x7f263c0a7fc0 2022-07-20T02:30:38.835 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:30:38.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 msgr2=0x7f261c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 0x7f261c055f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f2630008000 tx=0x7f2630009040).stop 2022-07-20T02:30:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 msgr2=0x7f263c147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f2634006cc0 tx=0x7f2634008040).stop 2022-07-20T02:30:38.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 shutdown_connections 2022-07-20T02:30:38.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f261c053a70 0x7f261c055f30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0b5ef0 0x7f263c0aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0a8c40 0x7f263c0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 --2- 172.21.15.26:0/800660311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f263c0a7fc0 0x7f263c147800 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 >> 172.21.15.26:0/800660311 conn(0x7f263c060310 msgr2=0x7f263c1496b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 shutdown_connections 2022-07-20T02:30:38.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.832+0000 7f2642844700 1 -- 172.21.15.26:0/800660311 wait complete. 2022-07-20T02:30:38.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:30:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.956+0000 7f7deeb2a700 1 Processor -- start 2022-07-20T02:30:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.956+0000 7f7deeb2a700 1 -- start start 2022-07-20T02:30:38.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80618a0 0x7f7de8147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7de80b22d0 0x7f7de8143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de8061c80 con 0x7f7de8143d60 2022-07-20T02:30:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de81441f0 con 0x7f7de80b22d0 2022-07-20T02:30:38.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80b2810 con 0x7f7de80618a0 2022-07-20T02:30:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7de80b22d0 0x7f7de8143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7dedb28700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80618a0 0x7f7de8147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:38.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7de80b22d0 0x7f7de8143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58152/0 (socket says 172.21.15.26:58152) 2022-07-20T02:30:38.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 -- 172.21.15.26:0/459539529 learned_addr learned my addr 172.21.15.26:0/459539529 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:38.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 -- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80618a0 msgr2=0x7f7de8147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:38.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 --2- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80618a0 0x7f7de8147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 -- 172.21.15.26:0/459539529 >> v1:172.21.15.120:6789/0 conn(0x7f7de8143d60 legacy=0x7f7de80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 -- 172.21.15.26:0/459539529 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de80b2ab0 con 0x7f7de80b22d0 2022-07-20T02:30:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7ded327700 1 --2- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de80b22d0 0x7f7de8143870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7dd8002f70 tx=0x7f7dd8006eb0).ready entity=mon.1 client_cookie=a21d2e8fcb459775 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:38.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7decb26700 1 -- 172.21.15.26:0/459539529 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd800d570 con 0x7f7de80b22d0 2022-07-20T02:30:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7decb26700 1 -- 172.21.15.26:0/459539529 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8000e90 con 0x7f7de80b22d0 2022-07-20T02:30:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7decb26700 1 -- 172.21.15.26:0/459539529 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd800d900 con 0x7f7de80b22d0 2022-07-20T02:30:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de80b22d0 msgr2=0x7f7de8143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de80b22d0 0x7f7de8143870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7dd8002f70 tx=0x7f7dd8006eb0).stop 2022-07-20T02:30:38.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- 172.21.15.26:0/459539529 shutdown_connections 2022-07-20T02:30:38.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de80b22d0 0x7f7de8143870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/459539529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80618a0 0x7f7de8147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:38.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.960+0000 7f7deeb2a700 1 -- 172.21.15.26:0/459539529 >> 172.21.15.26:0/459539529 conn(0x7f7de8066f70 msgr2=0x7f7de8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:38.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- 172.21.15.26:0/459539529 shutdown_connections 2022-07-20T02:30:38.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- 172.21.15.26:0/459539529 wait complete. 2022-07-20T02:30:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 Processor -- start 2022-07-20T02:30:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- start start 2022-07-20T02:30:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80b22d0 0x7f7de8152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de8143d60 0x7f7de8153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8152750 con 0x7f7de80618a0 2022-07-20T02:30:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80b5a90 con 0x7f7de8143d60 2022-07-20T02:30:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7deeb2a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80b5d30 con 0x7f7de80b22d0 2022-07-20T02:30:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7dedb28700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7dedb28700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38464/0 (socket says 172.21.15.26:38464) 2022-07-20T02:30:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7dedb28700 1 -- 172.21.15.26:0/1567976826 learned_addr learned my addr 172.21.15.26:0/1567976826 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7ded327700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80b22d0 0x7f7de8152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.964+0000 7f7dee329700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de8143d60 0x7f7de8153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 -- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80b22d0 msgr2=0x7f7de8152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80b22d0 0x7f7de8152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 -- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de8143d60 msgr2=0x7f7de8153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de8143d60 0x7f7de8153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 -- 172.21.15.26:0/1567976826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd800b1e0 con 0x7f7de80618a0 2022-07-20T02:30:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dedb28700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f7de400c000 tx=0x7f7de4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7de4011750 con 0x7f7de80618a0 2022-07-20T02:30:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de80b5fd0 con 0x7f7de80618a0 2022-07-20T02:30:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de40118b0 con 0x7f7de80618a0 2022-07-20T02:30:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7de4011c10 con 0x7f7de80618a0 2022-07-20T02:30:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de813d210 con 0x7f7de80618a0 2022-07-20T02:30:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de80574e0 con 0x7f7de80618a0 2022-07-20T02:30:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.968+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7de4011d70 con 0x7f7de80618a0 2022-07-20T02:30:39.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.972+0000 7f7dd6ffd700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 0x7f7dc8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.972+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f7de406c1a0 con 0x7f7de80618a0 2022-07-20T02:30:39.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.972+0000 7f7ded327700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 0x7f7dc8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.972+0000 7f7ded327700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 0x7f7dc8055f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f7dd801c000 tx=0x7f7dd8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:38.976+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7de40431a0 con 0x7f7de80618a0 2022-07-20T02:30:39.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7de80b6160 con 0x7f7de80618a0 2022-07-20T02:30:39.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7dd6ffd700 1 -- 172.21.15.26:0/1567976826 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v61) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7de40467c0 con 0x7f7de80618a0 2022-07-20T02:30:39.104 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:30:39.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 msgr2=0x7f7dc8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 0x7f7dc8055f30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f7dd801c000 tx=0x7f7dd8015000).stop 2022-07-20T02:30:39.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 msgr2=0x7f7de80b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f7de400c000 tx=0x7f7de4006df0).stop 2022-07-20T02:30:39.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 shutdown_connections 2022-07-20T02:30:39.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7dc8053a70 0x7f7dc8055f30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7de8143d60 0x7f7de8153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7de80b22d0 0x7f7de8152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 --2- 172.21.15.26:0/1567976826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7de80618a0 0x7f7de80b02f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 >> 172.21.15.26:0/1567976826 conn(0x7f7de8066f70 msgr2=0x7f7de80ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:39.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 shutdown_connections 2022-07-20T02:30:39.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.104+0000 7f7deeb2a700 1 -- 172.21.15.26:0/1567976826 wait complete. 2022-07-20T02:30:39.120 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:30:39.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 Processor -- start 2022-07-20T02:30:39.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 -- start start 2022-07-20T02:30:39.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9d040b5ef0 0x7f9d041474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d041479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040b62d0 con 0x7f9d040a8c40 2022-07-20T02:30:39.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d04147eb0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d08fa2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0414aac0 con 0x7f9d040a7fc0 2022-07-20T02:30:39.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9d040b5ef0 0x7f9d041474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9d040b5ef0 0x7f9d041474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38476/0 (socket says 172.21.15.26:38476) 2022-07-20T02:30:39.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 -- 172.21.15.26:0/1472441129 learned_addr learned my addr 172.21.15.26:0/1472441129 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:39.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d02ffd700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d041479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 -- 172.21.15.26:0/1472441129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 msgr2=0x7f9d041479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:39.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d027fc700 1 -- 172.21.15.26:0/1472441129 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982312326 0 0) 0x7f9d040b62d0 con 0x7f9d040a8c40 2022-07-20T02:30:39.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d041479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 -- 172.21.15.26:0/1472441129 >> v1:172.21.15.120:6789/0 conn(0x7f9d040a8c40 legacy=0x7f9d040aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 -- 172.21.15.26:0/1472441129 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0414ad60 con 0x7f9d040b5ef0 2022-07-20T02:30:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d02ffd700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d041479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:39.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d037fe700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d041474d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f9cec002f70 tx=0x7f9cec006eb0).ready entity=mon.0 client_cookie=85981a63737adfd7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d027fc700 1 -- 172.21.15.26:0/1472441129 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cec00d570 con 0x7f9d040b5ef0 2022-07-20T02:30:39.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d027fc700 1 -- 172.21.15.26:0/1472441129 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec000e90 con 0x7f9d040b5ef0 2022-07-20T02:30:39.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.228+0000 7f9d027fc700 1 -- 172.21.15.26:0/1472441129 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cec00d8c0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- 172.21.15.26:0/1472441129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 msgr2=0x7f9d041474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d041474d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f9cec002f70 tx=0x7f9cec006eb0).stop 2022-07-20T02:30:39.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- 172.21.15.26:0/1472441129 shutdown_connections 2022-07-20T02:30:39.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d041479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/1472441129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d041474d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- 172.21.15.26:0/1472441129 >> 172.21.15.26:0/1472441129 conn(0x7f9d04060310 msgr2=0x7f9d040b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:39.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- 172.21.15.26:0/1472441129 shutdown_connections 2022-07-20T02:30:39.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- 172.21.15.26:0/1472441129 wait complete. 2022-07-20T02:30:39.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 Processor -- start 2022-07-20T02:30:39.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- start start 2022-07-20T02:30:39.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d040b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d040a8c40 0x7f9d040643a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040aa370 con 0x7f9d040b5ef0 2022-07-20T02:30:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04064d80 con 0x7f9d040a8c40 2022-07-20T02:30:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d08fa2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040ab210 con 0x7f9d040a7fc0 2022-07-20T02:30:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d03fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d037fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d040b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d03fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38482/0 (socket says 172.21.15.26:38482) 2022-07-20T02:30:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d03fff700 1 -- 172.21.15.26:0/2485994839 learned_addr learned my addr 172.21.15.26:0/2485994839 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:39.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.232+0000 7f9d02ffd700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d040a8c40 0x7f9d040643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 -- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 msgr2=0x7f9d040b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d040b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 -- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d040a8c40 msgr2=0x7f9d040643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d040a8c40 0x7f9d040643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 -- 172.21.15.26:0/2485994839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4009470 con 0x7f9d040b5ef0 2022-07-20T02:30:39.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d03fff700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f9cfc000bc0 tx=0x7f9cfc007a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cfc012400 con 0x7f9d040b5ef0 2022-07-20T02:30:39.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc012560 con 0x7f9d040b5ef0 2022-07-20T02:30:39.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec00b1e0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cfc0128c0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d040aa840 con 0x7f9d040b5ef0 2022-07-20T02:30:39.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d040aaa00 con 0x7f9d040b5ef0 2022-07-20T02:30:39.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9cfc019030 con 0x7f9d040b5ef0 2022-07-20T02:30:39.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 0x7f9ce8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.240+0000 7f9d037fe700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 0x7f9ce8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.236+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f9cfc06aed0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.240+0000 7f9d037fe700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 0x7f9ce8055f60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9cec01c000 tx=0x7f9cec015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.244+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cfc042080 con 0x7f9d040b5ef0 2022-07-20T02:30:39.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.376+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9d040ab350 con 0x7f9d040b5ef0 2022-07-20T02:30:39.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.376+0000 7f9d00ff9700 1 -- 172.21.15.26:0/2485994839 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) v1 ==== 84+0+143 (secure 0 0 0) 0x7f9cfc0456a0 con 0x7f9d040b5ef0 2022-07-20T02:30:39.378 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:39.379 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:30:39.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:39.380 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:30:39.380 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:30:39.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:30:39.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:30:39.381 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:30:39.381 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:30:39.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:30:39.383 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:30:39.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 msgr2=0x7f9ce8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 0x7f9ce8055f60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9cec01c000 tx=0x7f9cec015000).stop 2022-07-20T02:30:39.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 msgr2=0x7f9d04064890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f9cfc000bc0 tx=0x7f9cfc007a50).stop 2022-07-20T02:30:39.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 shutdown_connections 2022-07-20T02:30:39.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d040b5ef0 0x7f9d04064890 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9ce8053aa0 0x7f9ce8055f60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d040a8c40 0x7f9d040643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 --2- 172.21.15.26:0/2485994839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d040a7fc0 0x7f9d040b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 >> 172.21.15.26:0/2485994839 conn(0x7f9d04060310 msgr2=0x7f9d0405cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:39.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 shutdown_connections 2022-07-20T02:30:39.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.380+0000 7f9d08fa2700 1 -- 172.21.15.26:0/2485994839 wait complete. 2022-07-20T02:30:39.397 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:30:39.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:30:39.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 Processor -- start 2022-07-20T02:30:39.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 -- start start 2022-07-20T02:30:39.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0aff40 0x7feefc066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feefc0621f0 0x7feefc067020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feefc0b0f50 con 0x7feefc060250 2022-07-20T02:30:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feefc060db0 con 0x7feefc0621f0 2022-07-20T02:30:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef027b0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc13e7f0 con 0x7feefc0aff40 2022-07-20T02:30:39.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feefc0621f0 0x7feefc067020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feefc0621f0 0x7feefc067020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38486/0 (socket says 172.21.15.26:38486) 2022-07-20T02:30:39.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 -- 172.21.15.26:0/1055215460 learned_addr learned my addr 172.21.15.26:0/1055215460 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef017ae700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0aff40 0x7feefc066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 -- 172.21.15.26:0/1055215460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0aff40 msgr2=0x7feefc066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0aff40 0x7feefc066b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 -- 172.21.15.26:0/1055215460 >> v1:172.21.15.120:6789/0 conn(0x7feefc060250 legacy=0x7feefc0606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 -- 172.21.15.26:0/1055215460 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc05f8e0 con 0x7feefc0621f0 2022-07-20T02:30:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.516+0000 7fef00fad700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0621f0 0x7feefc067020 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7feeec002f70 tx=0x7feeec006eb0).ready entity=mon.0 client_cookie=35c6e346d1cbc1fc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7feeebfff700 1 -- 172.21.15.26:0/1055215460 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feeec00d5c0 con 0x7feefc0621f0 2022-07-20T02:30:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7feeebfff700 1 -- 172.21.15.26:0/1055215460 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec000e90 con 0x7feefc0621f0 2022-07-20T02:30:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7feeebfff700 1 -- 172.21.15.26:0/1055215460 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feeec00d950 con 0x7feefc0621f0 2022-07-20T02:30:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- 172.21.15.26:0/1055215460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0621f0 msgr2=0x7feefc067020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0621f0 0x7feefc067020 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7feeec002f70 tx=0x7feeec006eb0).stop 2022-07-20T02:30:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- 172.21.15.26:0/1055215460 shutdown_connections 2022-07-20T02:30:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0621f0 0x7feefc067020 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- 172.21.15.26:0/1055215460 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0aff40 0x7feefc066b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- 172.21.15.26:0/1055215460 >> 172.21.15.26:0/1055215460 conn(0x7feefc0b3050 msgr2=0x7feefc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- 172.21.15.26:0/1055215460 shutdown_connections 2022-07-20T02:30:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- 172.21.15.26:0/1055215460 wait complete. 2022-07-20T02:30:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 Processor -- start 2022-07-20T02:30:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- start start 2022-07-20T02:30:39.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feefc060250 0x7feefc153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 0x7feefc063a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc1423a0 con 0x7feefc0aff40 2022-07-20T02:30:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc154670 con 0x7feefc060250 2022-07-20T02:30:39.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef027b0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc0ad130 con 0x7feefc0621f0 2022-07-20T02:30:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.520+0000 7fef01faf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38496/0 (socket says 172.21.15.26:38496) 2022-07-20T02:30:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 -- 172.21.15.26:0/225159936 learned_addr learned my addr 172.21.15.26:0/225159936 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:39.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef017ae700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feefc060250 0x7feefc153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef00fad700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 0x7feefc063a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 -- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 msgr2=0x7feefc063a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 0x7feefc063a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 -- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feefc060250 msgr2=0x7feefc153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:39.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feefc060250 0x7feefc153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:39.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 -- 172.21.15.26:0/225159936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec00b1e0 con 0x7feefc0aff40 2022-07-20T02:30:39.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef00fad700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 0x7feefc063a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:39.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef01faf700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7feefc0b0dc0 tx=0x7feef4007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef400e4a0 con 0x7feefc0aff40 2022-07-20T02:30:39.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef400e600 con 0x7feefc0aff40 2022-07-20T02:30:39.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc0acbf0 con 0x7feefc0aff40 2022-07-20T02:30:39.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef400e960 con 0x7feefc0aff40 2022-07-20T02:30:39.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc0ac350 con 0x7feefc0aff40 2022-07-20T02:30:39.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feefc05fb10 con 0x7feefc0aff40 2022-07-20T02:30:39.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.524+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feef4019030 con 0x7feefc0aff40 2022-07-20T02:30:39.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.528+0000 7feeea7fc700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 0x7feedc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:39.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.528+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5600+0+0 (secure 0 0 0) 0x7feef406b450 con 0x7feefc0aff40 2022-07-20T02:30:39.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.528+0000 7fef017ae700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 0x7feedc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:39.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.528+0000 7fef017ae700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 0x7feedc055ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7feef800c000 tx=0x7feef8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:39.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.532+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feef40424b0 con 0x7feefc0aff40 2022-07-20T02:30:39.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:39.656+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7feefc0574e0 con 0x7feefc0aff40 2022-07-20T02:30:40.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.616+0000 7feeea7fc700 1 -- 172.21.15.26:0/225159936 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v62) v1 ==== 133+0+0 (secure 0 0 0) 0x7feef4045ad0 con 0x7feefc0aff40 2022-07-20T02:30:40.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 msgr2=0x7feedc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:40.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 0x7feedc055ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7feef800c000 tx=0x7feef8008040).stop 2022-07-20T02:30:40.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 msgr2=0x7feefc140f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:40.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7feefc0b0dc0 tx=0x7feef4007f40).stop 2022-07-20T02:30:40.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 shutdown_connections 2022-07-20T02:30:40.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feefc0aff40 0x7feefc140f70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feedc053b30 0x7feedc055ff0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feefc0621f0 0x7feefc063a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 --2- 172.21.15.26:0/225159936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feefc060250 0x7feefc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 >> 172.21.15.26:0/225159936 conn(0x7feefc0b3050 msgr2=0x7feefc0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:40.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 shutdown_connections 2022-07-20T02:30:40.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.624+0000 7fef027b0700 1 -- 172.21.15.26:0/225159936 wait complete. 2022-07-20T02:30:40.631 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:30:40.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:30:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.768+0000 7f707759e700 1 Processor -- start 2022-07-20T02:30:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.768+0000 7f707759e700 1 -- start start 2022-07-20T02:30:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f707805ce30 0x7f7078066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 0x7f70780acb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7078060c80 0x7f7078061110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7078146bb0 con 0x7f707805ce30 2022-07-20T02:30:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7078146e50 con 0x7f70780ac6f0 2022-07-20T02:30:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70781470f0 con 0x7f7078060c80 2022-07-20T02:30:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f707805ce30 0x7f7078066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f707805ce30 0x7f7078066c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38500/0 (socket says 172.21.15.26:38500) 2022-07-20T02:30:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 -- 172.21.15.26:0/2581619578 learned_addr learned my addr 172.21.15.26:0/2581619578 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f7076d9d700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7078060c80 0x7f7078061110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f706dd9b700 1 --2- 172.21.15.26:0/2581619578 >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 0x7f70780acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 -- 172.21.15.26:0/2581619578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7078060c80 msgr2=0x7f7078061110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7078060c80 0x7f7078061110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 -- 172.21.15.26:0/2581619578 >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 msgr2=0x7f70780acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 --2- 172.21.15.26:0/2581619578 >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 0x7f70780acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 -- 172.21.15.26:0/2581619578 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078147390 con 0x7f707805ce30 2022-07-20T02:30:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f706dd9b700 1 --2- 172.21.15.26:0/2581619578 >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 0x7f70780acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707659c700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f707805ce30 0x7f7078066c00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f7068002f70 tx=0x7f7068006eb0).ready entity=mon.0 client_cookie=8317b3f885f48158 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:40.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f7075d9b700 1 -- 172.21.15.26:0/2581619578 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f706800d570 con 0x7f707805ce30 2022-07-20T02:30:40.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f7075d9b700 1 -- 172.21.15.26:0/2581619578 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7068000e90 con 0x7f707805ce30 2022-07-20T02:30:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f7075d9b700 1 -- 172.21.15.26:0/2581619578 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f706800d900 con 0x7f707805ce30 2022-07-20T02:30:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- 172.21.15.26:0/2581619578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f707805ce30 msgr2=0x7f7078066c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f707805ce30 0x7f7078066c00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f7068002f70 tx=0x7f7068006eb0).stop 2022-07-20T02:30:40.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- 172.21.15.26:0/2581619578 shutdown_connections 2022-07-20T02:30:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7078060c80 0x7f7078061110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- 172.21.15.26:0/2581619578 >> v2:172.21.15.120:3300/0 conn(0x7f70780ac6f0 0x7f70780acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 --2- 172.21.15.26:0/2581619578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f707805ce30 0x7f7078066c00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- 172.21.15.26:0/2581619578 >> 172.21.15.26:0/2581619578 conn(0x7f70780b7090 msgr2=0x7f707805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:40.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- 172.21.15.26:0/2581619578 shutdown_connections 2022-07-20T02:30:40.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.772+0000 7f707759e700 1 -- 172.21.15.26:0/2581619578 wait complete. 2022-07-20T02:30:40.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.776+0000 7f707759e700 1 Processor -- start 2022-07-20T02:30:40.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.776+0000 7f707759e700 1 -- start start 2022-07-20T02:30:40.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.776+0000 7f707759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707805ce30 0x7f707813cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70780ac6f0 0x7f70781563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70781558e0 con 0x7f7078060c80 2022-07-20T02:30:40.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078147bf0 con 0x7f70780ac6f0 2022-07-20T02:30:40.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7078147e90 con 0x7f707805ce30 2022-07-20T02:30:40.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38508/0 (socket says 172.21.15.26:38508) 2022-07-20T02:30:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 -- 172.21.15.26:0/3143148318 learned_addr learned my addr 172.21.15.26:0/3143148318 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707659c700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707805ce30 0x7f707813cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f7076d9d700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70780ac6f0 0x7f70781563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 -- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707805ce30 msgr2=0x7f707813cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707805ce30 0x7f707813cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 -- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70780ac6f0 msgr2=0x7f70781563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70780ac6f0 0x7f70781563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 -- 172.21.15.26:0/3143148318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706800b1e0 con 0x7f7078060c80 2022-07-20T02:30:40.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706dd9b700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f7060006940 tx=0x7f7060008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:40.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7060011780 con 0x7f7078060c80 2022-07-20T02:30:40.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70600118e0 con 0x7f7078060c80 2022-07-20T02:30:40.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7078148130 con 0x7f7078060c80 2022-07-20T02:30:40.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7060011c40 con 0x7f7078060c80 2022-07-20T02:30:40.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707813d680 con 0x7f7078060c80 2022-07-20T02:30:40.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.780+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707805d210 con 0x7f7078060c80 2022-07-20T02:30:40.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.784+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7060012420 con 0x7f7078060c80 2022-07-20T02:30:40.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.784+0000 7f706ffff700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 0x7f705c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:40.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.784+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f706006bc30 con 0x7f7078060c80 2022-07-20T02:30:40.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.784+0000 7f707659c700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 0x7f705c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:40.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.784+0000 7f707659c700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 0x7f705c055e10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f706801c000 tx=0x7f7068015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:40.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.792+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7060042b60 con 0x7f7078060c80 2022-07-20T02:30:40.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:40.920+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f70780574e0 con 0x7f7078060c80 2022-07-20T02:30:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.868+0000 7f706ffff700 1 -- 172.21.15.26:0/3143148318 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v64) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7060046180 con 0x7f7078060c80 2022-07-20T02:30:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 msgr2=0x7f705c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 0x7f705c055e10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f706801c000 tx=0x7f7068015000).stop 2022-07-20T02:30:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 msgr2=0x7f7078155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f7060006940 tx=0x7f7060008040).stop 2022-07-20T02:30:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 shutdown_connections 2022-07-20T02:30:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7078060c80 0x7f7078155ef0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:41.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f705c053950 0x7f705c055e10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:41.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70780ac6f0 0x7f70781563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:41.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 --2- 172.21.15.26:0/3143148318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f707805ce30 0x7f707813cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:41.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 >> 172.21.15.26:0/3143148318 conn(0x7f70780b7090 msgr2=0x7f70781458a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:41.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 shutdown_connections 2022-07-20T02:30:41.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:41.876+0000 7f707759e700 1 -- 172.21.15.26:0/3143148318 wait complete. 2022-07-20T02:30:41.882 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:30:41.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:30:42.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 Processor -- start 2022-07-20T02:30:42.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 -- start start 2022-07-20T02:30:42.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f78640b3260 0x7f7864067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 0x7f78640979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786405ff70 con 0x7f7864097eb0 2022-07-20T02:30:42.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78640b3640 con 0x7f78640b3260 2022-07-20T02:30:42.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7868fc8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78640601a0 con 0x7f7864061860 2022-07-20T02:30:42.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f78637fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f78640b3260 0x7f7864067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7862ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 0x7f78640979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7863fff700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7864097eb0 0x7f78640b1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:52012/0 (socket says 172.21.15.26:52012) 2022-07-20T02:30:42.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.008+0000 7f7863fff700 1 -- 172.21.15.26:0/4047879639 learned_addr learned my addr 172.21.15.26:0/4047879639 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:42.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7862ffd700 1 -- 172.21.15.26:0/4047879639 >> v2:172.21.15.120:3300/0 conn(0x7f78640b3260 msgr2=0x7f7864067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:42.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7862ffd700 1 --2- 172.21.15.26:0/4047879639 >> v2:172.21.15.120:3300/0 conn(0x7f78640b3260 0x7f7864067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7862ffd700 1 -- 172.21.15.26:0/4047879639 >> v1:172.21.15.120:6789/0 conn(0x7f7864097eb0 legacy=0x7f78640b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7862ffd700 1 -- 172.21.15.26:0/4047879639 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78640606e0 con 0x7f7864061860 2022-07-20T02:30:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7862ffd700 1 --2- 172.21.15.26:0/4047879639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 0x7f78640979c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f784c002f70 tx=0x7f784c006df0).ready entity=mon.2 client_cookie=b32178126820f1c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f78627fc700 1 -- 172.21.15.26:0/4047879639 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c00d570 con 0x7f7864061860 2022-07-20T02:30:42.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f78627fc700 1 -- 172.21.15.26:0/4047879639 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c000e90 con 0x7f7864061860 2022-07-20T02:30:42.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f78627fc700 1 -- 172.21.15.26:0/4047879639 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c00d8c0 con 0x7f7864061860 2022-07-20T02:30:42.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- 172.21.15.26:0/4047879639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 msgr2=0x7f78640979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 --2- 172.21.15.26:0/4047879639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 0x7f78640979c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f784c002f70 tx=0x7f784c006df0).stop 2022-07-20T02:30:42.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- 172.21.15.26:0/4047879639 shutdown_connections 2022-07-20T02:30:42.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 --2- 172.21.15.26:0/4047879639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7864061860 0x7f78640979c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 --2- 172.21.15.26:0/4047879639 >> v2:172.21.15.120:3300/0 conn(0x7f78640b3260 0x7f7864067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- 172.21.15.26:0/4047879639 >> 172.21.15.26:0/4047879639 conn(0x7f78640b67f0 msgr2=0x7f78640b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- 172.21.15.26:0/4047879639 shutdown_connections 2022-07-20T02:30:42.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- 172.21.15.26:0/4047879639 wait complete. 2022-07-20T02:30:42.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 Processor -- start 2022-07-20T02:30:42.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.012+0000 7f7868fc8700 1 -- start start 2022-07-20T02:30:42.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 0x7f7864063770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78640b3260 0x7f7864137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786409b820 con 0x7f7864097eb0 2022-07-20T02:30:42.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641542e0 con 0x7f7864061860 2022-07-20T02:30:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78640b29a0 con 0x7f78640b3260 2022-07-20T02:30:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7863fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78640b3260 0x7f7864137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f78637fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 0x7f7864063770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38526/0 (socket says 172.21.15.26:38526) 2022-07-20T02:30:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 -- 172.21.15.26:0/1168386872 learned_addr learned my addr 172.21.15.26:0/1168386872 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f78637fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 0x7f7864063770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58214/0 (socket says 172.21.15.26:58214) 2022-07-20T02:30:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 -- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78640b3260 msgr2=0x7f7864137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78640b3260 0x7f7864137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 -- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 msgr2=0x7f7864063770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 0x7f7864063770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 -- 172.21.15.26:0/1168386872 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c00b180 con 0x7f7864097eb0 2022-07-20T02:30:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7862ffd700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f784c002f70 tx=0x7f784c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c016070 con 0x7f7864097eb0 2022-07-20T02:30:42.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c00d8f0 con 0x7f7864097eb0 2022-07-20T02:30:42.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7864154570 con 0x7f7864097eb0 2022-07-20T02:30:42.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c013de0 con 0x7f7864097eb0 2022-07-20T02:30:42.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.016+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7864063030 con 0x7f7864097eb0 2022-07-20T02:30:42.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78640634a0 con 0x7f7864097eb0 2022-07-20T02:30:42.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f784c025030 con 0x7f7864097eb0 2022-07-20T02:30:42.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f7860ff9700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 0x7f7848055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f784c06cfd0 con 0x7f7864097eb0 2022-07-20T02:30:42.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f78637fe700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 0x7f7848055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.020+0000 7f78637fe700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 0x7f7848055fc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f785400b000 tx=0x7f7854006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.028+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f784c043db0 con 0x7f7864097eb0 2022-07-20T02:30:42.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.188+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f78640574e0 con 0x7f7864097eb0 2022-07-20T02:30:42.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.200+0000 7f7860ff9700 1 -- 172.21.15.26:0/1168386872 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v29) v1 ==== 189+0+0 (secure 0 0 0) 0x7f784c023020 con 0x7f7864097eb0 2022-07-20T02:30:42.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 msgr2=0x7f7848055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 0x7f7848055fc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f785400b000 tx=0x7f7854006df0).stop 2022-07-20T02:30:42.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 msgr2=0x7f7864150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f784c002f70 tx=0x7f784c014000).stop 2022-07-20T02:30:42.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 shutdown_connections 2022-07-20T02:30:42.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7864097eb0 0x7f7864150490 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7848053b00 0x7f7848055fc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78640b3260 0x7f7864137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 --2- 172.21.15.26:0/1168386872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864061860 0x7f7864063770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 >> 172.21.15.26:0/1168386872 conn(0x7f78640b67f0 msgr2=0x7f7864142370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 shutdown_connections 2022-07-20T02:30:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.204+0000 7f7868fc8700 1 -- 172.21.15.26:0/1168386872 wait complete. 2022-07-20T02:30:42.212 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 11 and data pool 12 2022-07-20T02:30:42.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:30:42.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 Processor -- start 2022-07-20T02:30:42.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 -- start start 2022-07-20T02:30:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc72c0b3260 0x7fc72c060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c0b3640 con 0x7fc72c0b1720 2022-07-20T02:30:42.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0615c0 con 0x7fc72c0b3260 2022-07-20T02:30:42.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc73085c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c061860 con 0x7fc72c0a95a0 2022-07-20T02:30:42.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc72c0b3260 0x7fc72c060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc72c0b3260 0x7fc72c060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38542/0 (socket says 172.21.15.26:38542) 2022-07-20T02:30:42.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 -- 172.21.15.26:0/2557035894 learned_addr learned my addr 172.21.15.26:0/2557035894 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72affd700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 -- 172.21.15.26:0/2557035894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 msgr2=0x7fc72c0676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:42.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 -- 172.21.15.26:0/2557035894 >> v1:172.21.15.120:6789/0 conn(0x7fc72c0b1720 legacy=0x7fc72c0b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.324+0000 7fc72b7fe700 1 -- 172.21.15.26:0/2557035894 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c061b00 con 0x7fc72c0b3260 2022-07-20T02:30:42.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc72b7fe700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b3260 0x7fc72c060950 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fc71c0032f0 tx=0x7fc71c009290).ready entity=mon.0 client_cookie=a2763659f3fe3634 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc72a7fc700 1 -- 172.21.15.26:0/2557035894 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c00b8a0 con 0x7fc72c0b3260 2022-07-20T02:30:42.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc72a7fc700 1 -- 172.21.15.26:0/2557035894 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c00ba00 con 0x7fc72c0b3260 2022-07-20T02:30:42.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc72a7fc700 1 -- 172.21.15.26:0/2557035894 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c00bb70 con 0x7fc72c0b3260 2022-07-20T02:30:42.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- 172.21.15.26:0/2557035894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b3260 msgr2=0x7fc72c060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b3260 0x7fc72c060950 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fc71c0032f0 tx=0x7fc71c009290).stop 2022-07-20T02:30:42.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- 172.21.15.26:0/2557035894 shutdown_connections 2022-07-20T02:30:42.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 --2- 172.21.15.26:0/2557035894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b3260 0x7fc72c060950 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- 172.21.15.26:0/2557035894 >> 172.21.15.26:0/2557035894 conn(0x7fc72c0b67f0 msgr2=0x7fc72c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- 172.21.15.26:0/2557035894 shutdown_connections 2022-07-20T02:30:42.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- 172.21.15.26:0/2557035894 wait complete. 2022-07-20T02:30:42.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 Processor -- start 2022-07-20T02:30:42.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.328+0000 7fc73085c700 1 -- start start 2022-07-20T02:30:42.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0659b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc72c0b3260 0x7fc72c063820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0aebc0 con 0x7fc72c0b1720 2022-07-20T02:30:42.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c064ed0 con 0x7fc72c0b3260 2022-07-20T02:30:42.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c14ed80 con 0x7fc72c0a95a0 2022-07-20T02:30:42.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38548/0 (socket says 172.21.15.26:38548) 2022-07-20T02:30:42.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0659b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72affd700 1 -- 172.21.15.26:0/1233141349 learned_addr learned my addr 172.21.15.26:0/1233141349 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72bfff700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc72c0b3260 0x7fc72c063820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 -- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc72c0b3260 msgr2=0x7fc72c063820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc72c0b3260 0x7fc72c063820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 -- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 msgr2=0x7fc72c0af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 -- 172.21.15.26:0/1233141349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c009530 con 0x7fc72c0a95a0 2022-07-20T02:30:42.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72affd700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:42.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc72b7fe700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0659b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc71c0032f0 tx=0x7fc71c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c021480 con 0x7fc72c0a95a0 2022-07-20T02:30:42.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c0215e0 con 0x7fc72c0a95a0 2022-07-20T02:30:42.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c065ea0 con 0x7fc72c0a95a0 2022-07-20T02:30:42.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c021940 con 0x7fc72c0a95a0 2022-07-20T02:30:42.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c0af6a0 con 0x7fc72c0a95a0 2022-07-20T02:30:42.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.332+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c0574e0 con 0x7fc72c0a95a0 2022-07-20T02:30:42.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.336+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc71c00f070 con 0x7fc72c0a95a0 2022-07-20T02:30:42.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.336+0000 7fc728ff9700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 0x7fc70c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.336+0000 7fc72affd700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 0x7fc70c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.336+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc71c06d530 con 0x7fc72c0a95a0 2022-07-20T02:30:42.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.336+0000 7fc72affd700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 0x7fc70c055f60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc72000c000 tx=0x7fc720008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.344+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc71c044210 con 0x7fc72c0a95a0 2022-07-20T02:30:42.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.472+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc72c060c20 con 0x7fc72c0a95a0 2022-07-20T02:30:42.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc728ff9700 1 -- 172.21.15.26:0/1233141349 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16335 (secure 0 0 0) 0x7fc71c026030 con 0x7fc72c0a95a0 2022-07-20T02:30:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:42.475 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":65,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:30:42.192929+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:30:39.711494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T02:30:40.978744+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:30:42.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 msgr2=0x7fc70c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 0x7fc70c055f60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc72000c000 tx=0x7fc720008040).stop 2022-07-20T02:30:42.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 msgr2=0x7fc72c0659b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0659b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc71c0032f0 tx=0x7fc71c016000).stop 2022-07-20T02:30:42.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 shutdown_connections 2022-07-20T02:30:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc72c0b1720 0x7fc72c0af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc70c053aa0 0x7fc70c055f60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc72c0b3260 0x7fc72c063820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 --2- 172.21.15.26:0/1233141349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc72c0a95a0 0x7fc72c0659b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 >> 172.21.15.26:0/1233141349 conn(0x7fc72c0b67f0 msgr2=0x7fc72c060370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 shutdown_connections 2022-07-20T02:30:42.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.476+0000 7fc73085c700 1 -- 172.21.15.26:0/1233141349 wait complete. 2022-07-20T02:30:42.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:30:42.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 Processor -- start 2022-07-20T02:30:42.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 -- start start 2022-07-20T02:30:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fed580614b0 0x7fed58061960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed580ae370 con 0x7fed580adf90 2022-07-20T02:30:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed580b28b0 con 0x7fed580614b0 2022-07-20T02:30:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed600c9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed580b0610 con 0x7fed580b14b0 2022-07-20T02:30:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5e666700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fed580614b0 0x7fed58061960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60074/0 (socket says 172.21.15.26:60074) 2022-07-20T02:30:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 -- 172.21.15.26:0/1470316790 learned_addr learned my addr 172.21.15.26:0/1470316790 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 -- 172.21.15.26:0/1470316790 >> v2:172.21.15.120:3300/0 conn(0x7fed580614b0 msgr2=0x7fed58061960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 --2- 172.21.15.26:0/1470316790 >> v2:172.21.15.120:3300/0 conn(0x7fed580614b0 0x7fed58061960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 -- 172.21.15.26:0/1470316790 >> v1:172.21.15.120:6789/0 conn(0x7fed580adf90 legacy=0x7fed58060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 -- 172.21.15.26:0/1470316790 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed580b08b0 con 0x7fed580b14b0 2022-07-20T02:30:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5d664700 1 --2- 172.21.15.26:0/1470316790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fed500032f0 tx=0x7fed5000b530).ready entity=mon.2 client_cookie=20b1fa8986a76d15 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5ce63700 1 -- 172.21.15.26:0/1470316790 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed5000ddf0 con 0x7fed580b14b0 2022-07-20T02:30:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5ce63700 1 -- 172.21.15.26:0/1470316790 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed50010470 con 0x7fed580b14b0 2022-07-20T02:30:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.604+0000 7fed5ce63700 1 -- 172.21.15.26:0/1470316790 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed50010810 con 0x7fed580b14b0 2022-07-20T02:30:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- 172.21.15.26:0/1470316790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 msgr2=0x7fed580b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- 172.21.15.26:0/1470316790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fed500032f0 tx=0x7fed5000b530).stop 2022-07-20T02:30:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- 172.21.15.26:0/1470316790 shutdown_connections 2022-07-20T02:30:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- 172.21.15.26:0/1470316790 >> v2:172.21.15.120:3300/0 conn(0x7fed580614b0 0x7fed58061960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- 172.21.15.26:0/1470316790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580b14b0 0x7fed580b2c60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- 172.21.15.26:0/1470316790 >> 172.21.15.26:0/1470316790 conn(0x7fed580b66e0 msgr2=0x7fed580b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- 172.21.15.26:0/1470316790 shutdown_connections 2022-07-20T02:30:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- 172.21.15.26:0/1470316790 wait complete. 2022-07-20T02:30:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 Processor -- start 2022-07-20T02:30:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- start start 2022-07-20T02:30:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580614b0 0x7fed580642b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed580adf90 0x7fed58065e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed580a32f0 con 0x7fed580b14b0 2022-07-20T02:30:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58152f00 con 0x7fed580adf90 2022-07-20T02:30:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed600c9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed58153130 con 0x7fed580614b0 2022-07-20T02:30:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed5e666700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed5e666700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38566/0 (socket says 172.21.15.26:38566) 2022-07-20T02:30:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed5e666700 1 -- 172.21.15.26:0/3275038115 learned_addr learned my addr 172.21.15.26:0/3275038115 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed5d664700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed580adf90 0x7fed58065e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.608+0000 7fed5de65700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580614b0 0x7fed580642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 -- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580614b0 msgr2=0x7fed580642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580614b0 0x7fed580642b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 -- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed580adf90 msgr2=0x7fed58065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed580adf90 0x7fed58065e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 -- 172.21.15.26:0/3275038115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5000b7d0 con 0x7fed580b14b0 2022-07-20T02:30:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5e666700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fed5400c000 tx=0x7fed54006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed54011750 con 0x7fed580b14b0 2022-07-20T02:30:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed540118b0 con 0x7fed580b14b0 2022-07-20T02:30:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed581533d0 con 0x7fed580b14b0 2022-07-20T02:30:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed54011c10 con 0x7fed580b14b0 2022-07-20T02:30:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed580ab990 con 0x7fed580b14b0 2022-07-20T02:30:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed58064970 con 0x7fed580b14b0 2022-07-20T02:30:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fed54011d70 con 0x7fed580b14b0 2022-07-20T02:30:42.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 0x7fed34055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fed5406c4a0 con 0x7fed580b14b0 2022-07-20T02:30:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.612+0000 7fed5de65700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 0x7fed34055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.616+0000 7fed5de65700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 0x7fed34055ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fed4c00d000 tx=0x7fed4c007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.624+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed540430d0 con 0x7fed580b14b0 2022-07-20T02:30:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.748+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed580574e0 con 0x7fed580b14b0 2022-07-20T02:30:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed4b7fe700 1 -- 172.21.15.26:0/3275038115 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16335 (secure 0 0 0) 0x7fed540466f0 con 0x7fed580b14b0 2022-07-20T02:30:42.751 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:42.752 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":65,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:30:42.192929+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:30:39.711494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T02:30:40.978744+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:30:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 msgr2=0x7fed34055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 0x7fed34055ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fed4c00d000 tx=0x7fed4c007b90).stop 2022-07-20T02:30:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 msgr2=0x7fed58140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fed5400c000 tx=0x7fed54006df0).stop 2022-07-20T02:30:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 shutdown_connections 2022-07-20T02:30:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed580b14b0 0x7fed58140920 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed34053a10 0x7fed34055ed0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed580adf90 0x7fed58065e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 --2- 172.21.15.26:0/3275038115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed580614b0 0x7fed580642b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 >> 172.21.15.26:0/3275038115 conn(0x7fed580b66e0 msgr2=0x7fed581447c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 shutdown_connections 2022-07-20T02:30:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.752+0000 7fed600c9700 1 -- 172.21.15.26:0/3275038115 wait complete. 2022-07-20T02:30:42.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:30:42.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 Processor -- start 2022-07-20T02:30:42.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 -- start start 2022-07-20T02:30:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 0x7f35e80a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35e8147d70 0x7f35e80b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80b2f10 con 0x7f35e8060f10 2022-07-20T02:30:42.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e80a8e30 con 0x7f35e8147d70 2022-07-20T02:30:42.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35ecffc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80a7310 con 0x7f35e80b2b30 2022-07-20T02:30:42.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35e8147d70 0x7f35e80b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35e8147d70 0x7f35e80b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38574/0 (socket says 172.21.15.26:38574) 2022-07-20T02:30:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 -- 172.21.15.26:0/2889642841 learned_addr learned my addr 172.21.15.26:0/2889642841 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e659c700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 0x7f35e80a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 -- 172.21.15.26:0/2889642841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 msgr2=0x7f35e80a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 0x7f35e80a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 -- 172.21.15.26:0/2889642841 >> v1:172.21.15.26:6789/0 conn(0x7f35e8060f10 legacy=0x7f35e8147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 -- 172.21.15.26:0/2889642841 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e80a7500 con 0x7f35e8147d70 2022-07-20T02:30:42.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e659c700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 0x7f35e80a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:42.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e6d9d700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e8147d70 0x7f35e80b1990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f35e00032f0 tx=0x7f35e0009290).ready entity=mon.0 client_cookie=958e363000d89ed3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e559a700 1 -- 172.21.15.26:0/2889642841 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e000b8a0 con 0x7f35e8147d70 2022-07-20T02:30:42.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e559a700 1 -- 172.21.15.26:0/2889642841 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e000ba00 con 0x7f35e8147d70 2022-07-20T02:30:42.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.872+0000 7f35e559a700 1 -- 172.21.15.26:0/2889642841 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e000bb70 con 0x7f35e8147d70 2022-07-20T02:30:42.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- 172.21.15.26:0/2889642841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e8147d70 msgr2=0x7f35e80b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e8147d70 0x7f35e80b1990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f35e00032f0 tx=0x7f35e0009290).stop 2022-07-20T02:30:42.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- 172.21.15.26:0/2889642841 shutdown_connections 2022-07-20T02:30:42.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e8147d70 0x7f35e80b1990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 --2- 172.21.15.26:0/2889642841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e80b2b30 0x7f35e80a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- 172.21.15.26:0/2889642841 >> 172.21.15.26:0/2889642841 conn(0x7f35e80b6110 msgr2=0x7f35e80b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:42.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- 172.21.15.26:0/2889642841 shutdown_connections 2022-07-20T02:30:42.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- 172.21.15.26:0/2889642841 wait complete. 2022-07-20T02:30:42.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 Processor -- start 2022-07-20T02:30:42.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.876+0000 7f35ecffc700 1 -- start start 2022-07-20T02:30:42.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 0x7f35e80ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 0x7f35e815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80acdc0 con 0x7f35e80b2b30 2022-07-20T02:30:42.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80ad060 con 0x7f35e8147d70 2022-07-20T02:30:42.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8159f60 con 0x7f35e8060f10 2022-07-20T02:30:42.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38578/0 (socket says 172.21.15.26:38578) 2022-07-20T02:30:42.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 -- 172.21.15.26:0/1364855152 learned_addr learned my addr 172.21.15.26:0/1364855152 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e6d9d700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 0x7f35e815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e659c700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 0x7f35e80ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 -- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 msgr2=0x7f35e80ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 0x7f35e80ab280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 -- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 msgr2=0x7f35e815a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 0x7f35e815a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 -- 172.21.15.26:0/1364855152 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0009530 con 0x7f35e80b2b30 2022-07-20T02:30:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e659c700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 0x7f35e80ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e6d9d700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 0x7f35e815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35e5d9b700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f35e8061db0 tx=0x7f35dc007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc00ce30 con 0x7f35e80b2b30 2022-07-20T02:30:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc007240 con 0x7f35e80b2b30 2022-07-20T02:30:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e81543c0 con 0x7f35e80b2b30 2022-07-20T02:30:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc006510 con 0x7f35e80b2b30 2022-07-20T02:30:42.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e80ac9d0 con 0x7f35e80b2b30 2022-07-20T02:30:42.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.880+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e815a120 con 0x7f35e80b2b30 2022-07-20T02:30:42.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.884+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35dc01b070 con 0x7f35e80b2b30 2022-07-20T02:30:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.884+0000 7f35d77fe700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 0x7f35c4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.884+0000 7f35e659c700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 0x7f35c4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.884+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f35dc06b640 con 0x7f35e80b2b30 2022-07-20T02:30:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.884+0000 7f35e659c700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 0x7f35c4055f60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f35d800c000 tx=0x7f35d8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:42.888+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35dc042400 con 0x7f35e80b2b30 2022-07-20T02:30:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.048+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f35e8159d70 con 0x7f35e80b2b30 2022-07-20T02:30:43.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.212+0000 7f35d77fe700 1 -- 172.21.15.26:0/1364855152 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35dc045a20 con 0x7f35e80b2b30 2022-07-20T02:30:43.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 msgr2=0x7f35c4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 0x7f35c4055f60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f35d800c000 tx=0x7f35d8008040).stop 2022-07-20T02:30:43.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 msgr2=0x7f35e815a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f35e8061db0 tx=0x7f35dc007c00).stop 2022-07-20T02:30:43.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 shutdown_connections 2022-07-20T02:30:43.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35e80b2b30 0x7f35e815a400 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f35c4053aa0 0x7f35c4055f60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e8147d70 0x7f35e815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 --2- 172.21.15.26:0/1364855152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e8060f10 0x7f35e80ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 >> 172.21.15.26:0/1364855152 conn(0x7f35e80b6110 msgr2=0x7f35e8066bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 shutdown_connections 2022-07-20T02:30:43.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.216+0000 7f35ecffc700 1 -- 172.21.15.26:0/1364855152 wait complete. 2022-07-20T02:30:43.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 Processor -- start 2022-07-20T02:30:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- start start 2022-07-20T02:30:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 0x7f2f24148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2f24066a80 0x7f2f24066ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2414b040 con 0x7f2f24066a80 2022-07-20T02:30:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f240a8860 con 0x7f2f24061050 2022-07-20T02:30:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240b1bc0 con 0x7f2f240b2920 2022-07-20T02:30:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2f24066a80 0x7f2f24066ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2f24066a80 0x7f2f24066ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38588/0 (socket says 172.21.15.26:38588) 2022-07-20T02:30:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 -- 172.21.15.26:0/3012282216 learned_addr learned my addr 172.21.15.26:0/3012282216 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f2259c700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f21d9b700 1 --2- 172.21.15.26:0/3012282216 >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 0x7f2f24148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 -- 172.21.15.26:0/3012282216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 msgr2=0x7f2f24147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 -- 172.21.15.26:0/3012282216 >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 msgr2=0x7f2f24148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 --2- 172.21.15.26:0/3012282216 >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 0x7f2f24148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 -- 172.21.15.26:0/3012282216 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f24147700 con 0x7f2f24066a80 2022-07-20T02:30:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f21d9b700 1 --2- 172.21.15.26:0/3012282216 >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 0x7f2f24148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:30:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f22d9d700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24066a80 0x7f2f24066ec0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f2f1c0032f0 tx=0x7f2f1c009290).ready entity=mon.0 client_cookie=d6dce6aa25e759ac server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f2159a700 1 -- 172.21.15.26:0/3012282216 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f1c00b870 con 0x7f2f24066a80 2022-07-20T02:30:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f2159a700 1 -- 172.21.15.26:0/3012282216 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c00b9d0 con 0x7f2f24066a80 2022-07-20T02:30:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f2159a700 1 -- 172.21.15.26:0/3012282216 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f1c00bb40 con 0x7f2f24066a80 2022-07-20T02:30:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- 172.21.15.26:0/3012282216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24066a80 msgr2=0x7f2f24066ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24066a80 0x7f2f24066ec0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f2f1c0032f0 tx=0x7f2f1c009290).stop 2022-07-20T02:30:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- 172.21.15.26:0/3012282216 shutdown_connections 2022-07-20T02:30:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24066a80 0x7f2f24066ec0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/3012282216 >> v2:172.21.15.120:3300/0 conn(0x7f2f24061050 0x7f2f24148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/3012282216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- 172.21.15.26:0/3012282216 >> 172.21.15.26:0/3012282216 conn(0x7f2f240b60a0 msgr2=0x7f2f240b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- 172.21.15.26:0/3012282216 shutdown_connections 2022-07-20T02:30:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.348+0000 7f2f28ec4700 1 -- 172.21.15.26:0/3012282216 wait complete. 2022-07-20T02:30:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 Processor -- start 2022-07-20T02:30:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- start start 2022-07-20T02:30:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f24066a80 0x7f2f24145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240a9770 con 0x7f2f24061050 2022-07-20T02:30:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240a9a10 con 0x7f2f24066a80 2022-07-20T02:30:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240a9cb0 con 0x7f2f240b2920 2022-07-20T02:30:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38590/0 (socket says 172.21.15.26:38590) 2022-07-20T02:30:43.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 -- 172.21.15.26:0/562617756 learned_addr learned my addr 172.21.15.26:0/562617756 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:43.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f21d9b700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f24066a80 0x7f2f24145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f22d9d700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 -- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 msgr2=0x7f2f24145cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24145cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 -- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f24066a80 msgr2=0x7f2f24145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f24066a80 0x7f2f24145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 -- 172.21.15.26:0/562617756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c009530 con 0x7f2f24061050 2022-07-20T02:30:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f2259c700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f2f1800a000 tx=0x7f2f18008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f18011780 con 0x7f2f24061050 2022-07-20T02:30:43.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f180118e0 con 0x7f2f24061050 2022-07-20T02:30:43.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f24063c80 con 0x7f2f24061050 2022-07-20T02:30:43.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f18011c40 con 0x7f2f24061050 2022-07-20T02:30:43.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.352+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f24064bd0 con 0x7f2f24061050 2022-07-20T02:30:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2405dc30 con 0x7f2f24061050 2022-07-20T02:30:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2f1801c460 con 0x7f2f24061050 2022-07-20T02:30:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f137fe700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 0x7f2f00055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2f1806c310 con 0x7f2f24061050 2022-07-20T02:30:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f21d9b700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 0x7f2f00055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.356+0000 7f2f21d9b700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 0x7f2f00055f30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f2f1400c000 tx=0x7f2f14008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.364+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2f2405dc30 con 0x7f2f24061050 2022-07-20T02:30:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.528+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f2405dc30 con 0x7f2f24061050 2022-07-20T02:30:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.528+0000 7f2f137fe700 1 -- 172.21.15.26:0/562617756 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f2f2405dc30 con 0x7f2f24061050 2022-07-20T02:30:43.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:43.528 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:30:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 msgr2=0x7f2f00055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 0x7f2f00055f30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f2f1400c000 tx=0x7f2f14008040).stop 2022-07-20T02:30:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 msgr2=0x7f2f2414e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f2f1800a000 tx=0x7f2f18008040).stop 2022-07-20T02:30:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 shutdown_connections 2022-07-20T02:30:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2f00053a70 0x7f2f00055f30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f240b2920 0x7f2f24145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f24066a80 0x7f2f24145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 --2- 172.21.15.26:0/562617756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f24061050 0x7f2f2414e750 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 >> 172.21.15.26:0/562617756 conn(0x7f2f240b60a0 msgr2=0x7f2f2414a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 shutdown_connections 2022-07-20T02:30:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.532+0000 7f2f28ec4700 1 -- 172.21.15.26:0/562617756 wait complete. 2022-07-20T02:30:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:30:43.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:30:43.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.652+0000 7fef5dd10700 1 Processor -- start 2022-07-20T02:30:43.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.652+0000 7fef5dd10700 1 -- start start 2022-07-20T02:30:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 0x7fef580b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580aa1e0 con 0x7fef58060950 2022-07-20T02:30:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580ae110 con 0x7fef580af6d0 2022-07-20T02:30:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580a9f40 con 0x7fef580add30 2022-07-20T02:30:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef57fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fef58060950 0x7fef58060d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37880/0 (socket says 172.21.15.26:37880) 2022-07-20T02:30:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef57fff700 1 -- 172.21.15.26:0/2920617481 learned_addr learned my addr 172.21.15.26:0/2920617481 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5cd0e700 1 --2- 172.21.15.26:0/2920617481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 0x7fef580b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef577fe700 1 -- 172.21.15.26:0/2920617481 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402644388 0 0) 0x7fef580aa1e0 con 0x7fef58060950 2022-07-20T02:30:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef577fe700 1 -- 172.21.15.26:0/2920617481 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef3c003260 con 0x7fef58060950 2022-07-20T02:30:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5cd0e700 1 -- 172.21.15.26:0/2920617481 >> v1:172.21.15.120:6789/0 conn(0x7fef580af6d0 legacy=0x7fef580afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5cd0e700 1 -- 172.21.15.26:0/2920617481 >> v1:172.21.15.26:6789/0 conn(0x7fef58060950 legacy=0x7fef58060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5cd0e700 1 -- 172.21.15.26:0/2920617481 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580b1110 con 0x7fef580add30 2022-07-20T02:30:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5cd0e700 1 --2- 172.21.15.26:0/2920617481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 0x7fef580b2ab0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fef4c002f70 tx=0x7fef4c006df0).ready entity=mon.2 client_cookie=9885a749fb3fa162 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef577fe700 1 -- 172.21.15.26:0/2920617481 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef4c00d5c0 con 0x7fef580add30 2022-07-20T02:30:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef577fe700 1 -- 172.21.15.26:0/2920617481 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c000e90 con 0x7fef580add30 2022-07-20T02:30:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef577fe700 1 -- 172.21.15.26:0/2920617481 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef4c00d910 con 0x7fef580add30 2022-07-20T02:30:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- 172.21.15.26:0/2920617481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 msgr2=0x7fef580b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 --2- 172.21.15.26:0/2920617481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 0x7fef580b2ab0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fef4c002f70 tx=0x7fef4c006df0).stop 2022-07-20T02:30:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- 172.21.15.26:0/2920617481 shutdown_connections 2022-07-20T02:30:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 --2- 172.21.15.26:0/2920617481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef580add30 0x7fef580b2ab0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- 172.21.15.26:0/2920617481 >> 172.21.15.26:0/2920617481 conn(0x7fef580b3050 msgr2=0x7fef580b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- 172.21.15.26:0/2920617481 shutdown_connections 2022-07-20T02:30:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.656+0000 7fef5dd10700 1 -- 172.21.15.26:0/2920617481 wait complete. 2022-07-20T02:30:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.660+0000 7fef5dd10700 1 Processor -- start 2022-07-20T02:30:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.660+0000 7fef5dd10700 1 -- start start 2022-07-20T02:30:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef58060950 0x7fef5805fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef580af6d0 0x7fef580acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5813fed0 con 0x7fef580add30 2022-07-20T02:30:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580ad3d0 con 0x7fef580af6d0 2022-07-20T02:30:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580aa990 con 0x7fef58060950 2022-07-20T02:30:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38606/0 (socket says 172.21.15.26:38606) 2022-07-20T02:30:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 -- 172.21.15.26:0/1539810300 learned_addr learned my addr 172.21.15.26:0/1539810300 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5cd0e700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef58060950 0x7fef5805fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5d50f700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef580af6d0 0x7fef580acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 -- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef58060950 msgr2=0x7fef5805fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef58060950 0x7fef5805fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 -- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef580af6d0 msgr2=0x7fef580acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef580af6d0 0x7fef580acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 -- 172.21.15.26:0/1539810300 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c00b180 con 0x7fef580add30 2022-07-20T02:30:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef57fff700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fef44001110 tx=0x7fef44007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef440070d0 con 0x7fef580add30 2022-07-20T02:30:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef44007230 con 0x7fef580add30 2022-07-20T02:30:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef580aac30 con 0x7fef580add30 2022-07-20T02:30:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef44005610 con 0x7fef580add30 2022-07-20T02:30:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.664+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef580ab370 con 0x7fef580add30 2022-07-20T02:30:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580b13b0 con 0x7fef580add30 2022-07-20T02:30:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef4401a060 con 0x7fef580add30 2022-07-20T02:30:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef55ffb700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 0x7fef38055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fef4406b3b0 con 0x7fef580add30 2022-07-20T02:30:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef5cd0e700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 0x7fef38055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.668+0000 7fef5cd0e700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 0x7fef38055e40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fef4c01a000 tx=0x7fef4c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.672+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef44042170 con 0x7fef580add30 2022-07-20T02:30:43.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.800+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef580574e0 con 0x7fef580add30 2022-07-20T02:30:43.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.800+0000 7fef55ffb700 1 -- 172.21.15.26:0/1539810300 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16322 (secure 0 0 0) 0x7fef44018020 con 0x7fef580add30 2022-07-20T02:30:43.801 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:43.802 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":66,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:30:43.196141+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:30:39.711494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T02:30:40.978744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:30:43.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 msgr2=0x7fef38055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 0x7fef38055e40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fef4c01a000 tx=0x7fef4c014000).stop 2022-07-20T02:30:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 msgr2=0x7fef580ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fef44001110 tx=0x7fef44007b80).stop 2022-07-20T02:30:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 shutdown_connections 2022-07-20T02:30:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef580add30 0x7fef580ac9f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fef38053980 0x7fef38055e40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef580af6d0 0x7fef580acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 --2- 172.21.15.26:0/1539810300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef58060950 0x7fef5805fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 >> 172.21.15.26:0/1539810300 conn(0x7fef580b3050 msgr2=0x7fef581438a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 shutdown_connections 2022-07-20T02:30:43.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.804+0000 7fef5dd10700 1 -- 172.21.15.26:0/1539810300 wait complete. 2022-07-20T02:30:43.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:30:43.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.936+0000 7f8369b4e700 1 Processor -- start 2022-07-20T02:30:43.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.936+0000 7f8369b4e700 1 -- start start 2022-07-20T02:30:43.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 0x7f83641445b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83640b3640 con 0x7f83640b3260 2022-07-20T02:30:43.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8368b4c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f83640b3260 0x7f83640604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37892/0 (socket says 172.21.15.26:37892) 2022-07-20T02:30:43.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8368b4c700 1 -- 172.21.15.26:0/3638060934 learned_addr learned my addr 172.21.15.26:0/3638060934 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:43.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8364095300 con 0x7f8364097c50 2022-07-20T02:30:43.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83640955a0 con 0x7f8364060c00 2022-07-20T02:30:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8363fff700 1 --2- 172.21.15.26:0/3638060934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 0x7f83641445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f83637fe700 1 -- 172.21.15.26:0/3638060934 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105737895 0 0) 0x7f83640b3640 con 0x7f83640b3260 2022-07-20T02:30:43.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f83637fe700 1 -- 172.21.15.26:0/3638060934 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8348003260 con 0x7f83640b3260 2022-07-20T02:30:43.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8363fff700 1 -- 172.21.15.26:0/3638060934 >> v1:172.21.15.120:6789/0 conn(0x7f8364097c50 legacy=0x7f8364144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8363fff700 1 -- 172.21.15.26:0/3638060934 >> v1:172.21.15.26:6789/0 conn(0x7f83640b3260 legacy=0x7f83640604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8363fff700 1 -- 172.21.15.26:0/3638060934 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8364095840 con 0x7f8364060c00 2022-07-20T02:30:43.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8363fff700 1 --2- 172.21.15.26:0/3638060934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 0x7f83641445b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8350002f70 tx=0x7f8350006df0).ready entity=mon.2 client_cookie=f4894847efaf89c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f83637fe700 1 -- 172.21.15.26:0/3638060934 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835000d570 con 0x7f8364060c00 2022-07-20T02:30:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f83637fe700 1 -- 172.21.15.26:0/3638060934 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8350000e90 con 0x7f8364060c00 2022-07-20T02:30:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f83637fe700 1 -- 172.21.15.26:0/3638060934 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835000d8c0 con 0x7f8364060c00 2022-07-20T02:30:43.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 msgr2=0x7f83641445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 --2- 172.21.15.26:0/3638060934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 0x7f83641445b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8350002f70 tx=0x7f8350006df0).stop 2022-07-20T02:30:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 shutdown_connections 2022-07-20T02:30:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 --2- 172.21.15.26:0/3638060934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364060c00 0x7f83641445b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 >> 172.21.15.26:0/3638060934 conn(0x7f83640b67f0 msgr2=0x7f83640b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 shutdown_connections 2022-07-20T02:30:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.940+0000 7f8369b4e700 1 -- 172.21.15.26:0/3638060934 wait complete. 2022-07-20T02:30:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.944+0000 7f8369b4e700 1 Processor -- start 2022-07-20T02:30:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.944+0000 7f8369b4e700 1 -- start start 2022-07-20T02:30:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.944+0000 7f8369b4e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.944+0000 7f8369b4e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364097c50 0x7f83641531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83640b3260 0x7f8364064290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83641527e0 con 0x7f8364060c00 2022-07-20T02:30:43.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8364152a80 con 0x7f83640b3260 2022-07-20T02:30:43.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8364152d20 con 0x7f8364097c50 2022-07-20T02:30:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38618/0 (socket says 172.21.15.26:38618) 2022-07-20T02:30:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 -- 172.21.15.26:0/2433033792 learned_addr learned my addr 172.21.15.26:0/2433033792 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8363fff700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364097c50 0x7f83641531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f836934d700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83640b3260 0x7f8364064290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 -- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364097c50 msgr2=0x7f83641531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364097c50 0x7f83641531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 -- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83640b3260 msgr2=0x7f8364064290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83640b3260 0x7f8364064290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 -- 172.21.15.26:0/2433033792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835000b180 con 0x7f8364060c00 2022-07-20T02:30:43.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8368b4c700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f8358001110 tx=0x7f8358007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83580070d0 con 0x7f8364060c00 2022-07-20T02:30:43.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358007230 con 0x7f8364060c00 2022-07-20T02:30:43.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836405f3d0 con 0x7f8364060c00 2022-07-20T02:30:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8358005610 con 0x7f8364060c00 2022-07-20T02:30:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8364064780 con 0x7f8364060c00 2022-07-20T02:30:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.948+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8364152ee0 con 0x7f8364060c00 2022-07-20T02:30:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.952+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f835801a060 con 0x7f8364060c00 2022-07-20T02:30:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.952+0000 7f8361ffb700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 0x7f8344055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.952+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f835806b750 con 0x7f8364060c00 2022-07-20T02:30:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.952+0000 7f8363fff700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 0x7f8344055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.952+0000 7f8363fff700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 0x7f8344055fc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f835001a000 tx=0x7f8350014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:43.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:43.956+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8358042510 con 0x7f8364060c00 2022-07-20T02:30:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.128+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f83640993f0 con 0x7f8364060c00 2022-07-20T02:30:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8361ffb700 1 -- 172.21.15.26:0/2433033792 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8358018070 con 0x7f8364060c00 2022-07-20T02:30:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 msgr2=0x7f8344055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 0x7f8344055fc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f835001a000 tx=0x7f8350014000).stop 2022-07-20T02:30:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 msgr2=0x7f8364097040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f8358001110 tx=0x7f8358007b40).stop 2022-07-20T02:30:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 shutdown_connections 2022-07-20T02:30:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8344053b00 0x7f8344055fc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83640b3260 0x7f8364064290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8364097c50 0x7f83641531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 --2- 172.21.15.26:0/2433033792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8364060c00 0x7f8364097040 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 >> 172.21.15.26:0/2433033792 conn(0x7f83640b67f0 msgr2=0x7f83641408b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 shutdown_connections 2022-07-20T02:30:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.136+0000 7f8369b4e700 1 -- 172.21.15.26:0/2433033792 wait complete. 2022-07-20T02:30:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:30:44.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:30:44.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.268+0000 7fc14ad60700 1 Processor -- start 2022-07-20T02:30:44.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.268+0000 7fc14ad60700 1 -- start start 2022-07-20T02:30:44.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc14405f840 0x7fc144146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440b1ae0 con 0x7fc1441465b0 2022-07-20T02:30:44.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440b1d80 con 0x7fc1440b24c0 2022-07-20T02:30:44.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1440b2e20 con 0x7fc14405f840 2022-07-20T02:30:44.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc149d5e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc1441465b0 0x7fc144066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37906/0 (socket says 172.21.15.26:37906) 2022-07-20T02:30:44.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc149d5e700 1 -- 172.21.15.26:0/1093463512 learned_addr learned my addr 172.21.15.26:0/1093463512 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:44.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14a55f700 1 --2- 172.21.15.26:0/1093463512 >> v2:172.21.15.120:3300/0 conn(0x7fc14405f840 0x7fc144146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694040473 0 0) 0x7fc1440b1ae0 con 0x7fc1441465b0 2022-07-20T02:30:44.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c003260 con 0x7fc1441465b0 2022-07-20T02:30:44.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283273803 0 0) 0x7fc1440b1d80 con 0x7fc1440b24c0 2022-07-20T02:30:44.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1440b1ae0 con 0x7fc1440b24c0 2022-07-20T02:30:44.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14a55f700 1 -- 172.21.15.26:0/1093463512 >> v1:172.21.15.120:6789/0 conn(0x7fc1440b24c0 legacy=0x7fc1440604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14a55f700 1 -- 172.21.15.26:0/1093463512 >> v1:172.21.15.26:6789/0 conn(0x7fc1441465b0 legacy=0x7fc144066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14a55f700 1 -- 172.21.15.26:0/1093463512 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1440b2020 con 0x7fc14405f840 2022-07-20T02:30:44.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14a55f700 1 --2- 172.21.15.26:0/1093463512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc14405f840 0x7fc144146990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc13c0032f0 tx=0x7fc13c0091d0).ready entity=mon.1 client_cookie=e3560444bb1004a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13c00b600 con 0x7fc14405f840 2022-07-20T02:30:44.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c00b760 con 0x7fc14405f840 2022-07-20T02:30:44.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc148d5c700 1 -- 172.21.15.26:0/1093463512 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13c00b8d0 con 0x7fc14405f840 2022-07-20T02:30:44.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 -- 172.21.15.26:0/1093463512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc14405f840 msgr2=0x7fc144146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.272+0000 7fc14ad60700 1 --2- 172.21.15.26:0/1093463512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc14405f840 0x7fc144146990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc13c0032f0 tx=0x7fc13c0091d0).stop 2022-07-20T02:30:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- 172.21.15.26:0/1093463512 shutdown_connections 2022-07-20T02:30:44.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 --2- 172.21.15.26:0/1093463512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc14405f840 0x7fc144146990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- 172.21.15.26:0/1093463512 >> 172.21.15.26:0/1093463512 conn(0x7fc1440b3050 msgr2=0x7fc1440b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:44.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- 172.21.15.26:0/1093463512 shutdown_connections 2022-07-20T02:30:44.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- 172.21.15.26:0/1093463512 wait complete. 2022-07-20T02:30:44.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 Processor -- start 2022-07-20T02:30:44.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- start start 2022-07-20T02:30:44.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1441465b0 0x7fc144141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1441420b0 0x7fc144064af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440ad500 con 0x7fc14405f840 2022-07-20T02:30:44.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144152e20 con 0x7fc1441465b0 2022-07-20T02:30:44.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14ad60700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc144153080 con 0x7fc1441420b0 2022-07-20T02:30:44.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc149d5e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc149d5e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38632/0 (socket says 172.21.15.26:38632) 2022-07-20T02:30:44.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14955d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1441465b0 0x7fc144141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc14a55f700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1441420b0 0x7fc144064af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc149d5e700 1 -- 172.21.15.26:0/4207334392 learned_addr learned my addr 172.21.15.26:0/4207334392 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:44.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.276+0000 7fc149d5e700 1 -- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1441420b0 msgr2=0x7fc144064af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc149d5e700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1441420b0 0x7fc144064af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc149d5e700 1 -- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1441465b0 msgr2=0x7fc144141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc149d5e700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1441465b0 0x7fc144141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc149d5e700 1 -- 172.21.15.26:0/4207334392 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c009470 con 0x7fc14405f840 2022-07-20T02:30:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc149d5e700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fc1400011d0 tx=0x7fc140007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc14000ce70 con 0x7fc14405f840 2022-07-20T02:30:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1400070d0 con 0x7fc14405f840 2022-07-20T02:30:44.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc140007430 con 0x7fc14405f840 2022-07-20T02:30:44.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc144153320 con 0x7fc14405f840 2022-07-20T02:30:44.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc144064fe0 con 0x7fc14405f840 2022-07-20T02:30:44.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1440674c0 con 0x7fc14405f840 2022-07-20T02:30:44.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc140010070 con 0x7fc14405f840 2022-07-20T02:30:44.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 0x7fc124055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc14006b6e0 con 0x7fc14405f840 2022-07-20T02:30:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.280+0000 7fc14955d700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 0x7fc124055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.284+0000 7fc14955d700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 0x7fc124055f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc13400d000 tx=0x7fc134007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.284+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc140042390 con 0x7fc14405f840 2022-07-20T02:30:44.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.452+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc1440574e0 con 0x7fc14405f840 2022-07-20T02:30:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc132ffd700 1 -- 172.21.15.26:0/4207334392 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc1400459b0 con 0x7fc14405f840 2022-07-20T02:30:44.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 msgr2=0x7fc124055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 0x7fc124055f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc13400d000 tx=0x7fc134007a60).stop 2022-07-20T02:30:44.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 msgr2=0x7fc144142700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fc1400011d0 tx=0x7fc140007b50).stop 2022-07-20T02:30:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 shutdown_connections 2022-07-20T02:30:44.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc124053aa0 0x7fc124055f60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1441420b0 0x7fc144064af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1441465b0 0x7fc144141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 --2- 172.21.15.26:0/4207334392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc14405f840 0x7fc144142700 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.460+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 >> 172.21.15.26:0/4207334392 conn(0x7fc1440b3050 msgr2=0x7fc14413f980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:44.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.464+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 shutdown_connections 2022-07-20T02:30:44.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.464+0000 7fc14ad60700 1 -- 172.21.15.26:0/4207334392 wait complete. 2022-07-20T02:30:44.469 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:30:44.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:44.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 Processor -- start 2022-07-20T02:30:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 -- start start 2022-07-20T02:30:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd0c00a7fc0 0x7fd0c01479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c00b62d0 con 0x7fd0c00a8c40 2022-07-20T02:30:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c0147eb0 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0c492e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c014aac0 con 0x7fd0c00b5ef0 2022-07-20T02:30:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd0c00a7fc0 0x7fd0c01479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0bffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd0c00a8c40 0x7fd0c00aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37922/0 (socket says 172.21.15.26:37922) 2022-07-20T02:30:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0bffff700 1 -- 172.21.15.26:0/75138726 learned_addr learned my addr 172.21.15.26:0/75138726 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0bf7fe700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 -- 172.21.15.26:0/75138726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 msgr2=0x7fd0c01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:44.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c01474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 -- 172.21.15.26:0/75138726 >> v1:172.21.15.26:6789/0 conn(0x7fd0c00a8c40 legacy=0x7fd0c00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 -- 172.21.15.26:0/75138726 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c014ad60 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0beffd700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c01479c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fd0b0002f70 tx=0x7fd0b0006df0).ready entity=mon.0 client_cookie=5029913df1580ebf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0be7fc700 1 -- 172.21.15.26:0/75138726 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b000d570 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0be7fc700 1 -- 172.21.15.26:0/75138726 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b0000e90 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.584+0000 7fd0be7fc700 1 -- 172.21.15.26:0/75138726 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b000d8c0 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- 172.21.15.26:0/75138726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a7fc0 msgr2=0x7fd0c01479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c01479c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fd0b0002f70 tx=0x7fd0b0006df0).stop 2022-07-20T02:30:44.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- 172.21.15.26:0/75138726 shutdown_connections 2022-07-20T02:30:44.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c01479c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- 172.21.15.26:0/75138726 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- 172.21.15.26:0/75138726 >> 172.21.15.26:0/75138726 conn(0x7fd0c0060310 msgr2=0x7fd0c00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- 172.21.15.26:0/75138726 shutdown_connections 2022-07-20T02:30:44.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- 172.21.15.26:0/75138726 wait complete. 2022-07-20T02:30:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 Processor -- start 2022-07-20T02:30:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- start start 2022-07-20T02:30:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c0063b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0066cc0 con 0x7fd0c00a8c40 2022-07-20T02:30:44.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0066f60 con 0x7fd0c00a7fc0 2022-07-20T02:30:44.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0c492e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0067200 con 0x7fd0c00b5ef0 2022-07-20T02:30:44.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38648/0 (socket says 172.21.15.26:38648) 2022-07-20T02:30:44.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 -- 172.21.15.26:0/3677568115 learned_addr learned my addr 172.21.15.26:0/3677568115 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:44.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0bf7fe700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c0063b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0bffff700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 -- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 msgr2=0x7fd0c015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 -- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c00a7fc0 msgr2=0x7fd0c0063b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c0063b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 -- 172.21.15.26:0/3677568115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b000b180 con 0x7fd0c00a8c40 2022-07-20T02:30:44.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0beffd700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fd0b0002f70 tx=0x7fd0b0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b000d8f0 con 0x7fd0c00a8c40 2022-07-20T02:30:44.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.588+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b0011b40 con 0x7fd0c00a8c40 2022-07-20T02:30:44.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b0011ea0 con 0x7fd0c00a8c40 2022-07-20T02:30:44.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c00674a0 con 0x7fd0c00a8c40 2022-07-20T02:30:44.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c0067920 con 0x7fd0c00a8c40 2022-07-20T02:30:44.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c0067630 con 0x7fd0c00a8c40 2022-07-20T02:30:44.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd0b001a040 con 0x7fd0c00a8c40 2022-07-20T02:30:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bcff9700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 0x7fd0a0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd0b006ce90 con 0x7fd0c00a8c40 2022-07-20T02:30:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bf7fe700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 0x7fd0a0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:44.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.592+0000 7fd0bf7fe700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 0x7fd0a0055e40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fd0b400c000 tx=0x7fd0b4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:44.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.596+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd0b0043c50 con 0x7fd0c00a8c40 2022-07-20T02:30:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.760+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0c014b000 con 0x7fd0c00a8c40 2022-07-20T02:30:44.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.760+0000 7fd0bcff9700 1 -- 172.21.15.26:0/3677568115 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7fd0b0023020 con 0x7fd0c00a8c40 2022-07-20T02:30:44.764 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:44.764 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:30:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 msgr2=0x7fd0a0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 0x7fd0a0055e40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fd0b400c000 tx=0x7fd0b4008040).stop 2022-07-20T02:30:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 msgr2=0x7fd0c0159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fd0b0002f70 tx=0x7fd0b0015000).stop 2022-07-20T02:30:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 shutdown_connections 2022-07-20T02:30:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0c00a8c40 0x7fd0c0159c70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0a0053980 0x7fd0a0055e40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0c00b5ef0 0x7fd0c015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 --2- 172.21.15.26:0/3677568115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0c00a7fc0 0x7fd0c0063b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:44.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 >> 172.21.15.26:0/3677568115 conn(0x7fd0c0060310 msgr2=0x7fd0c00b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:44.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 shutdown_connections 2022-07-20T02:30:44.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:44.764+0000 7fd0c492e700 1 -- 172.21.15.26:0/3677568115 wait complete. 2022-07-20T02:30:44.773 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:30:45.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:30:42.193367+0000', 'modified': '2022-07-20T02:30:43.197173+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5997}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5997': {'gid': 5997, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2926574302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2926574302}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2926574302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:30:45.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:30:45.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:30:45.235 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5997,"want_state":"up:active","state":"up:active","rank_uptime":3.0204307730000002,"mdsmap_epoch":30,"osdmap_epoch":66,"osdmap_epoch_barrier":66,"uptime":8.348091642} 2022-07-20T02:30:45.236 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5997, "mdsmap_epoch": 30, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "rank_uptime": 3.020430773, "state": "up:active", "uptime": 8.348091642, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:30:45.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:30:45.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 Processor -- start 2022-07-20T02:30:45.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 -- start start 2022-07-20T02:30:45.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60480979c0 0x7f6048097e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604805ff70 con 0x7f6048061860 2022-07-20T02:30:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604805cc90 con 0x7f60480979c0 2022-07-20T02:30:45.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60480b3640 con 0x7f60480b3260 2022-07-20T02:30:45.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f6047fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60480979c0 0x7f6048097e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f6047fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60170/0 (socket says 172.21.15.26:60170) 2022-07-20T02:30:45.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f6047fff700 1 -- 172.21.15.26:0/3458007850 learned_addr learned my addr 172.21.15.26:0/3458007850 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:45.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60480979c0 0x7f6048097e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58350/0 (socket says 172.21.15.26:58350) 2022-07-20T02:30:45.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f60477fe700 1 -- 172.21.15.26:0/3458007850 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786169882 0 0) 0x7f604805ff70 con 0x7f6048061860 2022-07-20T02:30:45.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f60477fe700 1 -- 172.21.15.26:0/3458007850 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003260 con 0x7f6048061860 2022-07-20T02:30:45.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 -- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 msgr2=0x7f6048067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:45.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 -- 172.21.15.26:0/3458007850 >> v1:172.21.15.120:6789/0 conn(0x7f6048061860 legacy=0x7f604805c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 -- 172.21.15.26:0/3458007850 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480606e0 con 0x7f60480979c0 2022-07-20T02:30:45.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f6047fff700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:45.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604cda6700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480979c0 0x7f6048097e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f60400032f0 tx=0x7f60400091d0).ready entity=mon.1 client_cookie=a38128d6db3e4668 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f60477fe700 1 -- 172.21.15.26:0/3458007850 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604000b600 con 0x7f60480979c0 2022-07-20T02:30:45.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f60477fe700 1 -- 172.21.15.26:0/3458007850 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604000b760 con 0x7f60480979c0 2022-07-20T02:30:45.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f60477fe700 1 -- 172.21.15.26:0/3458007850 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604000b8d0 con 0x7f60480979c0 2022-07-20T02:30:45.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 -- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480979c0 msgr2=0x7f6048097e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.348+0000 7f604d5a7700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480979c0 0x7f6048097e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f60400032f0 tx=0x7f60400091d0).stop 2022-07-20T02:30:45.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- 172.21.15.26:0/3458007850 shutdown_connections 2022-07-20T02:30:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480979c0 0x7f6048097e30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 --2- 172.21.15.26:0/3458007850 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480b3260 0x7f6048067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- 172.21.15.26:0/3458007850 >> 172.21.15.26:0/3458007850 conn(0x7f60480b67f0 msgr2=0x7f60480b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:45.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- 172.21.15.26:0/3458007850 shutdown_connections 2022-07-20T02:30:45.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- 172.21.15.26:0/3458007850 wait complete. 2022-07-20T02:30:45.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 Processor -- start 2022-07-20T02:30:45.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- start start 2022-07-20T02:30:45.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480979c0 0x7f604813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480b3260 0x7f6048155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6048094dc0 con 0x7f6048061860 2022-07-20T02:30:45.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604809aee0 con 0x7f60480b3260 2022-07-20T02:30:45.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604d5a7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604809b180 con 0x7f60480979c0 2022-07-20T02:30:45.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38660/0 (socket says 172.21.15.26:38660) 2022-07-20T02:30:45.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 -- 172.21.15.26:0/1256149667 learned_addr learned my addr 172.21.15.26:0/1256149667 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:45.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f604cda6700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480b3260 0x7f6048155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f603f7fe700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480979c0 0x7f604813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 -- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480979c0 msgr2=0x7f604813ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480979c0 0x7f604813ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 -- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480b3260 msgr2=0x7f6048155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480b3260 0x7f6048155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.352+0000 7f6047fff700 1 -- 172.21.15.26:0/1256149667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603800b460 con 0x7f6048061860 2022-07-20T02:30:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f6047fff700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f603800e000 tx=0x7f6038006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603800ee10 con 0x7f6048061860 2022-07-20T02:30:45.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603800da50 con 0x7f6048061860 2022-07-20T02:30:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603800ddb0 con 0x7f6048061860 2022-07-20T02:30:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6040009470 con 0x7f6048061860 2022-07-20T02:30:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6048094ab0 con 0x7f6048061860 2022-07-20T02:30:45.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048094880 con 0x7f6048061860 2022-07-20T02:30:45.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.356+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6038010440 con 0x7f6048061860 2022-07-20T02:30:45.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.360+0000 7f6045ffb700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 0x7f602c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.360+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f603806ca10 con 0x7f6048061860 2022-07-20T02:30:45.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.360+0000 7f603f7fe700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 0x7f602c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.360+0000 7f603f7fe700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 0x7f602c055fc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f603000d000 tx=0x7f6030007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.364+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6038043660 con 0x7f6048061860 2022-07-20T02:30:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.492+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60480574e0 con 0x7f6048061860 2022-07-20T02:30:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.492+0000 7f6045ffb700 1 -- 172.21.15.26:0/1256149667 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f6038046c80 con 0x7f6048061860 2022-07-20T02:30:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:45.496 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":66,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:30:43.196141+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:30:39.711494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T02:30:40.978744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:30:45.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 msgr2=0x7f602c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 0x7f602c055fc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f603000d000 tx=0x7f6030007a60).stop 2022-07-20T02:30:45.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 msgr2=0x7f604809b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f603800e000 tx=0x7f6038006f40).stop 2022-07-20T02:30:45.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 shutdown_connections 2022-07-20T02:30:45.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f602c053b00 0x7f602c055fc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60480b3260 0x7f6048155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60480979c0 0x7f604813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 --2- 172.21.15.26:0/1256149667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6048061860 0x7f604809b8c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 >> 172.21.15.26:0/1256149667 conn(0x7f60480b67f0 msgr2=0x7f60480b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:45.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 shutdown_connections 2022-07-20T02:30:45.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.496+0000 7f604d5a7700 1 -- 172.21.15.26:0/1256149667 wait complete. 2022-07-20T02:30:45.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:30:39.711494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:30:40.978744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:30:45.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:30:45.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 Processor -- start 2022-07-20T02:30:45.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 -- start start 2022-07-20T02:30:45.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 0x7f5f58143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 0x7f5f580b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f581474e0 con 0x7f5f58060f00 2022-07-20T02:30:45.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5805d3b0 con 0x7f5f58147000 2022-07-20T02:30:45.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.656+0000 7f5f5759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f581431e0 con 0x7f5f5805cfd0 2022-07-20T02:30:45.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f56d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5f58060f00 0x7f5f5813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37952/0 (socket says 172.21.15.26:37952) 2022-07-20T02:30:45.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f56d9d700 1 -- 172.21.15.26:0/414880360 learned_addr learned my addr 172.21.15.26:0/414880360 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:45.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 --2- 172.21.15.26:0/414880360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 0x7f5f58143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f55d9b700 1 --2- 172.21.15.26:0/414880360 >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 0x7f5f580b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5559a700 1 -- 172.21.15.26:0/414880360 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285274065 0 0) 0x7f5f581474e0 con 0x7f5f58060f00 2022-07-20T02:30:45.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5559a700 1 -- 172.21.15.26:0/414880360 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c003260 con 0x7f5f58060f00 2022-07-20T02:30:45.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 -- 172.21.15.26:0/414880360 >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 msgr2=0x7f5f580b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:45.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 --2- 172.21.15.26:0/414880360 >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 0x7f5f580b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 -- 172.21.15.26:0/414880360 >> v1:172.21.15.26:6789/0 conn(0x7f5f58060f00 legacy=0x7f5f5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 -- 172.21.15.26:0/414880360 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f58143480 con 0x7f5f5805cfd0 2022-07-20T02:30:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f55d9b700 1 --2- 172.21.15.26:0/414880360 >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 0x7f5f580b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:30:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5659c700 1 --2- 172.21.15.26:0/414880360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 0x7f5f58143bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5f48002f70 tx=0x7f5f48006df0).ready entity=mon.2 client_cookie=aa698d98c5fab392 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5559a700 1 -- 172.21.15.26:0/414880360 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4800d4a0 con 0x7f5f5805cfd0 2022-07-20T02:30:45.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5559a700 1 -- 172.21.15.26:0/414880360 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f48000e90 con 0x7f5f5805cfd0 2022-07-20T02:30:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5559a700 1 -- 172.21.15.26:0/414880360 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4800d7f0 con 0x7f5f5805cfd0 2022-07-20T02:30:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 -- 172.21.15.26:0/414880360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 msgr2=0x7f5f58143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 --2- 172.21.15.26:0/414880360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 0x7f5f58143bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5f48002f70 tx=0x7f5f48006df0).stop 2022-07-20T02:30:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 -- 172.21.15.26:0/414880360 shutdown_connections 2022-07-20T02:30:45.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 --2- 172.21.15.26:0/414880360 >> v2:172.21.15.120:3300/0 conn(0x7f5f58147000 0x7f5f580b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 --2- 172.21.15.26:0/414880360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f5805cfd0 0x7f5f58143bc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 -- 172.21.15.26:0/414880360 >> 172.21.15.26:0/414880360 conn(0x7f5f580620c0 msgr2=0x7f5f5805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:45.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 -- 172.21.15.26:0/414880360 shutdown_connections 2022-07-20T02:30:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.660+0000 7f5f5759e700 1 -- 172.21.15.26:0/414880360 wait complete. 2022-07-20T02:30:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 Processor -- start 2022-07-20T02:30:45.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- start start 2022-07-20T02:30:45.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f58060f00 0x7f5f581394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f58137500 0x7f5f58155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580b0120 con 0x7f5f58147000 2022-07-20T02:30:45.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ad670 con 0x7f5f58060f00 2022-07-20T02:30:45.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ad8a0 con 0x7f5f58137500 2022-07-20T02:30:45.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38678/0 (socket says 172.21.15.26:38678) 2022-07-20T02:30:45.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f56d9d700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f58137500 0x7f5f58155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 -- 172.21.15.26:0/3342685950 learned_addr learned my addr 172.21.15.26:0/3342685950 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:45.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5659c700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f58060f00 0x7f5f581394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 -- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f58137500 msgr2=0x7f5f58155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f58137500 0x7f5f58155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 -- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f58060f00 msgr2=0x7f5f581394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f58060f00 0x7f5f581394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 -- 172.21.15.26:0/3342685950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4000b180 con 0x7f5f58147000 2022-07-20T02:30:45.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f55d9b700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f5f580af4d0 tx=0x7f5f40006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f40010780 con 0x7f5f58147000 2022-07-20T02:30:45.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f400108e0 con 0x7f5f58147000 2022-07-20T02:30:45.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4800b180 con 0x7f5f58147000 2022-07-20T02:30:45.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f40010c40 con 0x7f5f58147000 2022-07-20T02:30:45.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.664+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f580adde0 con 0x7f5f58147000 2022-07-20T02:30:45.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f58143720 con 0x7f5f58147000 2022-07-20T02:30:45.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f4001c460 con 0x7f5f58147000 2022-07-20T02:30:45.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f4f7fe700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 0x7f5f30055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5f4006c710 con 0x7f5f58147000 2022-07-20T02:30:45.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f5659c700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 0x7f5f30055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.668+0000 7f5f5659c700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 0x7f5f30055f30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f5f4801a000 tx=0x7f5f48014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.676+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f400432e0 con 0x7f5f58147000 2022-07-20T02:30:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.800+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5f580574e0 con 0x7f5f58147000 2022-07-20T02:30:45.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.800+0000 7f5f4f7fe700 1 -- 172.21.15.26:0/3342685950 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v66) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5f40046900 con 0x7f5f58147000 2022-07-20T02:30:45.805 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:30:45.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 msgr2=0x7f5f30055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 0x7f5f30055f30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f5f4801a000 tx=0x7f5f48014000).stop 2022-07-20T02:30:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 msgr2=0x7f5f58137010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f5f580af4d0 tx=0x7f5f40006df0).stop 2022-07-20T02:30:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 shutdown_connections 2022-07-20T02:30:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f58147000 0x7f5f58137010 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f30053a70 0x7f5f30055f30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f58137500 0x7f5f58155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 --2- 172.21.15.26:0/3342685950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f58060f00 0x7f5f581394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 >> 172.21.15.26:0/3342685950 conn(0x7f5f5805cfd0 msgr2=0x7f5f5813aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 shutdown_connections 2022-07-20T02:30:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.804+0000 7f5f5759e700 1 -- 172.21.15.26:0/3342685950 wait complete. 2022-07-20T02:30:45.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:30:45.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.928+0000 7f79e0cca700 1 Processor -- start 2022-07-20T02:30:45.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.928+0000 7f79e0cca700 1 -- start start 2022-07-20T02:30:45.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f79dc0a7620 0x7f79dc0b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0b2ea0 con 0x7f79dc148580 2022-07-20T02:30:45.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc147900 con 0x7f79dc0a7620 2022-07-20T02:30:45.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc147ba0 con 0x7f79dc0b2ac0 2022-07-20T02:30:45.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79dad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79dc148580 0x7f79dc066850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37966/0 (socket says 172.21.15.26:37966) 2022-07-20T02:30:45.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f79dc0a7620 0x7f79dc0b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79dad9d700 1 -- 172.21.15.26:0/2065977614 learned_addr learned my addr 172.21.15.26:0/2065977614 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:45.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79da59c700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 -- 172.21.15.26:0/2065977614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 msgr2=0x7f79dc0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:45.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 -- 172.21.15.26:0/2065977614 >> v1:172.21.15.26:6789/0 conn(0x7f79dc148580 legacy=0x7f79dc066850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 -- 172.21.15.26:0/2065977614 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc147e40 con 0x7f79dc0a7620 2022-07-20T02:30:45.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d9d9b700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc0a7620 0x7f79dc0b0460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f79cc0032f0 tx=0x7f79cc0091d0).ready entity=mon.0 client_cookie=683790b70f1f2d0b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d959a700 1 -- 172.21.15.26:0/2065977614 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79cc009ea0 con 0x7f79dc0a7620 2022-07-20T02:30:45.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d959a700 1 -- 172.21.15.26:0/2065977614 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc00b8a0 con 0x7f79dc0a7620 2022-07-20T02:30:45.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79d959a700 1 -- 172.21.15.26:0/2065977614 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79cc00ba10 con 0x7f79dc0a7620 2022-07-20T02:30:45.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- 172.21.15.26:0/2065977614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc0a7620 msgr2=0x7f79dc0b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc0a7620 0x7f79dc0b0460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f79cc0032f0 tx=0x7f79cc0091d0).stop 2022-07-20T02:30:45.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- 172.21.15.26:0/2065977614 shutdown_connections 2022-07-20T02:30:45.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc0a7620 0x7f79dc0b0460 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 --2- 172.21.15.26:0/2065977614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- 172.21.15.26:0/2065977614 >> 172.21.15.26:0/2065977614 conn(0x7f79dc0b60a0 msgr2=0x7f79dc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:45.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- 172.21.15.26:0/2065977614 shutdown_connections 2022-07-20T02:30:45.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.932+0000 7f79e0cca700 1 -- 172.21.15.26:0/2065977614 wait complete. 2022-07-20T02:30:45.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 Processor -- start 2022-07-20T02:30:45.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 -- start start 2022-07-20T02:30:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79dc0a7620 0x7f79dc0aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc159d00 con 0x7f79dc148580 2022-07-20T02:30:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc14e2a0 con 0x7f79dc0a7620 2022-07-20T02:30:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79e0cca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc14e4d0 con 0x7f79dc0b2ac0 2022-07-20T02:30:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79dad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79dad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38692/0 (socket says 172.21.15.26:38692) 2022-07-20T02:30:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79dad9d700 1 -- 172.21.15.26:0/301859026 learned_addr learned my addr 172.21.15.26:0/301859026 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79d9d9b700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.936+0000 7f79da59c700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79dc0a7620 0x7f79dc0aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 -- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 msgr2=0x7f79dc0aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 -- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79dc0a7620 msgr2=0x7f79dc0aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79dc0a7620 0x7f79dc0aa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 -- 172.21.15.26:0/301859026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc009470 con 0x7f79dc148580 2022-07-20T02:30:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79dad9d700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f79dc148460 tx=0x7f79d4007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d4001d70 con 0x7f79dc148580 2022-07-20T02:30:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d40070d0 con 0x7f79dc148580 2022-07-20T02:30:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc14e770 con 0x7f79dc148580 2022-07-20T02:30:45.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d4007430 con 0x7f79dc148580 2022-07-20T02:30:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc0ae8a0 con 0x7f79dc148580 2022-07-20T02:30:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f79d401a060 con 0x7f79dc148580 2022-07-20T02:30:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc1480e0 con 0x7f79dc148580 2022-07-20T02:30:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 0x7f79b8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f79d406b5b0 con 0x7f79dc148580 2022-07-20T02:30:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.940+0000 7f79da59c700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 0x7f79b8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:45.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.944+0000 7f79da59c700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 0x7f79b8055ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f79d000c000 tx=0x7f79d0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:45.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:45.948+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79d4042370 con 0x7f79dc148580 2022-07-20T02:30:46.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.072+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f79dc0574e0 con 0x7f79dc148580 2022-07-20T02:30:46.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.076+0000 7f79cb7fe700 1 -- 172.21.15.26:0/301859026 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7f79d4018070 con 0x7f79dc148580 2022-07-20T02:30:46.076 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:30:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.076+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 msgr2=0x7f79b8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.076+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 0x7f79b8055ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f79d000c000 tx=0x7f79d0008040).stop 2022-07-20T02:30:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.076+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 msgr2=0x7f79dc0af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.076+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f79dc148460 tx=0x7f79d4007c00).stop 2022-07-20T02:30:46.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 shutdown_connections 2022-07-20T02:30:46.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79dc148580 0x7f79dc0af320 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79b8053b30 0x7f79b8055ff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79dc0b2ac0 0x7f79dc0aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 --2- 172.21.15.26:0/301859026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79dc0a7620 0x7f79dc0aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 >> 172.21.15.26:0/301859026 conn(0x7f79dc0b60a0 msgr2=0x7f79dc149620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 shutdown_connections 2022-07-20T02:30:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.080+0000 7f79e0cca700 1 -- 172.21.15.26:0/301859026 wait complete. 2022-07-20T02:30:46.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:30:46.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.204+0000 7f712b1b2700 1 Processor -- start 2022-07-20T02:30:46.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.204+0000 7f712b1b2700 1 -- start start 2022-07-20T02:30:46.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f712405ce30 0x7f71240b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71240b5270 0x7f7124144740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7124147bb0 con 0x7f71240b0ed0 2022-07-20T02:30:46.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7124061080 con 0x7f712405ce30 2022-07-20T02:30:46.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71240b3960 con 0x7f71240b5270 2022-07-20T02:30:46.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f712405ce30 0x7f71240b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f712405ce30 0x7f71240b14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38698/0 (socket says 172.21.15.26:38698) 2022-07-20T02:30:46.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 -- 172.21.15.26:0/1277231537 learned_addr learned my addr 172.21.15.26:0/1277231537 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:46.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a9b1700 1 --2- 172.21.15.26:0/1277231537 >> v2:172.21.15.120:3300/0 conn(0x7f71240b5270 0x7f7124144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 -- 172.21.15.26:0/1277231537 >> v2:172.21.15.120:3300/0 conn(0x7f71240b5270 msgr2=0x7f7124144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:46.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 --2- 172.21.15.26:0/1277231537 >> v2:172.21.15.120:3300/0 conn(0x7f71240b5270 0x7f7124144740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 -- 172.21.15.26:0/1277231537 >> v1:172.21.15.120:6789/0 conn(0x7f71240b0ed0 legacy=0x7f71241474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 -- 172.21.15.26:0/1277231537 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240b3c00 con 0x7f712405ce30 2022-07-20T02:30:46.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712a1b0700 1 --2- 172.21.15.26:0/1277231537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712405ce30 0x7f71240b14f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f7120002f70 tx=0x7f7120006eb0).ready entity=mon.0 client_cookie=543a985854e6b014 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f71291ae700 1 -- 172.21.15.26:0/1277231537 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712000d5c0 con 0x7f712405ce30 2022-07-20T02:30:46.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f71291ae700 1 -- 172.21.15.26:0/1277231537 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120000e90 con 0x7f712405ce30 2022-07-20T02:30:46.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f71291ae700 1 -- 172.21.15.26:0/1277231537 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712000d950 con 0x7f712405ce30 2022-07-20T02:30:46.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- 172.21.15.26:0/1277231537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712405ce30 msgr2=0x7f71240b14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 --2- 172.21.15.26:0/1277231537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712405ce30 0x7f71240b14f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f7120002f70 tx=0x7f7120006eb0).stop 2022-07-20T02:30:46.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- 172.21.15.26:0/1277231537 shutdown_connections 2022-07-20T02:30:46.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 --2- 172.21.15.26:0/1277231537 >> v2:172.21.15.120:3300/0 conn(0x7f71240b5270 0x7f7124144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 --2- 172.21.15.26:0/1277231537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712405ce30 0x7f71240b14f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- 172.21.15.26:0/1277231537 >> 172.21.15.26:0/1277231537 conn(0x7f71240620c0 msgr2=0x7f712405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- 172.21.15.26:0/1277231537 shutdown_connections 2022-07-20T02:30:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.208+0000 7f712b1b2700 1 -- 172.21.15.26:0/1277231537 wait complete. 2022-07-20T02:30:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 Processor -- start 2022-07-20T02:30:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- start start 2022-07-20T02:30:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71240b0ed0 0x7f71240b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71240b5270 0x7f712414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712414acd0 con 0x7f712414a820 2022-07-20T02:30:46.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71240b0410 con 0x7f71240b0ed0 2022-07-20T02:30:46.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7124148a00 con 0x7f71240b5270 2022-07-20T02:30:46.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38704/0 (socket says 172.21.15.26:38704) 2022-07-20T02:30:46.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 -- 172.21.15.26:0/3122587245 learned_addr learned my addr 172.21.15.26:0/3122587245 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:46.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a1b0700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71240b0ed0 0x7f71240b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f71299af700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71240b5270 0x7f712414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 -- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71240b5270 msgr2=0x7f712414a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71240b5270 0x7f712414a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 -- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71240b0ed0 msgr2=0x7f71240b8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71240b0ed0 0x7f71240b8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 -- 172.21.15.26:0/3122587245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712000b1e0 con 0x7f712414a820 2022-07-20T02:30:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712a9b1700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f711c00c000 tx=0x7f711c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f711c010780 con 0x7f712414a820 2022-07-20T02:30:46.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c0108e0 con 0x7f712414a820 2022-07-20T02:30:46.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7124148ca0 con 0x7f712414a820 2022-07-20T02:30:46.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f711c010c40 con 0x7f712414a820 2022-07-20T02:30:46.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.212+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71240ae9e0 con 0x7f712414a820 2022-07-20T02:30:46.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f711c01c460 con 0x7f712414a820 2022-07-20T02:30:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71240af0d0 con 0x7f712414a820 2022-07-20T02:30:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f71177fe700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 0x7f7104055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f711c06c4f0 con 0x7f712414a820 2022-07-20T02:30:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f712a1b0700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 0x7f7104055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.216+0000 7f712a1b0700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 0x7f7104055f30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f712001a000 tx=0x7f7120015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.220+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f711c043250 con 0x7f712414a820 2022-07-20T02:30:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.348+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7124149140 con 0x7f712414a820 2022-07-20T02:30:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f71177fe700 1 -- 172.21.15.26:0/3122587245 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+10 (secure 0 0 0) 0x7f711c046870 con 0x7f712414a820 2022-07-20T02:30:46.351 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:30:46.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 msgr2=0x7f7104055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 0x7f7104055f30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f712001a000 tx=0x7f7120015000).stop 2022-07-20T02:30:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 msgr2=0x7f71240b97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f711c00c000 tx=0x7f711c006df0).stop 2022-07-20T02:30:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 shutdown_connections 2022-07-20T02:30:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f712414a820 0x7f71240b97c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7104053a70 0x7f7104055f30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71240b5270 0x7f712414a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 --2- 172.21.15.26:0/3122587245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71240b0ed0 0x7f71240b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 >> 172.21.15.26:0/3122587245 conn(0x7f712405ce30 msgr2=0x7f7124144300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 shutdown_connections 2022-07-20T02:30:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.352+0000 7f712b1b2700 1 -- 172.21.15.26:0/3122587245 wait complete. 2022-07-20T02:30:46.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:30:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.480+0000 7f5bdcd42700 1 Processor -- start 2022-07-20T02:30:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.480+0000 7f5bdcd42700 1 -- start start 2022-07-20T02:30:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5bd80b0690 0x7f5bd80af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bd80abd80 0x7f5bd814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd80b00e0 con 0x7f5bd805c8c0 2022-07-20T02:30:46.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd80acf00 con 0x7f5bd80b0690 2022-07-20T02:30:46.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd80accf0 con 0x7f5bd80abd80 2022-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5bd80b0690 0x7f5bd80af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5bd80b0690 0x7f5bd80af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38710/0 (socket says 172.21.15.26:38710) 2022-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 -- 172.21.15.26:0/824447689 learned_addr learned my addr 172.21.15.26:0/824447689 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bd7fff700 1 --2- 172.21.15.26:0/824447689 >> v2:172.21.15.120:3300/0 conn(0x7f5bd80abd80 0x7f5bd814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 -- 172.21.15.26:0/824447689 >> v2:172.21.15.120:3300/0 conn(0x7f5bd80abd80 msgr2=0x7f5bd814bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:46.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 --2- 172.21.15.26:0/824447689 >> v2:172.21.15.120:3300/0 conn(0x7f5bd80abd80 0x7f5bd814bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 -- 172.21.15.26:0/824447689 >> v1:172.21.15.120:6789/0 conn(0x7f5bd805c8c0 legacy=0x7f5bd80ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 -- 172.21.15.26:0/824447689 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd805cca0 con 0x7f5bd80b0690 2022-07-20T02:30:46.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bceffd700 1 --2- 172.21.15.26:0/824447689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd80b0690 0x7f5bd80af150 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f5bd80ad4d0 tx=0x7f5bc0006eb0).ready entity=mon.0 client_cookie=aa7b6655ed0c52ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bd6ffd700 1 -- 172.21.15.26:0/824447689 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc000d610 con 0x7f5bd80b0690 2022-07-20T02:30:46.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bd6ffd700 1 -- 172.21.15.26:0/824447689 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0000e90 con 0x7f5bd80b0690 2022-07-20T02:30:46.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bd6ffd700 1 -- 172.21.15.26:0/824447689 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc000d960 con 0x7f5bd80b0690 2022-07-20T02:30:46.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- 172.21.15.26:0/824447689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd80b0690 msgr2=0x7f5bd80af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/824447689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd80b0690 0x7f5bd80af150 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f5bd80ad4d0 tx=0x7f5bc0006eb0).stop 2022-07-20T02:30:46.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- 172.21.15.26:0/824447689 shutdown_connections 2022-07-20T02:30:46.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/824447689 >> v2:172.21.15.120:3300/0 conn(0x7f5bd80abd80 0x7f5bd814bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/824447689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd80b0690 0x7f5bd80af150 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- 172.21.15.26:0/824447689 >> 172.21.15.26:0/824447689 conn(0x7f5bd8060310 msgr2=0x7f5bd80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- 172.21.15.26:0/824447689 shutdown_connections 2022-07-20T02:30:46.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.484+0000 7f5bdcd42700 1 -- 172.21.15.26:0/824447689 wait complete. 2022-07-20T02:30:46.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 Processor -- start 2022-07-20T02:30:46.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- start start 2022-07-20T02:30:46.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bd80abd80 0x7f5bd809ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bd80a8090 0x7f5bd80a8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80a8a30 con 0x7f5bd805c8c0 2022-07-20T02:30:46.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80a0670 con 0x7f5bd80abd80 2022-07-20T02:30:46.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80a0810 con 0x7f5bd80a8090 2022-07-20T02:30:46.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38716/0 (socket says 172.21.15.26:38716) 2022-07-20T02:30:46.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 -- 172.21.15.26:0/2365068936 learned_addr learned my addr 172.21.15.26:0/2365068936 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:46.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bceffd700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bd80abd80 0x7f5bd809ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd7fff700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bd80a8090 0x7f5bd80a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 -- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bd80a8090 msgr2=0x7f5bd80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bd80a8090 0x7f5bd80a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 -- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bd80abd80 msgr2=0x7f5bd809ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bd80abd80 0x7f5bd809ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 -- 172.21.15.26:0/2365068936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc000b1e0 con 0x7f5bd805c8c0 2022-07-20T02:30:46.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd77fe700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5bd80a2060 tx=0x7f5bc8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc8012400 con 0x7f5bd805c8c0 2022-07-20T02:30:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8012560 con 0x7f5bd805c8c0 2022-07-20T02:30:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd80a2110 con 0x7f5bd805c8c0 2022-07-20T02:30:46.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc80128c0 con 0x7f5bd805c8c0 2022-07-20T02:30:46.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.488+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd80a2370 con 0x7f5bd805c8c0 2022-07-20T02:30:46.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd80a27b0 con 0x7f5bd805c8c0 2022-07-20T02:30:46.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5bc8019030 con 0x7f5bd805c8c0 2022-07-20T02:30:46.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bd57fa700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 0x7f5bbc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5bc806b3b0 con 0x7f5bd805c8c0 2022-07-20T02:30:46.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bceffd700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 0x7f5bbc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.492+0000 7f5bceffd700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 0x7f5bbc055fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5bc0019000 tx=0x7f5bc0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.500+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bc8042170 con 0x7f5bd805c8c0 2022-07-20T02:30:46.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.628+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5bd80afe40 con 0x7f5bd805c8c0 2022-07-20T02:30:46.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.628+0000 7f5bd57fa700 1 -- 172.21.15.26:0/2365068936 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5bc8045790 con 0x7f5bd805c8c0 2022-07-20T02:30:46.630 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:30:46.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 msgr2=0x7f5bbc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 0x7f5bbc055fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5bc0019000 tx=0x7f5bc0015000).stop 2022-07-20T02:30:46.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 msgr2=0x7f5bd809f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5bd80a2060 tx=0x7f5bc8007c50).stop 2022-07-20T02:30:46.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 shutdown_connections 2022-07-20T02:30:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5bbc053b00 0x7f5bbc055fc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bd80a8090 0x7f5bd80a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bd80abd80 0x7f5bd809ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 --2- 172.21.15.26:0/2365068936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bd805c8c0 0x7f5bd809f700 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 >> 172.21.15.26:0/2365068936 conn(0x7f5bd8060310 msgr2=0x7f5bd8062740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 shutdown_connections 2022-07-20T02:30:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.632+0000 7f5bdcd42700 1 -- 172.21.15.26:0/2365068936 wait complete. 2022-07-20T02:30:46.646 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:30:46.647 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:30:46.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 Processor -- start 2022-07-20T02:30:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 -- start start 2022-07-20T02:30:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f92a405cfd0 0x7f92a4143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f92a4060f00 0x7f92a413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a41474e0 con 0x7f92a4147000 2022-07-20T02:30:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a40b4e40 con 0x7f92a405cfd0 2022-07-20T02:30:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a8dce700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a405d3b0 con 0x7f92a4060f00 2022-07-20T02:30:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a2ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92a4147000 0x7f92a40b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38004/0 (socket says 172.21.15.26:38004) 2022-07-20T02:30:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a2ffd700 1 -- 172.21.15.26:0/559449168 learned_addr learned my addr 172.21.15.26:0/559449168 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:30:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 --2- 172.21.15.26:0/559449168 >> v2:172.21.15.26:3300/0 conn(0x7f92a405cfd0 0x7f92a4143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a3fff700 1 --2- 172.21.15.26:0/559449168 >> v2:172.21.15.120:3300/0 conn(0x7f92a4060f00 0x7f92a413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927216251 0 0) 0x7f92a41474e0 con 0x7f92a4147000 2022-07-20T02:30:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9288003260 con 0x7f92a4147000 2022-07-20T02:30:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 -- 172.21.15.26:0/559449168 >> v2:172.21.15.120:3300/0 conn(0x7f92a4060f00 msgr2=0x7f92a413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:30:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237134893 0 0) 0x7f9288003260 con 0x7f92a4147000 2022-07-20T02:30:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 --2- 172.21.15.26:0/559449168 >> v2:172.21.15.120:3300/0 conn(0x7f92a4060f00 0x7f92a413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 -- 172.21.15.26:0/559449168 >> v1:172.21.15.26:6789/0 conn(0x7f92a4147000 legacy=0x7f92a40b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 -- 172.21.15.26:0/559449168 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a41431e0 con 0x7f92a405cfd0 2022-07-20T02:30:46.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a37fe700 1 --2- 172.21.15.26:0/559449168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a405cfd0 0x7f92a4143bc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f928c001800 tx=0x7f9294006df0).ready entity=mon.0 client_cookie=3f0e6a681a9effb0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f928c001660 con 0x7f92a405cfd0 2022-07-20T02:30:46.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9294000e90 con 0x7f92a405cfd0 2022-07-20T02:30:46.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.752+0000 7f92a27fc700 1 -- 172.21.15.26:0/559449168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9294012460 con 0x7f92a405cfd0 2022-07-20T02:30:46.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- 172.21.15.26:0/559449168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a405cfd0 msgr2=0x7f92a4143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 --2- 172.21.15.26:0/559449168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a405cfd0 0x7f92a4143bc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f928c001800 tx=0x7f9294006df0).stop 2022-07-20T02:30:46.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- 172.21.15.26:0/559449168 shutdown_connections 2022-07-20T02:30:46.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 --2- 172.21.15.26:0/559449168 >> v2:172.21.15.120:3300/0 conn(0x7f92a4060f00 0x7f92a413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 --2- 172.21.15.26:0/559449168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a405cfd0 0x7f92a4143bc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- 172.21.15.26:0/559449168 >> 172.21.15.26:0/559449168 conn(0x7f92a40620c0 msgr2=0x7f92a405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- 172.21.15.26:0/559449168 shutdown_connections 2022-07-20T02:30:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- 172.21.15.26:0/559449168 wait complete. 2022-07-20T02:30:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 Processor -- start 2022-07-20T02:30:46.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.756+0000 7f92a8dce700 1 -- start start 2022-07-20T02:30:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4060f00 0x7f92a4145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92a40adb80 0x7f92a41557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4148cb0 con 0x7f92a4147000 2022-07-20T02:30:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40b2f30 con 0x7f92a4060f00 2022-07-20T02:30:46.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40b33d0 con 0x7f92a40adb80 2022-07-20T02:30:46.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38730/0 (socket says 172.21.15.26:38730) 2022-07-20T02:30:46.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 -- 172.21.15.26:0/81465603 learned_addr learned my addr 172.21.15.26:0/81465603 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:30:46.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a3fff700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92a40adb80 0x7f92a41557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a37fe700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4060f00 0x7f92a4145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 -- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92a40adb80 msgr2=0x7f92a41557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92a40adb80 0x7f92a41557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 -- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4060f00 msgr2=0x7f92a4145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4060f00 0x7f92a4145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 -- 172.21.15.26:0/81465603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929400b180 con 0x7f92a4147000 2022-07-20T02:30:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a2ffd700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f928c0018c0 tx=0x7f928c001e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f928c0072e0 con 0x7f92a4147000 2022-07-20T02:30:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c007440 con 0x7f92a4147000 2022-07-20T02:30:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a4145f30 con 0x7f92a4147000 2022-07-20T02:30:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f928c005800 con 0x7f92a4147000 2022-07-20T02:30:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.760+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a41488d0 con 0x7f92a4147000 2022-07-20T02:30:46.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4148af0 con 0x7f92a4147000 2022-07-20T02:30:46.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f928c01a060 con 0x7f92a4147000 2022-07-20T02:30:46.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a0ff9700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 0x7f927c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:30:46.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a37fe700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 0x7f927c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:30:46.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f928c06bb30 con 0x7f92a4147000 2022-07-20T02:30:46.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.764+0000 7f92a37fe700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 0x7f927c055f60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f9294018000 tx=0x7f9294014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:30:46.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.768+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f928c0427e0 con 0x7f92a4147000 2022-07-20T02:30:46.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:46.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:46.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:46.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:30:46.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.932+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92a4148e80 con 0x7f92a4147000 2022-07-20T02:30:46.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.932+0000 7f92a0ff9700 1 -- 172.21.15.26:0/81465603 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f928c018030 con 0x7f92a4147000 2022-07-20T02:30:46.938 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:30:46.938 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:30:46.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 msgr2=0x7f927c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 0x7f927c055f60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f9294018000 tx=0x7f9294014000).stop 2022-07-20T02:30:46.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 msgr2=0x7f92a40ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:30:46.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f928c0018c0 tx=0x7f928c001e60).stop 2022-07-20T02:30:46.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 shutdown_connections 2022-07-20T02:30:46.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92a4147000 0x7f92a40ad690 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f927c053aa0 0x7f927c055f60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92a40adb80 0x7f92a41557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 --2- 172.21.15.26:0/81465603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4060f00 0x7f92a4145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:30:46.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 >> 172.21.15.26:0/81465603 conn(0x7f92a405cfd0 msgr2=0x7f92a413bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:30:46.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 shutdown_connections 2022-07-20T02:30:46.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:30:46.936+0000 7f92a8dce700 1 -- 172.21.15.26:0/81465603 wait complete. 2022-07-20T02:30:46.960 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:30:47.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:30:42.193367+0000', 'modified': '2022-07-20T02:30:43.197173+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5997}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5997': {'gid': 5997, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2926574302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2926574302}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2926574302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:30:47.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:30:47.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:30:47.400 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":5997,"want_state":"up:active","state":"up:active","rank_uptime":5.1957106270000004,"mdsmap_epoch":30,"osdmap_epoch":66,"osdmap_epoch_barrier":66,"uptime":10.523366036000001} 2022-07-20T02:30:47.401 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 5997, "mdsmap_epoch": 30, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "rank_uptime": 5.195710627, "state": "up:active", "uptime": 10.523366036, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:30:47.401 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:30:47.402 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:30:47.402 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:30:47.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.403 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:47.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:30:47.440 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:30:47.440 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:30:47.498 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:30:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:30:47.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.501 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:30:47.513 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:30:47.514 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:47.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:30:47.560 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:30:47.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:30:47.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:30:47.612+0000 7f44fafe8080 -1 init, newargv = 0x56044b483510 newargc=9ceph-fuse[22735]: starting ceph client 2022-07-20T02:30:47.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:30:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:30:47.621 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:30:47.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.628 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:30:47.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22735]: starting fuse 2022-07-20T02:30:47.641 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:30:47.641 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:30:47.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:30:47.643 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:30:47.643 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:30:47.644 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:30:47.644 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:30:47.645 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.645 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:30:47.646 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:30:47.646 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:30:47.647 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.647 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:30:47.648 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:30:47.649 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:30:47.649 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:30:47.650 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.650 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:30:47.651 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:30:47.651 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:30:47.651 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:30:47.652 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:30:47.652 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.653 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:30:47.653 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:30:47.729 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22735.asok 2022-07-20T02:30:47.734 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22735.asok 2022-07-20T02:30:47.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.735 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22735.asok status 2022-07-20T02:30:47.826 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:30:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:30:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:30:47.827 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:30:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:30:47.828 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:30:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:30:47.829 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22735", 2022-07-20T02:30:47.830 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:30:47.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:30:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:30:47.831 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:30:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 6147, 2022-07-20T02:30:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:30:47.832 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:30:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:30:47.833 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 6147 2022-07-20T02:30:47.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:30:47.834 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:30:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:30:47.835 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:30:47.836 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 4124970326 2022-07-20T02:30:47.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:30:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:30:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:30:47.837 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:30:47.838 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:30:47.839 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 4124970326 2022-07-20T02:30:47.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:30:47.840 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.6147 172.21.15.159:0/4124970326", 2022-07-20T02:30:47.840 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/4124970326", 2022-07-20T02:30:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:30:47.841 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 30, 2022-07-20T02:30:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 66, 2022-07-20T02:30:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:30:47.842 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:30:47.843 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:30:47.843 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:30:47.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.846 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:47.847 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:47.857 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:30:47.858 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:30:47.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.859 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:47.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:30:47.922 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:30:47.923 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:30:47.979 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:30:47.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:30:47.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:30:47.982 DEBUG:teuthology.orchestra.run.smithi159:> import sys 2022-07-20T02:30:47.982 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:30:47.983 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:30:47.983 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.984 DEBUG:teuthology.orchestra.run.smithi159:> n = 1000 2022-07-20T02:30:47.984 DEBUG:teuthology.orchestra.run.smithi159:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T02:30:47.984 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.985 DEBUG:teuthology.orchestra.run.smithi159:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T02:30:47.986 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T02:30:47.987 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:30:47.987 DEBUG:teuthology.orchestra.run.smithi159:> for i in range(0, n): 2022-07-20T02:30:47.988 DEBUG:teuthology.orchestra.run.smithi159:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T02:30:47.988 DEBUG:teuthology.orchestra.run.smithi159:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T02:30:47.989 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:30:47.989 DEBUG:teuthology.orchestra.run.smithi159:> if True: 2022-07-20T02:30:47.989 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:30:47.990 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(f.fileno()) 2022-07-20T02:30:47.990 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:30:59.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:00.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 Processor -- start 2022-07-20T02:31:00.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 -- start start 2022-07-20T02:31:00.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57b80b30c0 0x7f57b8060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b8147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b814b730 con 0x7f57b805f840 2022-07-20T02:31:00.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b8095a50 con 0x7f57b80b30c0 2022-07-20T02:31:00.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b814b9d0 con 0x7f57b80952f0 2022-07-20T02:31:00.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57b80b30c0 0x7f57b8060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57b80b30c0 0x7f57b8060570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38738/0 (socket says 172.21.15.26:38738) 2022-07-20T02:31:00.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 -- 172.21.15.26:0/2922032150 learned_addr learned my addr 172.21.15.26:0/2922032150 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bd2a4700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b8147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 -- 172.21.15.26:0/2922032150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 msgr2=0x7f57b8147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:00.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b8147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 -- 172.21.15.26:0/2922032150 >> v1:172.21.15.120:6789/0 conn(0x7f57b805f840 legacy=0x7f57b8066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 -- 172.21.15.26:0/2922032150 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8147760 con 0x7f57b80b30c0 2022-07-20T02:31:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bdaa5700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b80b30c0 0x7f57b8060570 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f57b4002f70 tx=0x7f57b4006eb0).ready entity=mon.0 client_cookie=7377fb84c6658222 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bcaa3700 1 -- 172.21.15.26:0/2922032150 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b400d570 con 0x7f57b80b30c0 2022-07-20T02:31:00.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bcaa3700 1 -- 172.21.15.26:0/2922032150 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4000e90 con 0x7f57b80b30c0 2022-07-20T02:31:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57bcaa3700 1 -- 172.21.15.26:0/2922032150 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b400d900 con 0x7f57b80b30c0 2022-07-20T02:31:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 -- 172.21.15.26:0/2922032150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b80b30c0 msgr2=0x7f57b8060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.012+0000 7f57beaa7700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b80b30c0 0x7f57b8060570 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f57b4002f70 tx=0x7f57b4006eb0).stop 2022-07-20T02:31:00.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- 172.21.15.26:0/2922032150 shutdown_connections 2022-07-20T02:31:00.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b8147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 --2- 172.21.15.26:0/2922032150 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b80b30c0 0x7f57b8060570 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- 172.21.15.26:0/2922032150 >> 172.21.15.26:0/2922032150 conn(0x7f57b80b67f0 msgr2=0x7f57b80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:00.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- 172.21.15.26:0/2922032150 shutdown_connections 2022-07-20T02:31:00.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- 172.21.15.26:0/2922032150 wait complete. 2022-07-20T02:31:00.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 Processor -- start 2022-07-20T02:31:00.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- start start 2022-07-20T02:31:00.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b809ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b80b30c0 0x7f57b809b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80644c0 con 0x7f57b805f840 2022-07-20T02:31:00.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b809b7f0 con 0x7f57b80b30c0 2022-07-20T02:31:00.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57beaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b815aa40 con 0x7f57b80952f0 2022-07-20T02:31:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38744/0 (socket says 172.21.15.26:38744) 2022-07-20T02:31:00.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 -- 172.21.15.26:0/3126028828 learned_addr learned my addr 172.21.15.26:0/3126028828 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bd2a4700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b809ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57be2a6700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b80b30c0 0x7f57b809b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 -- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 msgr2=0x7f57b809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b809ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 -- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b80b30c0 msgr2=0x7f57b809b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b80b30c0 0x7f57b809b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 -- 172.21.15.26:0/3126028828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b400b1e0 con 0x7f57b805f840 2022-07-20T02:31:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.016+0000 7f57bdaa5700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f57b4002f70 tx=0x7f57b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b40117e0 con 0x7f57b805f840 2022-07-20T02:31:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4011940 con 0x7f57b805f840 2022-07-20T02:31:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b4011ca0 con 0x7f57b805f840 2022-07-20T02:31:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b80646f0 con 0x7f57b805f840 2022-07-20T02:31:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b814de00 con 0x7f57b805f840 2022-07-20T02:31:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b8064b90 con 0x7f57b805f840 2022-07-20T02:31:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f57b401a040 con 0x7f57b805f840 2022-07-20T02:31:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 0x7f57a0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f57b406cfb0 con 0x7f57b805f840 2022-07-20T02:31:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57bd2a4700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 0x7f57a0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.020+0000 7f57bd2a4700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 0x7f57a0055e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f57a800b000 tx=0x7f57a8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.028+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57b8064b90 con 0x7f57b805f840 2022-07-20T02:31:00.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b8064b90 con 0x7f57b805f840 2022-07-20T02:31:00.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57aeffd700 1 -- 172.21.15.26:0/3126028828 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f57b8064b90 con 0x7f57b805f840 2022-07-20T02:31:00.197 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:00.212 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:31:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 msgr2=0x7f57a0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 0x7f57a0055e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f57a800b000 tx=0x7f57a8006df0).stop 2022-07-20T02:31:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 msgr2=0x7f57b814d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f57b4002f70 tx=0x7f57b4015000).stop 2022-07-20T02:31:00.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 shutdown_connections 2022-07-20T02:31:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57a0053950 0x7f57a0055e10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b80b30c0 0x7f57b809b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57b80952f0 0x7f57b809ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 --2- 172.21.15.26:0/3126028828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57b805f840 0x7f57b814d250 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 >> 172.21.15.26:0/3126028828 conn(0x7f57b80b67f0 msgr2=0x7f57b814a840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:00.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 shutdown_connections 2022-07-20T02:31:00.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.192+0000 7f57beaa7700 1 -- 172.21.15.26:0/3126028828 wait complete. 2022-07-20T02:31:00.221 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:31:00.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-20T02:31:00.345 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T02:31:00.346 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T02:31:00.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.452+0000 7f5d034f5700 1 Processor -- start 2022-07-20T02:31:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.452+0000 7f5d034f5700 1 -- start start 2022-07-20T02:31:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cfc0b62d0 con 0x7f5cfc0b1850 2022-07-20T02:31:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cfc0b0d60 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0a9cf0 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38756/0 (socket says 172.21.15.26:38756) 2022-07-20T02:31:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 -- 172.21.15.26:0/67996204 learned_addr learned my addr 172.21.15.26:0/67996204 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d01cf2700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 -- 172.21.15.26:0/67996204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 msgr2=0x7f5cfc0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d014f1700 1 -- 172.21.15.26:0/67996204 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370335465 0 0) 0x7f5cfc0b62d0 con 0x7f5cfc0b1850 2022-07-20T02:31:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc0b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 -- 172.21.15.26:0/67996204 >> v1:172.21.15.120:6789/0 conn(0x7f5cfc0b1850 legacy=0x7f5cfc1426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 -- 172.21.15.26:0/67996204 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc0a9f90 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d01cf2700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d024f3700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f5cf8002f70 tx=0x7f5cf8006f40).ready entity=mon.0 client_cookie=50a7cdd0aed48919 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d014f1700 1 -- 172.21.15.26:0/67996204 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf800da80 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d014f1700 1 -- 172.21.15.26:0/67996204 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf8000e90 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d014f1700 1 -- 172.21.15.26:0/67996204 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf800ddd0 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- 172.21.15.26:0/67996204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b5ef0 msgr2=0x7f5cfc0b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f5cf8002f70 tx=0x7f5cf8006f40).stop 2022-07-20T02:31:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- 172.21.15.26:0/67996204 shutdown_connections 2022-07-20T02:31:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 --2- 172.21.15.26:0/67996204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc0b0380 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- 172.21.15.26:0/67996204 >> 172.21.15.26:0/67996204 conn(0x7f5cfc060310 msgr2=0x7f5cfc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- 172.21.15.26:0/67996204 shutdown_connections 2022-07-20T02:31:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.456+0000 7f5d034f5700 1 -- 172.21.15.26:0/67996204 wait complete. 2022-07-20T02:31:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 Processor -- start 2022-07-20T02:31:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 -- start start 2022-07-20T02:31:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc13dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc13c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0b4160 con 0x7f5cfc0b1850 2022-07-20T02:31:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0b5790 con 0x7f5cfc0b5ef0 2022-07-20T02:31:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc151ff0 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d01cf2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc13dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d01cf2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38760/0 (socket says 172.21.15.26:38760) 2022-07-20T02:31:00.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d01cf2700 1 -- 172.21.15.26:0/1784583195 learned_addr learned my addr 172.21.15.26:0/1784583195 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d02cf4700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc13c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 -- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cfc0b5ef0 msgr2=0x7f5cfc13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc13c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 -- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 msgr2=0x7f5cfc0626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 -- 172.21.15.26:0/1784583195 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec00b180 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d01cf2700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:00.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d024f3700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc13dff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5cfc064910 tx=0x7f5cfc0ac070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf800de40 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf8018400 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf800b460 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.460+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf8018760 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc14bbb0 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc0574e0 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5cf8021030 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5ceb7fe700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 0x7f5cdc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5cf806f480 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5d01cf2700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 0x7f5cdc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.464+0000 7f5d01cf2700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 0x7f5cdc055f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5cec00b000 tx=0x7f5cec006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.472+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5cf8046240 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.640+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cfc0b42b0 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.640+0000 7f5ceb7fe700 1 -- 172.21.15.26:0/1784583195 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f5cf8049860 con 0x7f5cfc0a97e0 2022-07-20T02:31:00.640 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:00.641 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:31:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 msgr2=0x7f5cdc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 0x7f5cdc055f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5cec00b000 tx=0x7f5cec006df0).stop 2022-07-20T02:31:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 msgr2=0x7f5cfc13dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc13dff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5cfc064910 tx=0x7f5cfc0ac070).stop 2022-07-20T02:31:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 shutdown_connections 2022-07-20T02:31:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cfc0b1850 0x7f5cfc0626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cdc053a70 0x7f5cdc055f30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cfc0b5ef0 0x7f5cfc13c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 --2- 172.21.15.26:0/1784583195 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cfc0a97e0 0x7f5cfc13dff0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 >> 172.21.15.26:0/1784583195 conn(0x7f5cfc060310 msgr2=0x7f5cfc143430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 shutdown_connections 2022-07-20T02:31:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.644+0000 7f5d034f5700 1 -- 172.21.15.26:0/1784583195 wait complete. 2022-07-20T02:31:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:31:00.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T02:31:00.789 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T02:31:00.790 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T02:31:00.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:00.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 Processor -- start 2022-07-20T02:31:00.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- start start 2022-07-20T02:31:00.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1de8067190 0x7f1de8148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80b1bc0 con 0x7f1de8061050 2022-07-20T02:31:00.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de814b040 con 0x7f1de8067190 2022-07-20T02:31:00.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80a8860 con 0x7f1de80b2920 2022-07-20T02:31:00.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60282/0 (socket says 172.21.15.26:60282) 2022-07-20T02:31:00.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 -- 172.21.15.26:0/57874191 learned_addr learned my addr 172.21.15.26:0/57874191 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1deda43700 1 --2- 172.21.15.26:0/57874191 >> v2:172.21.15.120:3300/0 conn(0x7f1de8067190 0x7f1de8148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 -- 172.21.15.26:0/57874191 >> v2:172.21.15.120:3300/0 conn(0x7f1de8067190 msgr2=0x7f1de8148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:00.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 --2- 172.21.15.26:0/57874191 >> v2:172.21.15.120:3300/0 conn(0x7f1de8067190 0x7f1de8148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 -- 172.21.15.26:0/57874191 >> v1:172.21.15.120:6789/0 conn(0x7f1de8061050 legacy=0x7f1de8066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 -- 172.21.15.26:0/57874191 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8147700 con 0x7f1de80b2920 2022-07-20T02:31:00.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ded242700 1 --2- 172.21.15.26:0/57874191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1de40032f0 tx=0x7f1de4009290).ready entity=mon.2 client_cookie=c29e4f58c4609133 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ddffff700 1 -- 172.21.15.26:0/57874191 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1de400b8a0 con 0x7f1de80b2920 2022-07-20T02:31:00.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ddffff700 1 -- 172.21.15.26:0/57874191 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400ba00 con 0x7f1de80b2920 2022-07-20T02:31:00.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1ddffff700 1 -- 172.21.15.26:0/57874191 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1de400bb70 con 0x7f1de80b2920 2022-07-20T02:31:00.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- 172.21.15.26:0/57874191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 msgr2=0x7f1de8147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 --2- 172.21.15.26:0/57874191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1de40032f0 tx=0x7f1de4009290).stop 2022-07-20T02:31:00.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- 172.21.15.26:0/57874191 shutdown_connections 2022-07-20T02:31:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 --2- 172.21.15.26:0/57874191 >> v2:172.21.15.120:3300/0 conn(0x7f1de8067190 0x7f1de8148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 --2- 172.21.15.26:0/57874191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de80b2920 0x7f1de8147d90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.900+0000 7f1def4a6700 1 -- 172.21.15.26:0/57874191 >> 172.21.15.26:0/57874191 conn(0x7f1de80b60a0 msgr2=0x7f1de80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:00.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.904+0000 7f1def4a6700 1 -- 172.21.15.26:0/57874191 shutdown_connections 2022-07-20T02:31:00.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.904+0000 7f1def4a6700 1 -- 172.21.15.26:0/57874191 wait complete. 2022-07-20T02:31:00.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.904+0000 7f1def4a6700 1 Processor -- start 2022-07-20T02:31:00.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.904+0000 7f1def4a6700 1 -- start start 2022-07-20T02:31:00.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de8067190 0x7f1de81574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 0x7f1de80664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80ab080 con 0x7f1de8061050 2022-07-20T02:31:00.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80ac240 con 0x7f1de80b2920 2022-07-20T02:31:00.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8157990 con 0x7f1de8067190 2022-07-20T02:31:00.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38772/0 (socket says 172.21.15.26:38772) 2022-07-20T02:31:00.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 -- 172.21.15.26:0/1698659048 learned_addr learned my addr 172.21.15.26:0/1698659048 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:00.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1deca41700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de8067190 0x7f1de81574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1deda43700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 0x7f1de80664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 -- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de8067190 msgr2=0x7f1de81574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de8067190 0x7f1de81574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 -- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 msgr2=0x7f1de80664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:00.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 0x7f1de80664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:00.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 -- 172.21.15.26:0/1698659048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4009530 con 0x7f1de8061050 2022-07-20T02:31:00.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1deda43700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 0x7f1de80664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:00.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1ded242700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f1de40032f0 tx=0x7f1de4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1de400b8a0 con 0x7f1de8061050 2022-07-20T02:31:00.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4021630 con 0x7f1de8061050 2022-07-20T02:31:00.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de80aba60 con 0x7f1de8061050 2022-07-20T02:31:00.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1de4021990 con 0x7f1de8061050 2022-07-20T02:31:00.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de80abfa0 con 0x7f1de8061050 2022-07-20T02:31:00.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.908+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de814ada0 con 0x7f1de8061050 2022-07-20T02:31:00.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.912+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1de4027020 con 0x7f1de8061050 2022-07-20T02:31:00.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.912+0000 7f1dde7fc700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 0x7f1dd0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:00.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.912+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1de406d430 con 0x7f1de8061050 2022-07-20T02:31:00.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.912+0000 7f1deca41700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 0x7f1dd0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:00.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.912+0000 7f1deca41700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 0x7f1dd0055ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1dd800d000 tx=0x7f1dd8000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:00.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:00.920+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1de4044190 con 0x7f1de8061050 2022-07-20T02:31:01.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de80574e0 con 0x7f1de8061050 2022-07-20T02:31:01.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1dde7fc700 1 -- 172.21.15.26:0/1698659048 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f1de4015020 con 0x7f1de8061050 2022-07-20T02:31:01.080 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:01.080 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:31:01.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 msgr2=0x7f1dd0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 0x7f1dd0055ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1dd800d000 tx=0x7f1dd8000bc0).stop 2022-07-20T02:31:01.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 msgr2=0x7f1de8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f1de40032f0 tx=0x7f1de4016000).stop 2022-07-20T02:31:01.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 shutdown_connections 2022-07-20T02:31:01.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1dd0053a10 0x7f1dd0055ed0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1de80b2920 0x7f1de80664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1de8067190 0x7f1de81574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 --2- 172.21.15.26:0/1698659048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8061050 0x7f1de8062b50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 >> 172.21.15.26:0/1698659048 conn(0x7f1de80b60a0 msgr2=0x7f1de814b180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:01.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 shutdown_connections 2022-07-20T02:31:01.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.080+0000 7f1def4a6700 1 -- 172.21.15.26:0/1698659048 wait complete. 2022-07-20T02:31:01.085 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:31:01.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 10 2022-07-20T02:31:01.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 Processor -- start 2022-07-20T02:31:01.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 -- start start 2022-07-20T02:31:01.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7f600b5ef0 0x7f7f601474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f600af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f600b62d0 con 0x7f7f600a7fc0 2022-07-20T02:31:01.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6014aac0 con 0x7f7f600b5ef0 2022-07-20T02:31:01.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f66c50700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6014ad60 con 0x7f7f600a8c40 2022-07-20T02:31:01.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7f600a7fc0 0x7f7f600aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38062/0 (socket says 172.21.15.26:38062) 2022-07-20T02:31:01.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 -- 172.21.15.26:0/417054137 learned_addr learned my addr 172.21.15.26:0/417054137 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:01.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6644f700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f600af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/417054137 >> v2:172.21.15.120:3300/0 conn(0x7f7f600b5ef0 0x7f7f601474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547628192 0 0) 0x7f7f600b62d0 con 0x7f7f600a7fc0 2022-07-20T02:31:01.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44003260 con 0x7f7f600a7fc0 2022-07-20T02:31:01.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543301614 0 0) 0x7f7f44003260 con 0x7f7f600a7fc0 2022-07-20T02:31:01.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 msgr2=0x7f7f600af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f600af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 >> v2:172.21.15.120:3300/0 conn(0x7f7f600b5ef0 msgr2=0x7f7f601474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 --2- 172.21.15.26:0/417054137 >> v2:172.21.15.120:3300/0 conn(0x7f7f600b5ef0 0x7f7f601474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6014b000 con 0x7f7f600a7fc0 2022-07-20T02:31:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f7f50001660 con 0x7f7f600a7fc0 2022-07-20T02:31:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 >> v1:172.21.15.26:6789/0 conn(0x7f7f600a7fc0 legacy=0x7f7f600aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f44003750 0x7f7f44003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f44004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f440049f0 0x7f7f44004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6014aac0 con 0x7f7f44003750 2022-07-20T02:31:01.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6014ad60 con 0x7f7f440049f0 2022-07-20T02:31:01.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4400b750 con 0x7f7f600a8c40 2022-07-20T02:31:01.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f44003750 0x7f7f44003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f44004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6644f700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f440049f0 0x7f7f44004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 -- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f440049f0 msgr2=0x7f7f44004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f440049f0 0x7f7f44004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 -- 172.21.15.26:0/417054137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f44003750 msgr2=0x7f7f44003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f44003750 0x7f7f44003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 -- 172.21.15.26:0/417054137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44003260 con 0x7f7f600a8c40 2022-07-20T02:31:01.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6644f700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f440049f0 0x7f7f44004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:01.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.204+0000 7f7f6544d700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f44004500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7f50001370 tx=0x7f7f50001e50).ready entity=mon.2 client_cookie=21be5584a6c82bb6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f500070d0 con 0x7f7f600a8c40 2022-07-20T02:31:01.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f50007230 con 0x7f7f600a8c40 2022-07-20T02:31:01.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f64c4c700 1 -- 172.21.15.26:0/417054137 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5000c8f0 con 0x7f7f600a8c40 2022-07-20T02:31:01.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 msgr2=0x7f7f44004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 --2- 172.21.15.26:0/417054137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f44004500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7f50001370 tx=0x7f7f50001e50).stop 2022-07-20T02:31:01.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f6644f700 1 -- 172.21.15.26:0/417054137 reap_dead start 2022-07-20T02:31:01.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- 172.21.15.26:0/417054137 shutdown_connections 2022-07-20T02:31:01.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- 172.21.15.26:0/417054137 >> 172.21.15.26:0/417054137 conn(0x7f7f60060310 msgr2=0x7f7f600b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:01.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- 172.21.15.26:0/417054137 shutdown_connections 2022-07-20T02:31:01.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- 172.21.15.26:0/417054137 wait complete. 2022-07-20T02:31:01.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 Processor -- start 2022-07-20T02:31:01.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.208+0000 7f7f66c50700 1 -- start start 2022-07-20T02:31:01.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f601467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 0x7f7f600b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60145730 con 0x7f7f600abde0 2022-07-20T02:31:01.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600a9430 con 0x7f7f60067b00 2022-07-20T02:31:01.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f601480b0 con 0x7f7f600a8c40 2022-07-20T02:31:01.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 0x7f7f600b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38796/0 (socket says 172.21.15.26:38796) 2022-07-20T02:31:01.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f65c4e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f601467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 -- 172.21.15.26:0/3491596269 learned_addr learned my addr 172.21.15.26:0/3491596269 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:01.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 -- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 msgr2=0x7f7f601467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f601467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 -- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 msgr2=0x7f7f600b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 0x7f7f600b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 -- 172.21.15.26:0/3491596269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f60148350 con 0x7f7f600abde0 2022-07-20T02:31:01.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6544d700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 0x7f7f600b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:01.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f601467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:01.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f6644f700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f7f5800b000 tx=0x7f7f580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f56ffd700 1 -- 172.21.15.26:0/3491596269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f58019070 con 0x7f7f600abde0 2022-07-20T02:31:01.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f56ffd700 1 -- 172.21.15.26:0/3491596269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5800f9e0 con 0x7f7f600abde0 2022-07-20T02:31:01.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600ac260 con 0x7f7f600abde0 2022-07-20T02:31:01.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f56ffd700 1 -- 172.21.15.26:0/3491596269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5800fd40 con 0x7f7f600abde0 2022-07-20T02:31:01.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f600b4260 con 0x7f7f600abde0 2022-07-20T02:31:01.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.212+0000 7f7f56ffd700 1 -- 172.21.15.26:0/3491596269 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7f58014440 con 0x7f7f600abde0 2022-07-20T02:31:01.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f56ffd700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 0x7f7f48055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f56ffd700 1 -- 172.21.15.26:0/3491596269 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7f5806c990 con 0x7f7f600abde0 2022-07-20T02:31:01.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 0x7f7f48055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 0x7f7f48055ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f7f5c00b000 tx=0x7f7f5c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 Processor -- start 2022-07-20T02:31:01.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 -- start start 2022-07-20T02:31:01.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2617ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x261b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x258a210 con 0x264fa90 2022-07-20T02:31:01.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x256a450 con 0x26176f0 2022-07-20T02:31:01.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2617390 con 0x261b010 2022-07-20T02:31:01.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38802/0 (socket says 172.21.15.26:38802) 2022-07-20T02:31:01.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 -- 172.21.15.26:0/2173362170 learned_addr learned my addr 172.21.15.26:0/2173362170 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:01.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6644f700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x261b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2617ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 -- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 msgr2=0x261b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x261b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 -- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 msgr2=0x2617ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2617ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 -- 172.21.15.26:0/2173362170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25c1ad0 con 0x264fa90 2022-07-20T02:31:01.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6644f700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x261b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:01.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7f5001b000 tx=0x7f7f50015000).ready entity=mon.0 client_cookie=2cd4ac1fc98e90e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f66c50700 1 -- 172.21.15.26:0/2173362170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5000c8f0 con 0x264fa90 2022-07-20T02:31:01.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f66c50700 1 -- 172.21.15.26:0/2173362170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5000ca50 con 0x264fa90 2022-07-20T02:31:01.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.216+0000 7f7f66c50700 1 -- 172.21.15.26:0/2173362170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5000cdb0 con 0x264fa90 2022-07-20T02:31:01.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- 172.21.15.26:0/2173362170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 msgr2=0x264fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7f5001b000 tx=0x7f7f50015000).stop 2022-07-20T02:31:01.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- 172.21.15.26:0/2173362170 shutdown_connections 2022-07-20T02:31:01.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x264fef0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x261b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- 172.21.15.26:0/2173362170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2617ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- 172.21.15.26:0/2173362170 >> 172.21.15.26:0/2173362170 conn(0x25a18f0 msgr2=0x2618470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:01.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- 172.21.15.26:0/2173362170 shutdown_connections 2022-07-20T02:31:01.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- 172.21.15.26:0/2173362170 wait complete. 2022-07-20T02:31:01.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- start start 2022-07-20T02:31:01.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 Processor -- start 2022-07-20T02:31:01.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2661620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x264ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2620080 con 0x261b010 2022-07-20T02:31:01.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x264b250 con 0x26176f0 2022-07-20T02:31:01.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x26202a0 con 0x264fa90 2022-07-20T02:31:01.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f72109740 10 client.0 ll_register_callbacks cb 0x25d10b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:31:01.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38808/0 (socket says 172.21.15.26:38808) 2022-07-20T02:31:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 -- 172.21.15.26:0/2476678625 learned_addr learned my addr 172.21.15.26:0/2476678625 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6644f700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x264ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2661620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 -- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 msgr2=0x264ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x264ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 -- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 msgr2=0x2661620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2661620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 -- 172.21.15.26:0/2476678625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f44003260 con 0x261b010 2022-07-20T02:31:01.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f6544d700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x26201c0 tx=0x7f7f5001f000).ready entity=mon.0 client_cookie=f791885988db0673 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.220+0000 7f7f40ff9700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:31:01.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5000c6d0 con 0x261b010 2022-07-20T02:31:01.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x25801c0 con 0x261b010 2022-07-20T02:31:01.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7f5002cdb0 con 0x261b010 2022-07-20T02:31:01.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f50004520 con 0x261b010 2022-07-20T02:31:01.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 10 client.6174 fetch_fsmap learned FSMap version 30 2022-07-20T02:31:01.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x25797b0 con 0x261b010 2022-07-20T02:31:01.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x25801c0 con 0x261b010 2022-07-20T02:31:01.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f7f50004c90 con 0x261b010 2022-07-20T02:31:01.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 10 client.6174 fetch_fsmap finished waiting for FSMap version 30 2022-07-20T02:31:01.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 10 client.6174 resolve_mds: resolved ID 'a' to GID 5997 2022-07-20T02:31:01.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 20 client.6174 populate_metadata read hostname 'smithi026' 2022-07-20T02:31:01.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x2616c00 0x265e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 4 client.6174 mds_command: new command op to 5997 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T02:31:01.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 --> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2578240 con 0x2616c00 2022-07-20T02:31:01.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x2616c00 0x265e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 10 client.6174 ms_handle_connect on v2:172.21.15.26:6836/2926574302 2022-07-20T02:31:01.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x2616c00 0x265e7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=ef764078bb301418 server_cookie=b8581bd75fba1d13 in_seq=0 out_seq=0 2022-07-20T02:31:01.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 1 -- 172.21.15.26:0/2476678625 <== mds.0 v2:172.21.15.26:6836/2926574302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x2578240 con 0x2616c00 2022-07-20T02:31:01.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 10 client.6174 handle_command_reply: tid=0 2022-07-20T02:31:01.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 client.6174 shutdown 2022-07-20T02:31:01.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 msgr2=0x261fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x26201c0 tx=0x7f7f5001f000).stop 2022-07-20T02:31:01.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 shutdown_connections 2022-07-20T02:31:01.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x2616c00 0x265e7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x261b010 0x261fb90 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x264ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 --2- 172.21.15.26:0/2476678625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x2661620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f40ff9700 0 client.6174 ms_handle_reset on v2:172.21.15.26:6836/2926574302 2022-07-20T02:31:01.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 >> 172.21.15.26:0/2476678625 conn(0x25a18f0 msgr2=0x261b740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:01.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 shutdown_connections 2022-07-20T02:31:01.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 1 -- 172.21.15.26:0/2476678625 wait complete. 2022-07-20T02:31:01.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.224+0000 7f7f72109740 20 client.6174 trim_cache size 0 max 16384 2022-07-20T02:31:01.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 Processor -- start 2022-07-20T02:31:01.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 -- start start 2022-07-20T02:31:01.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x261b010 0x261ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x2614ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x2578240 con 0x26176f0 2022-07-20T02:31:01.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x25904e0 con 0x261b010 2022-07-20T02:31:01.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x2650990 con 0x264fa90 2022-07-20T02:31:01.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38816/0 (socket says 172.21.15.26:38816) 2022-07-20T02:31:01.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 -- 172.21.15.26:0/3008019252 learned_addr learned my addr 172.21.15.26:0/3008019252 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:01.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f6644f700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x2614ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f6544d700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x261b010 0x261ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 -- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 msgr2=0x2614ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x2614ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 -- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x261b010 msgr2=0x261ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x261b010 0x261ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 -- 172.21.15.26:0/3008019252 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25823a0 con 0x26176f0 2022-07-20T02:31:01.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f6644f700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x2614ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:01.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x2590620 tx=0x7f7f5c018000).ready entity=mon.0 client_cookie=c6b05f5c2fcb00e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f42ffd700 1 -- 172.21.15.26:0/3008019252 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5c026040 con 0x26176f0 2022-07-20T02:31:01.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f42ffd700 1 -- 172.21.15.26:0/3008019252 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c02f070 con 0x26176f0 2022-07-20T02:31:01.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f42ffd700 1 -- 172.21.15.26:0/3008019252 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5c02bea0 con 0x26176f0 2022-07-20T02:31:01.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 -- 172.21.15.26:0/3008019252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 msgr2=0x261b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:01.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.236+0000 7f7f72109740 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x2590620 tx=0x7f7f5c018000).stop 2022-07-20T02:31:01.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/3008019252 shutdown_connections 2022-07-20T02:31:01.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x264fa90 0x2614ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x261b010 0x261ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- 172.21.15.26:0/3008019252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x26176f0 0x261b530 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/3008019252 >> 172.21.15.26:0/3008019252 conn(0x25a18f0 msgr2=0x2618240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:01.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/3008019252 shutdown_connections 2022-07-20T02:31:01.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/3008019252 wait complete. 2022-07-20T02:31:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- start start 2022-07-20T02:31:01.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 Processor -- start 2022-07-20T02:31:01.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x264bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x2657740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x26b1840 con 0x264fa90 2022-07-20T02:31:01.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x26b1d10 con 0x26176f0 2022-07-20T02:31:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x26aea30 con 0x261b010 2022-07-20T02:31:01.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 10 client.0 ll_register_callbacks cb 0x261f8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T02:31:01.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6544d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x2657740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38824/0 (socket says 172.21.15.26:38824) 2022-07-20T02:31:01.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 -- 172.21.15.26:0/1069121173 learned_addr learned my addr 172.21.15.26:0/1069121173 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:01.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x264bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 -- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 msgr2=0x2657740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x2657740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 -- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 msgr2=0x264bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:01.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x264bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:01.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 -- 172.21.15.26:0/1069121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c02b500 con 0x264fa90 2022-07-20T02:31:01.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6544d700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x2657740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:01.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2022-07-20T02:31:01.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f6644f700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f7f58026000 tx=0x7f7f5806c040).ready entity=mon.0 client_cookie=87d8e266408dfa5f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:01.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5802f070 con 0x264fa90 2022-07-20T02:31:01.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f5000ca50 con 0x264fa90 2022-07-20T02:31:01.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7f5801bb70 con 0x264fa90 2022-07-20T02:31:01.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f5801bd30 con 0x264fa90 2022-07-20T02:31:01.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 10 client.6180 fetch_fsmap learned FSMap version 30 2022-07-20T02:31:01.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x257ed90 con 0x264fa90 2022-07-20T02:31:01.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5000ca50 con 0x264fa90 2022-07-20T02:31:01.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f7f580431a0 con 0x264fa90 2022-07-20T02:31:01.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 10 client.6180 fetch_fsmap finished waiting for FSMap version 30 2022-07-20T02:31:01.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 10 client.6180 resolve_mds: resolved ID 'a' to GID 5997 2022-07-20T02:31:01.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 20 client.6180 populate_metadata read hostname 'smithi026' 2022-07-20T02:31:01.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x26aadf0 0x26b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:01.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 4 client.6180 mds_command: new command op to 5997 tid=0[{"prefix": "cache drop", "timeout": 10}] 2022-07-20T02:31:01.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.240+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 --> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] -- command(tid 0: {"prefix": "cache drop", "timeout": 10}) v1 -- 0x246d5e0 con 0x26aadf0 2022-07-20T02:31:01.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.244+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x26aadf0 0x26b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:01.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.244+0000 7f7f65c4e700 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x26aadf0 0x26b4390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=eeca567d90c2ffda server_cookie=ba9a32d65899421a in_seq=0 out_seq=0 2022-07-20T02:31:01.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:01.244+0000 7f7f41ffb700 10 client.6180 ms_handle_connect on v2:172.21.15.26:6836/2926574302 2022-07-20T02:31:11.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f41ffb700 1 -- 172.21.15.26:0/1069121173 <== mds.0 v2:172.21.15.26:6836/2926574302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+400 (crc 0 0 0) 0x246d5e0 con 0x26aadf0 2022-07-20T02:31:11.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f41ffb700 10 client.6180 handle_command_reply: tid=0 2022-07-20T02:31:11.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 client.6180 shutdown 2022-07-20T02:31:11.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 msgr2=0x26aa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:11.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f7f58026000 tx=0x7f7f5806c040).stop 2022-07-20T02:31:11.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 shutdown_connections 2022-07-20T02:31:11.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:6836/2926574302,v1:172.21.15.26:6837/2926574302] conn(0x26aadf0 0x26b4390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x264fa90 0x26aa900 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x261b010 0x2657740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 --2- 172.21.15.26:0/1069121173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x26176f0 0x264bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f41ffb700 0 client.6180 ms_handle_reset on v2:172.21.15.26:6836/2926574302 2022-07-20T02:31:11.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 >> 172.21.15.26:0/1069121173 conn(0x25a18f0 msgr2=0x261b510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:11.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 shutdown_connections 2022-07-20T02:31:11.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 1 -- 172.21.15.26:0/1069121173 wait complete. 2022-07-20T02:31:11.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f72109740 20 client.6180 trim_cache size 0 max 16384 2022-07-20T02:31:11.491 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:31:11.491 INFO:teuthology.orchestra.run.smithi026.stdout: "client_recall": { 2022-07-20T02:31:11.493 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": -110, 2022-07-20T02:31:11.493 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "(110) Connection timed out", 2022-07-20T02:31:11.493 INFO:teuthology.orchestra.run.smithi026.stdout: "recalled": 240 2022-07-20T02:31:11.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:11.494 INFO:teuthology.orchestra.run.smithi026.stdout: "flush_journal": { 2022-07-20T02:31:11.494 INFO:teuthology.orchestra.run.smithi026.stdout: "return_code": 0, 2022-07-20T02:31:11.495 INFO:teuthology.orchestra.run.smithi026.stdout: "message": "" 2022-07-20T02:31:11.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:11.496 INFO:teuthology.orchestra.run.smithi026.stdout: "trim_cache": { 2022-07-20T02:31:11.496 INFO:teuthology.orchestra.run.smithi026.stdout: "trimmed": 220 2022-07-20T02:31:11.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:11.497 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2022-07-20T02:31:11.497 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": { 2022-07-20T02:31:11.498 INFO:teuthology.orchestra.run.smithi026.stdout: "items": 15436, 2022-07-20T02:31:11.498 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes": 2428368 2022-07-20T02:31:11.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:11.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:11.499 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 10.231780247 2022-07-20T02:31:11.499 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:31:11.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 msgr2=0x7f7f48055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:11.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 0x7f7f48055ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f7f5c00b000 tx=0x7f7f5c006df0).stop 2022-07-20T02:31:11.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 msgr2=0x7f7f6014c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:11.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f7f5800b000 tx=0x7f7f580091d0).stop 2022-07-20T02:31:11.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 shutdown_connections 2022-07-20T02:31:11.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f600abde0 0x7f7f6014c210 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7f48053b30 0x7f7f48055ff0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f60067b00 0x7f7f600b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 --2- 172.21.15.26:0/3491596269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f600a8c40 0x7f7f601467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 >> 172.21.15.26:0/3491596269 conn(0x7f7f60060310 msgr2=0x7f7f60142e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 shutdown_connections 2022-07-20T02:31:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:11.476+0000 7f7f66c50700 1 -- 172.21.15.26:0/3491596269 wait complete. 2022-07-20T02:31:11.507 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:31:11.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:11.528 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:11.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:11.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:11.601 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:31:11.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:11.602 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:11.603 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:11.619 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:31:11.619 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:31:11.620 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:31:11.620 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:31:11.627 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:11.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22735]: fuse finished with error 0 and tester_r 0 2022-07-20T02:31:11.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:11.689 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:11.689 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:11.715 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:31:11.716 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:31:11.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:31:11.836 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:11.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:11.889 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:11.890 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:31:11.902 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:31:11.902 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:31:11.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:31:11.904 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:11.912 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:31:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:31:11.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:31:12.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.028+0000 7ff332120700 1 Processor -- start 2022-07-20T02:31:12.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.028+0000 7ff332120700 1 -- start start 2022-07-20T02:31:12.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff332120700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff32c0b0720 0x7ff32c0a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff332120700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 0x7ff32c0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff332120700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c0b1cf0 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff332120700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c0af990 con 0x7ff32c0b0720 2022-07-20T02:31:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff332120700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c0ae370 con 0x7ff32c0b15e0 2022-07-20T02:31:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff33091d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff32c0b0720 0x7ff32c0a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff33091d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff32c0b0720 0x7ff32c0a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38832/0 (socket says 172.21.15.26:38832) 2022-07-20T02:31:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff33091d700 1 -- 172.21.15.26:0/2854725199 learned_addr learned my addr 172.21.15.26:0/2854725199 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff33191f700 1 --2- 172.21.15.26:0/2854725199 >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 0x7ff32c0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.032+0000 7ff323fff700 1 -- 172.21.15.26:0/2854725199 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338248066 0 0) 0x7ff32c0b1cf0 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff323fff700 1 -- 172.21.15.26:0/2854725199 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318003240 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33091d700 1 -- 172.21.15.26:0/2854725199 >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 msgr2=0x7ff32c0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33091d700 1 --2- 172.21.15.26:0/2854725199 >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 0x7ff32c0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33091d700 1 -- 172.21.15.26:0/2854725199 >> v1:172.21.15.120:6789/0 conn(0x7ff32c0b5ef0 legacy=0x7ff32c0ad3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33091d700 1 -- 172.21.15.26:0/2854725199 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0b62d0 con 0x7ff32c0b0720 2022-07-20T02:31:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33191f700 1 --2- 172.21.15.26:0/2854725199 >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 0x7ff32c0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff33091d700 1 --2- 172.21.15.26:0/2854725199 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c0a9380 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7ff31c002f70 tx=0x7ff31c006e50).ready entity=mon.0 client_cookie=885071ce117fa6ea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff323fff700 1 -- 172.21.15.26:0/2854725199 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c006020 con 0x7ff32c0b0720 2022-07-20T02:31:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff323fff700 1 -- 172.21.15.26:0/2854725199 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c006180 con 0x7ff32c0b0720 2022-07-20T02:31:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff323fff700 1 -- 172.21.15.26:0/2854725199 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c006520 con 0x7ff32c0b0720 2022-07-20T02:31:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 -- 172.21.15.26:0/2854725199 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 msgr2=0x7ff32c0a9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 --2- 172.21.15.26:0/2854725199 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c0a9380 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7ff31c002f70 tx=0x7ff31c006e50).stop 2022-07-20T02:31:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 -- 172.21.15.26:0/2854725199 shutdown_connections 2022-07-20T02:31:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 --2- 172.21.15.26:0/2854725199 >> v2:172.21.15.120:3300/0 conn(0x7ff32c0b15e0 0x7ff32c0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 --2- 172.21.15.26:0/2854725199 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c0a9380 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 -- 172.21.15.26:0/2854725199 >> 172.21.15.26:0/2854725199 conn(0x7ff32c060310 msgr2=0x7ff32c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 -- 172.21.15.26:0/2854725199 shutdown_connections 2022-07-20T02:31:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.136+0000 7ff332120700 1 -- 172.21.15.26:0/2854725199 wait complete. 2022-07-20T02:31:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.140+0000 7ff332120700 1 Processor -- start 2022-07-20T02:31:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.140+0000 7ff332120700 1 -- start start 2022-07-20T02:31:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c067470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff32c0b15e0 0x7ff32c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 0x7ff32c15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c065810 con 0x7ff32c0b0720 2022-07-20T02:31:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c066bc0 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c065a40 con 0x7ff32c0b15e0 2022-07-20T02:31:12.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33111e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c067470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33091d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff32c0b15e0 0x7ff32c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 0x7ff32c15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33111e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c067470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38836/0 (socket says 172.21.15.26:38836) 2022-07-20T02:31:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33111e700 1 -- 172.21.15.26:0/624248373 learned_addr learned my addr 172.21.15.26:0/624248373 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 -- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff32c0b15e0 msgr2=0x7ff32c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff32c0b15e0 0x7ff32c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 -- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 msgr2=0x7ff32c067470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c067470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 -- 172.21.15.26:0/624248373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c00b1e0 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff33191f700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 0x7ff32c15f560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7ff32800c000 tx=0x7ff328006c90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff328009040 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32800c8b0 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c0a0970 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff32800cc10 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.144+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c0ab230 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c066220 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff32800cd70 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff3227fc700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 0x7ff314055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff32806c650 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff33111e700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 0x7ff314055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.148+0000 7ff33111e700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 0x7ff314055ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff32400d000 tx=0x7ff324007ea0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.160+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff328043410 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.288+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff32c065c00 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.288+0000 7ff3227fc700 1 -- 172.21.15.26:0/624248373 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v66) v1 ==== 81+0+34088 (secure 0 0 0) 0x7ff328046a30 con 0x7ff32c0b5ef0 2022-07-20T02:31:12.289 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:12.290 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:31:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 66, 2022-07-20T02:31:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:31:12.290 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:31:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:30:43.196141+0000", 2022-07-20T02:31:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:31:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:31:12.291 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:31:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:31:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:31:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:31:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:31:12.292 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:31:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:31:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:31:12.293 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:31:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:31:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:31:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:31:12.294 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 12, 2022-07-20T02:31:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:31:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:31:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:31:12.295 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:31:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:31:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:31:12.296 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:31:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:31:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:12.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:12.298 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:12.299 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:31:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:31:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:31:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:31:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:31:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:12.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:12.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:31:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:12.303 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:12.304 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:12.305 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:12.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:12.307 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:12.308 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:12.309 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:12.310 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:12.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:12.312 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:12.313 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:12.314 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:12.315 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:12.316 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:31:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:31:12.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.318 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:31:12.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:31:12.321 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:31:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:31:12.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:12.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:12.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:12.334 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:12.335 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:12.336 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:12.336 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:12.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:12.338 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:31:12.339 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:31:12.339 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:31:12.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:31:12.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:31:12.341 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:12.342 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:12.342 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:12.343 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:12.343 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:12.343 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:12.344 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:12.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.345 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:31:12.350 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:12.350 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:12.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:12.352 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:12.353 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:12.353 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:12.354 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:12.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:12.355 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:12.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:12.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:12.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:12.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:12.357 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:12.357 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:12.358 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:12.358 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:12.359 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:12.359 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:12.360 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:12.360 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:12.360 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:12.361 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:12.361 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:12.362 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:12.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:12.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.363 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:12.364 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:12.365 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:12.365 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:12.366 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:12.366 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:12.367 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:12.367 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:12.368 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:12.368 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:12.369 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:12.369 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:31:12.369 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:12.370 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:31:12.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.372 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 11, 2022-07-20T02:31:12.372 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:31:12.373 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:30:39.711494+0000", 2022-07-20T02:31:12.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:12.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:12.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:12.374 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:12.375 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:12.375 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:12.375 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:12.376 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:12.376 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:31:12.377 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:31:12.378 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:31:12.378 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:31:12.379 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:31:12.379 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:12.380 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:12.380 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:12.380 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:12.381 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:12.381 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:12.382 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:12.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.383 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "65", 2022-07-20T02:31:12.383 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:12.384 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:12.384 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:12.384 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:12.385 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:12.385 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:12.386 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:12.386 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:12.386 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:12.387 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:12.387 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:12.387 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:12.387 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:12.388 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:12.388 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:12.388 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:12.388 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:12.389 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:12.389 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:12.390 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:12.391 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:12.391 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:12.392 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:12.392 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:12.392 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:12.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:12.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.394 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:12.394 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:12.394 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:12.395 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:12.395 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:12.395 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:12.396 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:12.396 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:12.397 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:12.397 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:12.397 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:12.397 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:31:12.398 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:31:12.398 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:31:12.398 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:31:12.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.399 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:12.399 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:31:12.399 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:31:12.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.400 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 12, 2022-07-20T02:31:12.400 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:31:12.401 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:30:40.978744+0000", 2022-07-20T02:31:12.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:12.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:12.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:12.401 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:31:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:31:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:31:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:31:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:31:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "66", 2022-07-20T02:31:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:31:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:31:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:31:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:31:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:31:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:31:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:31:12.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:31:12.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:31:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:31:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:31:12.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:31:12.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.429 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:31:12.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:31:12.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:12.433 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.434 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.435 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:31:12.435 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:31:12.435 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:31:12.436 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:31:12.436 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.437 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.437 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.439 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:31:12.440 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:31:12.440 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.440 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.442 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.442 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.442 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.442 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.443 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:12.443 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.443 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:31:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:31:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.480 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:31:12.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:31:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:31:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:31:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:31:12.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:31:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:31:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:31:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:31:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:31:12.493 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:31:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:31:12.495 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.496 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.497 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.498 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.499 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:31:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:31:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:31:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:31:12.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:31:12.509 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:31:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:31:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:31:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:31:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:31:12.523 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:31:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:31:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:31:12.525 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.526 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:31:12.527 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:31:12.529 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:31:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.533 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:31:12.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:31:12.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:31:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:31:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.546 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:31:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:31:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:31:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:31:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:31:12.553 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:31:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.554 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:31:12.557 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:31:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.558 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.559 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:12.561 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.562 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:31:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:31:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:31:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:31:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:31:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:31:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:31:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:31:12.585 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:12.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.587 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:31:12.587 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:31:12.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:31:12.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:31:12.588 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.589 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.589 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.591 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:31:12.591 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:31:12.591 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.592 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.592 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.592 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.593 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.593 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.594 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.595 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:12.595 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.595 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:31:12.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:31:12.602 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.603 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:31:12.606 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:31:12.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.609 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:31:12.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:31:12.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.616 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:31:12.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:31:12.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:12.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:31:12.637 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:31:12.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:31:12.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:31:12.639 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.639 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.644 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.645 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:31:12.645 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:31:12.645 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.645 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.646 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.646 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.647 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:12.647 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:31:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:31:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:31:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:31:12.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:31:12.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:31:12.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.680 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:31:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:31:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:12.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:31:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:31:12.685 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:31:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:31:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.686 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.687 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:31:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:31:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:12.688 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:12.689 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:12.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:31:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:31:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:12.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:31:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:31:12.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:12.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:31:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:31:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:12.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:31:12.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:31:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:31:12.709 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:31:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:31:12.710 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:31:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:12.711 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:12.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:12.713 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:31:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:31:12.715 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.716 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.717 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:31:12.718 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.720 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:31:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.721 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.722 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.723 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:31:12.724 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:31:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.726 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:31:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:31:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:31:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:31:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:31:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:31:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:31:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:31:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.748 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:31:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:31:12.752 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:12.753 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:31:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.754 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:31:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:31:12.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:31:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:31:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:31:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:31:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:31:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:31:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:31:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:31:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:31:12.757 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:31:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:31:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:31:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:31:12.758 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:31:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:31:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:31:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:31:12.759 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:31:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:31:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:31:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:31:12.760 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:31:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:31:12.761 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:31:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:31:12.762 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:31:12.765 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:31:12.765 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:31:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 msgr2=0x7ff314055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 0x7ff314055ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff32400d000 tx=0x7ff324007ea0).stop 2022-07-20T02:31:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 msgr2=0x7ff32c15f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 0x7ff32c15f560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7ff32800c000 tx=0x7ff328006c90).stop 2022-07-20T02:31:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 shutdown_connections 2022-07-20T02:31:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff314053b30 0x7ff314055ff0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff32c0b5ef0 0x7ff32c15f560 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff32c0b15e0 0x7ff32c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 --2- 172.21.15.26:0/624248373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff32c0b0720 0x7ff32c067470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 >> 172.21.15.26:0/624248373 conn(0x7ff32c060310 msgr2=0x7ff32c0a99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 shutdown_connections 2022-07-20T02:31:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.292+0000 7ff332120700 1 -- 172.21.15.26:0/624248373 wait complete. 2022-07-20T02:31:12.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 Processor -- start 2022-07-20T02:31:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 -- start start 2022-07-20T02:31:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9f98144660 0x7f9f98144a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9f9805c580 0x7f9f98095e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f980b1330 con 0x7f9f980b1a30 2022-07-20T02:31:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f98145280 con 0x7f9f98144660 2022-07-20T02:31:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f9e430700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f98096300 con 0x7f9f9805c580 2022-07-20T02:31:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9f98144660 0x7f9f98144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9f98144660 0x7f9f98144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38844/0 (socket says 172.21.15.26:38844) 2022-07-20T02:31:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 -- 172.21.15.26:0/3592660866 learned_addr learned my addr 172.21.15.26:0/3592660866 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f977fe700 1 --2- 172.21.15.26:0/3592660866 >> v2:172.21.15.120:3300/0 conn(0x7f9f9805c580 0x7f9f98095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 -- 172.21.15.26:0/3592660866 >> v2:172.21.15.120:3300/0 conn(0x7f9f9805c580 msgr2=0x7f9f98095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 --2- 172.21.15.26:0/3592660866 >> v2:172.21.15.120:3300/0 conn(0x7f9f9805c580 0x7f9f98095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.888+0000 7f9f97fff700 1 -- 172.21.15.26:0/3592660866 >> v1:172.21.15.120:6789/0 conn(0x7f9f980b1a30 legacy=0x7f9f9813e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f97fff700 1 -- 172.21.15.26:0/3592660866 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f98144fe0 con 0x7f9f98144660 2022-07-20T02:31:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f97fff700 1 --2- 172.21.15.26:0/3592660866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f98144a40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f9f980b1910 tx=0x7f9f88006eb0).ready entity=mon.0 client_cookie=c696b83587d7fc28 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f96ffd700 1 -- 172.21.15.26:0/3592660866 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f8800d5c0 con 0x7f9f98144660 2022-07-20T02:31:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f96ffd700 1 -- 172.21.15.26:0/3592660866 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88000e90 con 0x7f9f98144660 2022-07-20T02:31:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f96ffd700 1 -- 172.21.15.26:0/3592660866 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f8800d910 con 0x7f9f98144660 2022-07-20T02:31:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- 172.21.15.26:0/3592660866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 msgr2=0x7f9f98144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 --2- 172.21.15.26:0/3592660866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f98144a40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f9f980b1910 tx=0x7f9f88006eb0).stop 2022-07-20T02:31:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- 172.21.15.26:0/3592660866 shutdown_connections 2022-07-20T02:31:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 --2- 172.21.15.26:0/3592660866 >> v2:172.21.15.120:3300/0 conn(0x7f9f9805c580 0x7f9f98095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 --2- 172.21.15.26:0/3592660866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f98144a40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- 172.21.15.26:0/3592660866 >> 172.21.15.26:0/3592660866 conn(0x7f9f980b5e70 msgr2=0x7f9f980b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- 172.21.15.26:0/3592660866 shutdown_connections 2022-07-20T02:31:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- 172.21.15.26:0/3592660866 wait complete. 2022-07-20T02:31:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 Processor -- start 2022-07-20T02:31:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.892+0000 7f9f9e430700 1 -- start start 2022-07-20T02:31:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9805c580 0x7f9f9809a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f980b1a30 0x7f9f980ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f98099e30 con 0x7f9f98144660 2022-07-20T02:31:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980aa970 con 0x7f9f9805c580 2022-07-20T02:31:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980aac10 con 0x7f9f980b1a30 2022-07-20T02:31:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f977fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f980b1a30 0x7f9f980ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38854/0 (socket says 172.21.15.26:38854) 2022-07-20T02:31:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 -- 172.21.15.26:0/2176164746 learned_addr learned my addr 172.21.15.26:0/2176164746 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f97fff700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9805c580 0x7f9f9809a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 -- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f980b1a30 msgr2=0x7f9f980ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f980b1a30 0x7f9f980ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 -- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9805c580 msgr2=0x7f9f9809a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9805c580 0x7f9f9809a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 -- 172.21.15.26:0/2176164746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8800b1e0 con 0x7f9f98144660 2022-07-20T02:31:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9c9cd700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f9f90002320 tx=0x7f9f900023f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f90006aa0 con 0x7f9f98144660 2022-07-20T02:31:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f90006c00 con 0x7f9f98144660 2022-07-20T02:31:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f98099b20 con 0x7f9f98144660 2022-07-20T02:31:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f90007030 con 0x7f9f98144660 2022-07-20T02:31:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f98095380 con 0x7f9f98144660 2022-07-20T02:31:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.896+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f980aadd0 con 0x7f9f98144660 2022-07-20T02:31:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.900+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9f90010030 con 0x7f9f98144660 2022-07-20T02:31:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.900+0000 7f9f957fa700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 0x7f9f78055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.900+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9f9006bfb0 con 0x7f9f98144660 2022-07-20T02:31:12.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.900+0000 7f9f97fff700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 0x7f9f78055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:12.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.900+0000 7f9f97fff700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 0x7f9f78055ed0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9f8801c000 tx=0x7f9f88015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:12.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:12.908+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f90042c60 con 0x7f9f98144660 2022-07-20T02:31:13.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f980b1590 con 0x7f9f98144660 2022-07-20T02:31:13.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f957fa700 1 -- 172.21.15.26:0/2176164746 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f9f90046280 con 0x7f9f98144660 2022-07-20T02:31:13.071 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:13.071 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:30:43.197173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5997":{"gid":5997,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2926574302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2926574302},{"type":"v1","addr":"172.21.15.26:6837","nonce":2926574302}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:31:13.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 msgr2=0x7f9f78055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 0x7f9f78055ed0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9f8801c000 tx=0x7f9f88015000).stop 2022-07-20T02:31:13.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 msgr2=0x7f9f980ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f9f90002320 tx=0x7f9f900023f0).stop 2022-07-20T02:31:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 shutdown_connections 2022-07-20T02:31:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9f98144660 0x7f9f980ab5a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9f78053a10 0x7f9f78055ed0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f980b1a30 0x7f9f980ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 --2- 172.21.15.26:0/2176164746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9805c580 0x7f9f9809a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 >> 172.21.15.26:0/2176164746 conn(0x7f9f980b5e70 msgr2=0x7f9f980b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 shutdown_connections 2022-07-20T02:31:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.072+0000 7f9f9e430700 1 -- 172.21.15.26:0/2176164746 wait complete. 2022-07-20T02:31:13.077 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2022-07-20T02:31:13.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:31:13.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 Processor -- start 2022-07-20T02:31:13.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 -- start start 2022-07-20T02:31:13.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a900b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 0x7f3a900b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a900b11f0 con 0x7f3a90060a90 2022-07-20T02:31:13.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a90061010 con 0x7f3a900affe0 2022-07-20T02:31:13.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a980ae700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90144970 con 0x7f3a900b27f0 2022-07-20T02:31:13.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a95649700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3a90060a90 0x7f3a901450b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38138/0 (socket says 172.21.15.26:38138) 2022-07-20T02:31:13.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a95649700 1 -- 172.21.15.26:0/3407599350 learned_addr learned my addr 172.21.15.26:0/3407599350 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:13.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a9664b700 1 --2- 172.21.15.26:0/3407599350 >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 0x7f3a900b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.196+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3407599350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a900b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a94e48700 1 -- 172.21.15.26:0/3407599350 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014455347 0 0) 0x7f3a900b11f0 con 0x7f3a90060a90 2022-07-20T02:31:13.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a94e48700 1 -- 172.21.15.26:0/3407599350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a78003260 con 0x7f3a90060a90 2022-07-20T02:31:13.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3407599350 >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 msgr2=0x7f3a900b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:13.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3407599350 >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 0x7f3a900b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3407599350 >> v1:172.21.15.26:6789/0 conn(0x7f3a90060a90 legacy=0x7f3a901450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3407599350 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90144c10 con 0x7f3a900b27f0 2022-07-20T02:31:13.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a9664b700 1 --2- 172.21.15.26:0/3407599350 >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 0x7f3a900b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:13.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3407599350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a900b2bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3a8c002f70 tx=0x7f3a8c006df0).ready entity=mon.2 client_cookie=b666a67940df13e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a94e48700 1 -- 172.21.15.26:0/3407599350 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a8c00d4a0 con 0x7f3a900b27f0 2022-07-20T02:31:13.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a94e48700 1 -- 172.21.15.26:0/3407599350 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c000e90 con 0x7f3a900b27f0 2022-07-20T02:31:13.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a94e48700 1 -- 172.21.15.26:0/3407599350 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a8c00d7f0 con 0x7f3a900b27f0 2022-07-20T02:31:13.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- 172.21.15.26:0/3407599350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 msgr2=0x7f3a900b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3407599350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a900b2bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3a8c002f70 tx=0x7f3a8c006df0).stop 2022-07-20T02:31:13.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- 172.21.15.26:0/3407599350 shutdown_connections 2022-07-20T02:31:13.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3407599350 >> v2:172.21.15.26:3300/0 conn(0x7f3a900affe0 0x7f3a900b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3407599350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a900b2bd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- 172.21.15.26:0/3407599350 >> 172.21.15.26:0/3407599350 conn(0x7f3a900b6110 msgr2=0x7f3a900b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- 172.21.15.26:0/3407599350 shutdown_connections 2022-07-20T02:31:13.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- 172.21.15.26:0/3407599350 wait complete. 2022-07-20T02:31:13.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 Processor -- start 2022-07-20T02:31:13.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.200+0000 7f3a980ae700 1 -- start start 2022-07-20T02:31:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 0x7f3a9013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a90140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900635a0 con 0x7f3a90060a90 2022-07-20T02:31:13.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90140650 con 0x7f3a900affe0 2022-07-20T02:31:13.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900653d0 con 0x7f3a900b27f0 2022-07-20T02:31:13.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38864/0 (socket says 172.21.15.26:38864) 2022-07-20T02:31:13.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3080041242 learned_addr learned my addr 172.21.15.26:0/3080041242 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:13.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a9664b700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a90140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95649700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 0x7f3a9013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 msgr2=0x7f3a90140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a90140160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 msgr2=0x7f3a9013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 0x7f3a9013fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 -- 172.21.15.26:0/3080041242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a88009470 con 0x7f3a90060a90 2022-07-20T02:31:13.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95649700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 0x7f3a9013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:13.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a95e4a700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f3a900636e0 tx=0x7f3a8c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a8c016070 con 0x7f3a90060a90 2022-07-20T02:31:13.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c013800 con 0x7f3a90060a90 2022-07-20T02:31:13.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c00b180 con 0x7f3a90060a90 2022-07-20T02:31:13.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a8c013b60 con 0x7f3a90060a90 2022-07-20T02:31:13.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a90062e60 con 0x7f3a90060a90 2022-07-20T02:31:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.204+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a900ae5d0 con 0x7f3a90060a90 2022-07-20T02:31:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.208+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3a8c024030 con 0x7f3a90060a90 2022-07-20T02:31:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.208+0000 7f3a7effd700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 0x7f3a70055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.208+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3a8c06ca70 con 0x7f3a90060a90 2022-07-20T02:31:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.208+0000 7f3a95649700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 0x7f3a70055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.208+0000 7f3a95649700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 0x7f3a70055e10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f3a8000b000 tx=0x7f3a80009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.216+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a8c0438e0 con 0x7f3a90060a90 2022-07-20T02:31:13.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.380+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3a900629b0 con 0x7f3a90060a90 2022-07-20T02:31:13.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:31:13.400+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:31:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a7effd700 1 -- 172.21.15.26:0/3080041242 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3a8c06ce60 con 0x7f3a90060a90 2022-07-20T02:31:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 msgr2=0x7f3a70055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 0x7f3a70055e10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f3a8000b000 tx=0x7f3a80009040).stop 2022-07-20T02:31:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 msgr2=0x7f3a90145ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f3a900636e0 tx=0x7f3a8c014000).stop 2022-07-20T02:31:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 shutdown_connections 2022-07-20T02:31:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3a70053950 0x7f3a70055e10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a900b27f0 0x7f3a90140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a900affe0 0x7f3a9013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 --2- 172.21.15.26:0/3080041242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a90060a90 0x7f3a90145ec0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 >> 172.21.15.26:0/3080041242 conn(0x7f3a900b6110 msgr2=0x7f3a900af710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 shutdown_connections 2022-07-20T02:31:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.408+0000 7f3a980ae700 1 -- 172.21.15.26:0/3080041242 wait complete. 2022-07-20T02:31:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:31:13.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:13.446 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:31:13.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 Processor -- start 2022-07-20T02:31:13.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 -- start start 2022-07-20T02:31:13.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff1c8060950 0x7ff1c8096220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c8067dc0 con 0x7ff1c80b3260 2022-07-20T02:31:13.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c80b3640 con 0x7ff1c80a95a0 2022-07-20T02:31:13.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c80615c0 con 0x7ff1c8060950 2022-07-20T02:31:13.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff1c80b3260 0x7ff1c80676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38164/0 (socket says 172.21.15.26:38164) 2022-07-20T02:31:13.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c659c700 1 -- 172.21.15.26:0/958708671 learned_addr learned my addr 172.21.15.26:0/958708671 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:13.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c6d9d700 1 --2- 172.21.15.26:0/958708671 >> v2:172.21.15.26:3300/0 conn(0x7ff1c8060950 0x7ff1c8096220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/958708671 >> v1:172.21.15.120:6789/0 conn(0x7ff1c80a95a0 legacy=0x7ff1c8095b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/958708671 >> v1:172.21.15.26:6789/0 conn(0x7ff1c80b3260 legacy=0x7ff1c80676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c559a700 1 -- 172.21.15.26:0/958708671 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737799845 0 0) 0x7ff1c8067dc0 con 0x7ff1c80b3260 2022-07-20T02:31:13.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/958708671 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8061860 con 0x7ff1c8060950 2022-07-20T02:31:13.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.536+0000 7ff1c6d9d700 1 --2- 172.21.15.26:0/958708671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c8060950 0x7ff1c8096220 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7ff1c00032f0 tx=0x7ff1c00091d0).ready entity=mon.0 client_cookie=2b9ea3284959519a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c559a700 1 -- 172.21.15.26:0/958708671 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c0009ea0 con 0x7ff1c8060950 2022-07-20T02:31:13.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c559a700 1 -- 172.21.15.26:0/958708671 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c000b8a0 con 0x7ff1c8060950 2022-07-20T02:31:13.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c559a700 1 -- 172.21.15.26:0/958708671 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c000ba10 con 0x7ff1c8060950 2022-07-20T02:31:13.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- 172.21.15.26:0/958708671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c8060950 msgr2=0x7ff1c8096220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 --2- 172.21.15.26:0/958708671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c8060950 0x7ff1c8096220 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7ff1c00032f0 tx=0x7ff1c00091d0).stop 2022-07-20T02:31:13.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- 172.21.15.26:0/958708671 shutdown_connections 2022-07-20T02:31:13.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 --2- 172.21.15.26:0/958708671 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c8060950 0x7ff1c8096220 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- 172.21.15.26:0/958708671 >> 172.21.15.26:0/958708671 conn(0x7ff1c80b67f0 msgr2=0x7ff1c80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- 172.21.15.26:0/958708671 shutdown_connections 2022-07-20T02:31:13.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- 172.21.15.26:0/958708671 wait complete. 2022-07-20T02:31:13.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 Processor -- start 2022-07-20T02:31:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.540+0000 7ff1c759e700 1 -- start start 2022-07-20T02:31:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c8060950 0x7ff1c80ab410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c80a95a0 0x7ff1c80af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8095d50 con 0x7ff1c80b3260 2022-07-20T02:31:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8095ff0 con 0x7ff1c8060950 2022-07-20T02:31:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8096290 con 0x7ff1c80a95a0 2022-07-20T02:31:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38892/0 (socket says 172.21.15.26:38892) 2022-07-20T02:31:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/2046315386 learned_addr learned my addr 172.21.15.26:0/2046315386 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c659c700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c8060950 0x7ff1c80ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c5d9b700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c80a95a0 0x7ff1c80af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c80a95a0 msgr2=0x7ff1c80af6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c80a95a0 0x7ff1c80af6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c8060950 msgr2=0x7ff1c80ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c8060950 0x7ff1c80ab410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 -- 172.21.15.26:0/2046315386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c0009470 con 0x7ff1c80b3260 2022-07-20T02:31:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c6d9d700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7ff1c00032f0 tx=0x7ff1c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c0009ea0 con 0x7ff1c80b3260 2022-07-20T02:31:13.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c001f3f0 con 0x7ff1c80b3260 2022-07-20T02:31:13.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c8096530 con 0x7ff1c80b3260 2022-07-20T02:31:13.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c001f750 con 0x7ff1c80b3260 2022-07-20T02:31:13.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c80b5870 con 0x7ff1c80b3260 2022-07-20T02:31:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.544+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c8152720 con 0x7ff1c80b3260 2022-07-20T02:31:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.548+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff1c0026030 con 0x7ff1c80b3260 2022-07-20T02:31:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.548+0000 7ff1b77fe700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 0x7ff1a4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.548+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6506+0+0 (secure 0 0 0) 0x7ff1c006cbf0 con 0x7ff1c80b3260 2022-07-20T02:31:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.548+0000 7ff1c659c700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 0x7ff1a4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.548+0000 7ff1c659c700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 0x7ff1a4055fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7ff1b800d000 tx=0x7ff1b8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.552+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1c0043960 con 0x7ff1c80b3260 2022-07-20T02:31:13.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.716+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1c8061b00 con 0x7ff1c80b3260 2022-07-20T02:31:13.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1b77fe700 1 -- 172.21.15.26:0/2046315386 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2132 (secure 0 0 0) 0x7ff1c0046f80 con 0x7ff1c80b3260 2022-07-20T02:31:13.720 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:13.720 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:30:42.193367+0000","modified":"2022-07-20T02:31:13.402323+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T02:31:13.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 msgr2=0x7ff1a4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 0x7ff1a4055fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7ff1b800d000 tx=0x7ff1b8009040).stop 2022-07-20T02:31:13.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 msgr2=0x7ff1c81429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7ff1c00032f0 tx=0x7ff1c0016000).stop 2022-07-20T02:31:13.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 shutdown_connections 2022-07-20T02:31:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1c80b3260 0x7ff1c81429f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1a4053b00 0x7ff1a4055fc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1c80a95a0 0x7ff1c80af6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 --2- 172.21.15.26:0/2046315386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1c8060950 0x7ff1c80ab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 >> 172.21.15.26:0/2046315386 conn(0x7ff1c80b67f0 msgr2=0x7ff1c80b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 shutdown_connections 2022-07-20T02:31:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.720+0000 7ff1c759e700 1 -- 172.21.15.26:0/2046315386 wait complete. 2022-07-20T02:31:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 31 2022-07-20T02:31:13.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:31:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 Processor -- start 2022-07-20T02:31:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 -- start start 2022-07-20T02:31:13.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3360147000 0x7f3360143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33601474e0 con 0x7f336005cfd0 2022-07-20T02:31:13.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600b4e40 con 0x7f3360060f00 2022-07-20T02:31:13.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f3365528700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336005d3b0 con 0x7f3360147000 2022-07-20T02:31:13.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f336005cfd0 0x7f33600b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38178/0 (socket says 172.21.15.26:38178) 2022-07-20T02:31:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335ffff700 1 -- 172.21.15.26:0/270379925 learned_addr learned my addr 172.21.15.26:0/270379925 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335f7fe700 1 --2- 172.21.15.26:0/270379925 >> v2:172.21.15.120:3300/0 conn(0x7f3360147000 0x7f3360143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982692238 0 0) 0x7f33601474e0 con 0x7f336005cfd0 2022-07-20T02:31:13.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c003260 con 0x7f336005cfd0 2022-07-20T02:31:13.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232033863 0 0) 0x7f33600b4e40 con 0x7f3360060f00 2022-07-20T02:31:13.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335f7fe700 1 -- 172.21.15.26:0/270379925 >> v1:172.21.15.120:6789/0 conn(0x7f3360060f00 legacy=0x7f336013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335f7fe700 1 -- 172.21.15.26:0/270379925 >> v1:172.21.15.26:6789/0 conn(0x7f336005cfd0 legacy=0x7f33600b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335f7fe700 1 -- 172.21.15.26:0/270379925 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33601431e0 con 0x7f3360147000 2022-07-20T02:31:13.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.844+0000 7f335f7fe700 1 --2- 172.21.15.26:0/270379925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360143bc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f3350002f70 tx=0x7f3350006df0).ready entity=mon.1 client_cookie=d56bb9c35136286b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f335000d610 con 0x7f3360147000 2022-07-20T02:31:13.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3350000e90 con 0x7f3360147000 2022-07-20T02:31:13.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f335effd700 1 -- 172.21.15.26:0/270379925 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f335000d960 con 0x7f3360147000 2022-07-20T02:31:13.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- 172.21.15.26:0/270379925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 msgr2=0x7f3360143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 --2- 172.21.15.26:0/270379925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360143bc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f3350002f70 tx=0x7f3350006df0).stop 2022-07-20T02:31:13.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- 172.21.15.26:0/270379925 shutdown_connections 2022-07-20T02:31:13.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 --2- 172.21.15.26:0/270379925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360143bc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- 172.21.15.26:0/270379925 >> 172.21.15.26:0/270379925 conn(0x7f33600620c0 msgr2=0x7f336005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:13.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- 172.21.15.26:0/270379925 shutdown_connections 2022-07-20T02:31:13.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- 172.21.15.26:0/270379925 wait complete. 2022-07-20T02:31:13.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 Processor -- start 2022-07-20T02:31:13.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- start start 2022-07-20T02:31:13.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3360145480 0x7f33601520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600af460 con 0x7f3360060f00 2022-07-20T02:31:13.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3360144230 con 0x7f3360147000 2022-07-20T02:31:13.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f3365528700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600b3690 con 0x7f3360145480 2022-07-20T02:31:13.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.848+0000 7f335ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38906/0 (socket says 172.21.15.26:38906) 2022-07-20T02:31:13.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f3364d27700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3360145480 0x7f33601520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58596/0 (socket says 172.21.15.26:58596) 2022-07-20T02:31:13.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 -- 172.21.15.26:0/2212172988 learned_addr learned my addr 172.21.15.26:0/2212172988 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:13.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 -- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3360145480 msgr2=0x7f33601520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3360145480 0x7f33601520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 -- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 msgr2=0x7f3360144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:13.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:13.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 -- 172.21.15.26:0/2212172988 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335000b180 con 0x7f3360060f00 2022-07-20T02:31:13.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335f7fe700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:13.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335ffff700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f33600b38a0 tx=0x7f3354007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f335400cdc0 con 0x7f3360060f00 2022-07-20T02:31:13.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33540070d0 con 0x7f3360060f00 2022-07-20T02:31:13.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3360145a00 con 0x7f3360060f00 2022-07-20T02:31:13.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3354007430 con 0x7f3360060f00 2022-07-20T02:31:13.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3360144700 con 0x7f3360060f00 2022-07-20T02:31:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3360143480 con 0x7f3360060f00 2022-07-20T02:31:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.852+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f335401a070 con 0x7f3360060f00 2022-07-20T02:31:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.856+0000 7f335d7fa700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 0x7f3340055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:13.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.856+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f335406b380 con 0x7f3360060f00 2022-07-20T02:31:13.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.856+0000 7f335f7fe700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 0x7f3340055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:13.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.856+0000 7f335f7fe700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 0x7f3340055fc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3350018000 tx=0x7f3350014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:13.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:13.860+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3354041f60 con 0x7f3360060f00 2022-07-20T02:31:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.024+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f33600574e0 con 0x7f3360060f00 2022-07-20T02:31:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.424+0000 7f335d7fa700 1 -- 172.21.15.26:0/2212172988 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3354045580 con 0x7f3360060f00 2022-07-20T02:31:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 msgr2=0x7f3340055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 0x7f3340055fc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3350018000 tx=0x7f3350014000).stop 2022-07-20T02:31:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 msgr2=0x7f33600affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f33600b38a0 tx=0x7f3354007aa0).stop 2022-07-20T02:31:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 shutdown_connections 2022-07-20T02:31:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3340053b00 0x7f3340055fc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3360145480 0x7f33601520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3360147000 0x7f3360144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 --2- 172.21.15.26:0/2212172988 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3360060f00 0x7f33600affe0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 >> 172.21.15.26:0/2212172988 conn(0x7f336005cfd0 msgr2=0x7f336005c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 shutdown_connections 2022-07-20T02:31:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.428+0000 7f3365528700 1 -- 172.21.15.26:0/2212172988 wait complete. 2022-07-20T02:31:14.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:31:14.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.560+0000 7f08b8e9d700 1 Processor -- start 2022-07-20T02:31:14.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.560+0000 7f08b8e9d700 1 -- start start 2022-07-20T02:31:14.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b4143090 0x7f08b4098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08b40b0810 0x7f08b40950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 0x7f08b40955d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b4143470 con 0x7f08b40b0810 2022-07-20T02:31:14.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b405fd90 con 0x7f08b40b0c50 2022-07-20T02:31:14.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4097d50 con 0x7f08b4143090 2022-07-20T02:31:14.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08b40b0810 0x7f08b40950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f08b40b0810 0x7f08b40950e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38916/0 (socket says 172.21.15.26:38916) 2022-07-20T02:31:14.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 -- 172.21.15.26:0/3637799722 learned_addr learned my addr 172.21.15.26:0/3637799722 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b2d9d700 1 --2- 172.21.15.26:0/3637799722 >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 0x7f08b40955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b259c700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b4143090 0x7f08b4098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 -- 172.21.15.26:0/3637799722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b4143090 msgr2=0x7f08b4098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:14.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b4143090 0x7f08b4098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 -- 172.21.15.26:0/3637799722 >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 msgr2=0x7f08b40955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:14.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/3637799722 >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 0x7f08b40955d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 -- 172.21.15.26:0/3637799722 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4097ff0 con 0x7f08b40b0810 2022-07-20T02:31:14.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b2d9d700 1 --2- 172.21.15.26:0/3637799722 >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 0x7f08b40955d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:14.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0810 0x7f08b40950e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f08b4098c00 tx=0x7f08a4009290).ready entity=mon.0 client_cookie=576cd125747c059d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b159a700 1 -- 172.21.15.26:0/3637799722 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a400b860 con 0x7f08b40b0810 2022-07-20T02:31:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b159a700 1 -- 172.21.15.26:0/3637799722 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400b9c0 con 0x7f08b40b0810 2022-07-20T02:31:14.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b159a700 1 -- 172.21.15.26:0/3637799722 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a400bb30 con 0x7f08b40b0810 2022-07-20T02:31:14.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- 172.21.15.26:0/3637799722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0810 msgr2=0x7f08b40950e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:14.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0810 0x7f08b40950e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f08b4098c00 tx=0x7f08a4009290).stop 2022-07-20T02:31:14.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- 172.21.15.26:0/3637799722 shutdown_connections 2022-07-20T02:31:14.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/3637799722 >> v2:172.21.15.120:3300/0 conn(0x7f08b40b0c50 0x7f08b40955d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0810 0x7f08b40950e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/3637799722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b4143090 0x7f08b4098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- 172.21.15.26:0/3637799722 >> 172.21.15.26:0/3637799722 conn(0x7f08b40b5e70 msgr2=0x7f08b40b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:14.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- 172.21.15.26:0/3637799722 shutdown_connections 2022-07-20T02:31:14.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.564+0000 7f08b8e9d700 1 -- 172.21.15.26:0/3637799722 wait complete. 2022-07-20T02:31:14.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 Processor -- start 2022-07-20T02:31:14.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- start start 2022-07-20T02:31:14.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b40b0810 0x7f08b409ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08b4143090 0x7f08b4152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409b680 con 0x7f08b40b0c50 2022-07-20T02:31:14.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409a550 con 0x7f08b4143090 2022-07-20T02:31:14.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409b8b0 con 0x7f08b40b0810 2022-07-20T02:31:14.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38920/0 (socket says 172.21.15.26:38920) 2022-07-20T02:31:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b40b0810 0x7f08b409ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b2d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08b4143090 0x7f08b4152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 -- 172.21.15.26:0/1623489774 learned_addr learned my addr 172.21.15.26:0/1623489774 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:14.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 -- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b40b0810 msgr2=0x7f08b409ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:14.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b40b0810 0x7f08b409ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 -- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08b4143090 msgr2=0x7f08b4152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:14.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08b4143090 0x7f08b4152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 -- 172.21.15.26:0/1623489774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a4009530 con 0x7f08b40b0c50 2022-07-20T02:31:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b1d9b700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f08b4098c00 tx=0x7f08a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:14.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a400b860 con 0x7f08b40b0c50 2022-07-20T02:31:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a40032f0 con 0x7f08b40b0c50 2022-07-20T02:31:14.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b409b140 con 0x7f08b40b0c50 2022-07-20T02:31:14.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a4020660 con 0x7f08b40b0c50 2022-07-20T02:31:14.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.568+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b4152a10 con 0x7f08b40b0c50 2022-07-20T02:31:14.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4143a00 con 0x7f08b40b0c50 2022-07-20T02:31:14.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f08a400f070 con 0x7f08b40b0c50 2022-07-20T02:31:14.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08ab7fe700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 0x7f0898055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:14.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f08a406cce0 con 0x7f08b40b0c50 2022-07-20T02:31:14.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08b259c700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 0x7f0898055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:14.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.572+0000 7f08b259c700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 0x7f0898055fc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f089c00c000 tx=0x7f089c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:14.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.576+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08a4043a50 con 0x7f08b40b0c50 2022-07-20T02:31:14.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:14.708+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f08b40574e0 con 0x7f08b40b0c50 2022-07-20T02:31:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.436+0000 7f08ab7fe700 1 -- 172.21.15.26:0/1623489774 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 185+0+0 (secure 0 0 0) 0x7f08a4015030 con 0x7f08b40b0c50 2022-07-20T02:31:15.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 msgr2=0x7f0898055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 0x7f0898055fc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f089c00c000 tx=0x7f089c008040).stop 2022-07-20T02:31:15.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 msgr2=0x7f08b4063fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f08b4098c00 tx=0x7f08a4016000).stop 2022-07-20T02:31:15.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 shutdown_connections 2022-07-20T02:31:15.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f08b40b0c50 0x7f08b4063fb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0898053b00 0x7f0898055fc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08b4143090 0x7f08b4152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 --2- 172.21.15.26:0/1623489774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08b40b0810 0x7f08b409ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 >> 172.21.15.26:0/1623489774 conn(0x7f08b40b5e70 msgr2=0x7f08b4060720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:15.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 shutdown_connections 2022-07-20T02:31:15.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.440+0000 7f08b8e9d700 1 -- 172.21.15.26:0/1623489774 wait complete. 2022-07-20T02:31:15.448 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:31:15.480 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:31:15.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 Processor -- start 2022-07-20T02:31:15.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 -- start start 2022-07-20T02:31:15.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 0x7fd1741426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:15.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740b62d0 con 0x7fd1740b5ef0 2022-07-20T02:31:15.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740a9cf0 con 0x7fd1740b0a90 2022-07-20T02:31:15.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd17aaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740a9f90 con 0x7fd1740a97e0 2022-07-20T02:31:15.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd179aa5700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd1740b5ef0 0x7fd1740b0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38206/0 (socket says 172.21.15.26:38206) 2022-07-20T02:31:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.604+0000 7fd179aa5700 1 -- 172.21.15.26:0/2320508412 learned_addr learned my addr 172.21.15.26:0/2320508412 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd1792a4700 1 --2- 172.21.15.26:0/2320508412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 0x7fd1741426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd178aa3700 1 -- 172.21.15.26:0/2320508412 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503272456 0 0) 0x7fd1740b62d0 con 0x7fd1740b5ef0 2022-07-20T02:31:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd178aa3700 1 -- 172.21.15.26:0/2320508412 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c003260 con 0x7fd1740b5ef0 2022-07-20T02:31:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd1792a4700 1 -- 172.21.15.26:0/2320508412 >> v1:172.21.15.120:6789/0 conn(0x7fd1740b0a90 legacy=0x7fd174142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd1792a4700 1 -- 172.21.15.26:0/2320508412 >> v1:172.21.15.26:6789/0 conn(0x7fd1740b5ef0 legacy=0x7fd1740b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd1792a4700 1 -- 172.21.15.26:0/2320508412 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740aa230 con 0x7fd1740a97e0 2022-07-20T02:31:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd1792a4700 1 --2- 172.21.15.26:0/2320508412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 0x7fd1741426a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd170002f70 tx=0x7fd170006df0).ready entity=mon.2 client_cookie=679f6dff89613ce4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:15.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd178aa3700 1 -- 172.21.15.26:0/2320508412 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17000d570 con 0x7fd1740a97e0 2022-07-20T02:31:15.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd178aa3700 1 -- 172.21.15.26:0/2320508412 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170000e90 con 0x7fd1740a97e0 2022-07-20T02:31:15.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd178aa3700 1 -- 172.21.15.26:0/2320508412 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17000d8c0 con 0x7fd1740a97e0 2022-07-20T02:31:15.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- 172.21.15.26:0/2320508412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 msgr2=0x7fd1741426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/2320508412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 0x7fd1741426a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd170002f70 tx=0x7fd170006df0).stop 2022-07-20T02:31:15.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- 172.21.15.26:0/2320508412 shutdown_connections 2022-07-20T02:31:15.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/2320508412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740a97e0 0x7fd1741426a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- 172.21.15.26:0/2320508412 >> 172.21.15.26:0/2320508412 conn(0x7fd174060310 msgr2=0x7fd1740b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- 172.21.15.26:0/2320508412 shutdown_connections 2022-07-20T02:31:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- 172.21.15.26:0/2320508412 wait complete. 2022-07-20T02:31:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 Processor -- start 2022-07-20T02:31:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.608+0000 7fd17aaa7700 1 -- start start 2022-07-20T02:31:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 0x7fd174153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740b5ef0 0x7fd17414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174153550 con 0x7fd1740b0a90 2022-07-20T02:31:15.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740a3d00 con 0x7fd1740a97e0 2022-07-20T02:31:15.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740a3fa0 con 0x7fd1740b5ef0 2022-07-20T02:31:15.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:15.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38934/0 (socket says 172.21.15.26:38934) 2022-07-20T02:31:15.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 -- 172.21.15.26:0/1561371791 learned_addr learned my addr 172.21.15.26:0/1561371791 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17a2a6700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740b5ef0 0x7fd17414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd179aa5700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 0x7fd174153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 -- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740b5ef0 msgr2=0x7fd17414f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740b5ef0 0x7fd17414f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 -- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 msgr2=0x7fd174153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 0x7fd174153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 -- 172.21.15.26:0/1561371791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17000b180 con 0x7fd1740b0a90 2022-07-20T02:31:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd179aa5700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 0x7fd174153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd1792a4700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7fd170002f70 tx=0x7fd170014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd170016070 con 0x7fd1740b0a90 2022-07-20T02:31:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170013910 con 0x7fd1740b0a90 2022-07-20T02:31:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd170013c70 con 0x7fd1740b0a90 2022-07-20T02:31:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1740a4240 con 0x7fd1740b0a90 2022-07-20T02:31:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1740b2560 con 0x7fd1740b0a90 2022-07-20T02:31:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1740afee0 con 0x7fd1740b0a90 2022-07-20T02:31:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.612+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd17000f030 con 0x7fd1740b0a90 2022-07-20T02:31:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.616+0000 7fd162ffd700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 0x7fd154055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.616+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fd17006cbd0 con 0x7fd1740b0a90 2022-07-20T02:31:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.616+0000 7fd179aa5700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 0x7fd154055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.616+0000 7fd179aa5700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 0x7fd154055fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd16400b000 tx=0x7fd164009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.620+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd170043a50 con 0x7fd1740b0a90 2022-07-20T02:31:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:15.744+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1740574e0 con 0x7fd1740b0a90 2022-07-20T02:31:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.440+0000 7fd162ffd700 1 -- 172.21.15.26:0/1561371791 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd170023070 con 0x7fd1740b0a90 2022-07-20T02:31:16.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 msgr2=0x7fd154055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 0x7fd154055fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd16400b000 tx=0x7fd164009040).stop 2022-07-20T02:31:16.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 msgr2=0x7fd1740b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7fd170002f70 tx=0x7fd170014000).stop 2022-07-20T02:31:16.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 shutdown_connections 2022-07-20T02:31:16.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1740b0a90 0x7fd1740b2070 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd154053b00 0x7fd154055fc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1740b5ef0 0x7fd17414f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 --2- 172.21.15.26:0/1561371791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1740a97e0 0x7fd174153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 >> 172.21.15.26:0/1561371791 conn(0x7fd174060310 msgr2=0x7fd1740ab390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:16.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 shutdown_connections 2022-07-20T02:31:16.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.448+0000 7fd17aaa7700 1 -- 172.21.15.26:0/1561371791 wait complete. 2022-07-20T02:31:16.461 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:31:16.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:31:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 Processor -- start 2022-07-20T02:31:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 -- start start 2022-07-20T02:31:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 0x7ff424066da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42405f840 con 0x7ff42405d050 2022-07-20T02:31:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4240af6b0 con 0x7ff4240aebc0 2022-07-20T02:31:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff42b869700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff424147720 con 0x7ff424066950 2022-07-20T02:31:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff428e04700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff42405d050 0x7ff4240af8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38222/0 (socket says 172.21.15.26:38222) 2022-07-20T02:31:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff428e04700 1 -- 172.21.15.26:0/766489620 learned_addr learned my addr 172.21.15.26:0/766489620 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff429e06700 1 --2- 172.21.15.26:0/766489620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 0x7ff424066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609938079 0 0) 0x7ff42405f840 con 0x7ff42405d050 2022-07-20T02:31:16.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff408003260 con 0x7ff42405d050 2022-07-20T02:31:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477317557 0 0) 0x7ff4240af6b0 con 0x7ff4240aebc0 2022-07-20T02:31:16.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42405f840 con 0x7ff4240aebc0 2022-07-20T02:31:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff429e06700 1 -- 172.21.15.26:0/766489620 >> v1:172.21.15.120:6789/0 conn(0x7ff4240aebc0 legacy=0x7ff4240aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff429e06700 1 -- 172.21.15.26:0/766489620 >> v1:172.21.15.26:6789/0 conn(0x7ff42405d050 legacy=0x7ff4240af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.580+0000 7ff429e06700 1 -- 172.21.15.26:0/766489620 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42405fa70 con 0x7ff424066950 2022-07-20T02:31:16.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff429e06700 1 --2- 172.21.15.26:0/766489620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 0x7ff424066da0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c006df0).ready entity=mon.2 client_cookie=76b98558c0ded8c1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c00d5c0 con 0x7ff424066950 2022-07-20T02:31:16.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c000e90 con 0x7ff424066950 2022-07-20T02:31:16.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff40ffff700 1 -- 172.21.15.26:0/766489620 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c00d910 con 0x7ff424066950 2022-07-20T02:31:16.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- 172.21.15.26:0/766489620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 msgr2=0x7ff424066da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 --2- 172.21.15.26:0/766489620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 0x7ff424066da0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c006df0).stop 2022-07-20T02:31:16.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- 172.21.15.26:0/766489620 shutdown_connections 2022-07-20T02:31:16.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 --2- 172.21.15.26:0/766489620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff424066950 0x7ff424066da0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- 172.21.15.26:0/766489620 >> 172.21.15.26:0/766489620 conn(0x7ff424060950 msgr2=0x7ff424060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:16.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- 172.21.15.26:0/766489620 shutdown_connections 2022-07-20T02:31:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- 172.21.15.26:0/766489620 wait complete. 2022-07-20T02:31:16.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 Processor -- start 2022-07-20T02:31:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- start start 2022-07-20T02:31:16.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff424066950 0x7ff4240a40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4240aebc0 0x7ff424155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240aaea0 con 0x7ff42405d050 2022-07-20T02:31:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240ab110 con 0x7ff424066950 2022-07-20T02:31:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff42b869700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240aac00 con 0x7ff4240aebc0 2022-07-20T02:31:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff429605700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff429605700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38946/0 (socket says 172.21.15.26:38946) 2022-07-20T02:31:16.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff429605700 1 -- 172.21.15.26:0/1504900870 learned_addr learned my addr 172.21.15.26:0/1504900870 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:16.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff428e04700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff424066950 0x7ff4240a40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.584+0000 7ff429e06700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4240aebc0 0x7ff424155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 -- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4240aebc0 msgr2=0x7ff424155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4240aebc0 0x7ff424155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 -- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff424066950 msgr2=0x7ff4240a40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff424066950 0x7ff4240a40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 -- 172.21.15.26:0/1504900870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c00b180 con 0x7ff42405d050 2022-07-20T02:31:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff429605700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7ff420001490 tx=0x7ff420007ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42000e4a0 con 0x7ff42405d050 2022-07-20T02:31:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42000e600 con 0x7ff42405d050 2022-07-20T02:31:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4241530b0 con 0x7ff42405d050 2022-07-20T02:31:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42000e960 con 0x7ff42405d050 2022-07-20T02:31:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4240a45a0 con 0x7ff42405d050 2022-07-20T02:31:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4240ab280 con 0x7ff42405d050 2022-07-20T02:31:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff420019030 con 0x7ff42405d050 2022-07-20T02:31:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 0x7ff400055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7ff42006b560 con 0x7ff42405d050 2022-07-20T02:31:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.588+0000 7ff428e04700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 0x7ff400055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.592+0000 7ff428e04700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 0x7ff400055f30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7ff41800d000 tx=0x7ff418002850).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.596+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff420042560 con 0x7ff42405d050 2022-07-20T02:31:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.724+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff42405fd10 con 0x7ff42405d050 2022-07-20T02:31:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.728+0000 7ff416ffd700 1 -- 172.21.15.26:0/1504900870 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff420045b80 con 0x7ff42405d050 2022-07-20T02:31:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 msgr2=0x7ff400055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 0x7ff400055f30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7ff41800d000 tx=0x7ff418002850).stop 2022-07-20T02:31:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 msgr2=0x7ff4240a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7ff420001490 tx=0x7ff420007ee0).stop 2022-07-20T02:31:16.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 shutdown_connections 2022-07-20T02:31:16.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff400053a70 0x7ff400055f30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4240aebc0 0x7ff424155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff424066950 0x7ff4240a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 --2- 172.21.15.26:0/1504900870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff42405d050 0x7ff4240a3bc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 >> 172.21.15.26:0/1504900870 conn(0x7ff424060950 msgr2=0x7ff424142e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 shutdown_connections 2022-07-20T02:31:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.732+0000 7ff42b869700 1 -- 172.21.15.26:0/1504900870 wait complete. 2022-07-20T02:31:16.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-20T02:31:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 Processor -- start 2022-07-20T02:31:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- start start 2022-07-20T02:31:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 0x7feaa0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feaa0147000 0x7feaa00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa01474e0 con 0x7feaa0060f00 2022-07-20T02:31:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa005d3b0 con 0x7feaa0147000 2022-07-20T02:31:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa01431e0 con 0x7feaa005cfd0 2022-07-20T02:31:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feaa0147000 0x7feaa00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feaa0147000 0x7feaa00b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38954/0 (socket says 172.21.15.26:38954) 2022-07-20T02:31:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 -- 172.21.15.26:0/983506028 learned_addr learned my addr 172.21.15.26:0/983506028 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9effd700 1 --2- 172.21.15.26:0/983506028 >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 0x7feaa0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 -- 172.21.15.26:0/983506028 >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 msgr2=0x7feaa0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 --2- 172.21.15.26:0/983506028 >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 0x7feaa0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 -- 172.21.15.26:0/983506028 >> v1:172.21.15.26:6789/0 conn(0x7feaa0060f00 legacy=0x7feaa013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 -- 172.21.15.26:0/983506028 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa0143480 con 0x7feaa0147000 2022-07-20T02:31:16.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9effd700 1 --2- 172.21.15.26:0/983506028 >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 0x7feaa0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:31:16.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9e7fc700 1 --2- 172.21.15.26:0/983506028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0147000 0x7feaa00b4730 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fea88002f70 tx=0x7fea88006eb0).ready entity=mon.0 client_cookie=d6fbaeee6f480973 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9dffb700 1 -- 172.21.15.26:0/983506028 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea8800d610 con 0x7feaa0147000 2022-07-20T02:31:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9dffb700 1 -- 172.21.15.26:0/983506028 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88000e90 con 0x7feaa0147000 2022-07-20T02:31:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9dffb700 1 -- 172.21.15.26:0/983506028 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea8800d9a0 con 0x7feaa0147000 2022-07-20T02:31:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- 172.21.15.26:0/983506028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0147000 msgr2=0x7feaa00b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 --2- 172.21.15.26:0/983506028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0147000 0x7feaa00b4730 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fea88002f70 tx=0x7fea88006eb0).stop 2022-07-20T02:31:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- 172.21.15.26:0/983506028 shutdown_connections 2022-07-20T02:31:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 --2- 172.21.15.26:0/983506028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0147000 0x7feaa00b4730 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 --2- 172.21.15.26:0/983506028 >> v2:172.21.15.120:3300/0 conn(0x7feaa005cfd0 0x7feaa0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- 172.21.15.26:0/983506028 >> 172.21.15.26:0/983506028 conn(0x7feaa00620c0 msgr2=0x7feaa005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.892+0000 7fea9ffff700 1 -- 172.21.15.26:0/983506028 shutdown_connections 2022-07-20T02:31:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 -- 172.21.15.26:0/983506028 wait complete. 2022-07-20T02:31:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 Processor -- start 2022-07-20T02:31:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 -- start start 2022-07-20T02:31:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaa00b7610 0x7feaa0152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 0x7feaa01527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0152ce0 con 0x7feaa0060f00 2022-07-20T02:31:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa01559e0 con 0x7feaa00b7a70 2022-07-20T02:31:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0155c80 con 0x7feaa00b7610 2022-07-20T02:31:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9e7fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaa00b7610 0x7feaa0152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 0x7feaa01527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38962/0 (socket says 172.21.15.26:38962) 2022-07-20T02:31:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9effd700 1 -- 172.21.15.26:0/1265346459 learned_addr learned my addr 172.21.15.26:0/1265346459 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 -- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaa00b7610 msgr2=0x7feaa0152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaa00b7610 0x7feaa0152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 -- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 msgr2=0x7feaa01458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 -- 172.21.15.26:0/1265346459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8800b1e0 con 0x7feaa00b7a70 2022-07-20T02:31:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9effd700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.896+0000 7fea9f7fe700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 0x7feaa01527f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7feaa0155900 tx=0x7fea98007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9800ce00 con 0x7feaa00b7a70 2022-07-20T02:31:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea98006aa0 con 0x7feaa00b7a70 2022-07-20T02:31:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa0155f20 con 0x7feaa00b7a70 2022-07-20T02:31:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea98006e00 con 0x7feaa00b7a70 2022-07-20T02:31:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa01566e0 con 0x7feaa00b7a70 2022-07-20T02:31:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa0143720 con 0x7feaa00b7a70 2022-07-20T02:31:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fea98010030 con 0x7feaa00b7a70 2022-07-20T02:31:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 0x7fea84055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:16.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fea9806bbd0 con 0x7feaa00b7a70 2022-07-20T02:31:16.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea9effd700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 0x7fea84055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:16.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.900+0000 7fea9effd700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 0x7fea84055e10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fea9000d000 tx=0x7fea90008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:16.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:16.908+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea98042ae0 con 0x7feaa00b7a70 2022-07-20T02:31:16.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:16.985 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:16.988 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:16.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:17.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.076+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7feaa00574e0 con 0x7feaa00b7a70 2022-07-20T02:31:17.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.456+0000 7fea97fff700 1 -- 172.21.15.26:0/1265346459 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v253) v1 ==== 147+0+0 (secure 0 0 0) 0x7fea98046100 con 0x7feaa00b7a70 2022-07-20T02:31:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 msgr2=0x7fea84055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 0x7fea84055e10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fea9000d000 tx=0x7fea90008040).stop 2022-07-20T02:31:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 msgr2=0x7feaa01527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 0x7feaa01527f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7feaa0155900 tx=0x7fea98007f10).stop 2022-07-20T02:31:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 shutdown_connections 2022-07-20T02:31:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fea84053950 0x7fea84055e10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaa00b7a70 0x7feaa01527f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaa00b7610 0x7feaa0152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 --2- 172.21.15.26:0/1265346459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaa0060f00 0x7feaa01458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 >> 172.21.15.26:0/1265346459 conn(0x7feaa005cfd0 msgr2=0x7feaa013a460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 shutdown_connections 2022-07-20T02:31:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.460+0000 7fea9ffff700 1 -- 172.21.15.26:0/1265346459 wait complete. 2022-07-20T02:31:17.475 INFO:tasks.cephfs_test_runner:test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T02:31:17.475 INFO:tasks.cephfs_test_runner:Starting test: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:31:17.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-20T02:31:17.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 Processor -- start 2022-07-20T02:31:17.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 -- start start 2022-07-20T02:31:17.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f79b00af4a0 0x7f79b0149d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:17.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b0067030 con 0x7f79b00aa070 2022-07-20T02:31:17.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00aeec0 con 0x7f79b00aeae0 2022-07-20T02:31:17.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b7190700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b0066af0 con 0x7f79b00af4a0 2022-07-20T02:31:17.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79affff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f79b00af4a0 0x7f79b0149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:17.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b572d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79b00aa070 0x7f79b014a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38252/0 (socket says 172.21.15.26:38252) 2022-07-20T02:31:17.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79b572d700 1 -- 172.21.15.26:0/14826795 learned_addr learned my addr 172.21.15.26:0/14826795 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:17.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79affff700 1 -- 172.21.15.26:0/14826795 >> v1:172.21.15.120:6789/0 conn(0x7f79b00aeae0 legacy=0x7f79b0061780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79affff700 1 -- 172.21.15.26:0/14826795 >> v1:172.21.15.26:6789/0 conn(0x7f79b00aa070 legacy=0x7f79b014a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79affff700 1 -- 172.21.15.26:0/14826795 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0066d90 con 0x7f79b00af4a0 2022-07-20T02:31:17.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79affff700 1 --2- 172.21.15.26:0/14826795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0149d70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f79a8001110 tx=0x7f79a00091d0).ready entity=mon.0 client_cookie=9b60c2a86d96e921 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:17.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79af7fe700 1 -- 172.21.15.26:0/14826795 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a0009ea0 con 0x7f79b00af4a0 2022-07-20T02:31:17.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79af7fe700 1 -- 172.21.15.26:0/14826795 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a000b760 con 0x7f79b00af4a0 2022-07-20T02:31:17.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.600+0000 7f79af7fe700 1 -- 172.21.15.26:0/14826795 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a000b8d0 con 0x7f79b00af4a0 2022-07-20T02:31:17.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- 172.21.15.26:0/14826795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 msgr2=0x7f79b0149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 --2- 172.21.15.26:0/14826795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0149d70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f79a8001110 tx=0x7f79a00091d0).stop 2022-07-20T02:31:17.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- 172.21.15.26:0/14826795 shutdown_connections 2022-07-20T02:31:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 --2- 172.21.15.26:0/14826795 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0149d70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- 172.21.15.26:0/14826795 >> 172.21.15.26:0/14826795 conn(0x7f79b00b66e0 msgr2=0x7f79b00b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- 172.21.15.26:0/14826795 shutdown_connections 2022-07-20T02:31:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- 172.21.15.26:0/14826795 wait complete. 2022-07-20T02:31:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 Processor -- start 2022-07-20T02:31:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.604+0000 7f79b7190700 1 -- start start 2022-07-20T02:31:17.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:17.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 0x7f79b0146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:17.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:17.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b0156aa0 con 0x7f79b00af4a0 2022-07-20T02:31:17.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00abbb0 con 0x7f79b00aeae0 2022-07-20T02:31:17.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00abe50 con 0x7f79b00aa070 2022-07-20T02:31:17.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:17.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b4f2c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:17.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79affff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 0x7f79b0146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:17.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b4f2c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60490/0 (socket says 172.21.15.26:60490) 2022-07-20T02:31:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b4f2c700 1 -- 172.21.15.26:0/3606618856 learned_addr learned my addr 172.21.15.26:0/3606618856 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38978/0 (socket says 172.21.15.26:38978) 2022-07-20T02:31:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 -- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 msgr2=0x7f79b00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 -- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 msgr2=0x7f79b0146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:17.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 0x7f79b0146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:17.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 -- 172.21.15.26:0/3606618856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a0009470 con 0x7f79b00af4a0 2022-07-20T02:31:17.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79affff700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 0x7f79b0146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:17.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b4f2c700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b572d700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f79a80011d0 tx=0x7f79a8007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a80070d0 con 0x7f79b00af4a0 2022-07-20T02:31:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8007230 con 0x7f79b00af4a0 2022-07-20T02:31:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b00ac0f0 con 0x7f79b00af4a0 2022-07-20T02:31:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a8005610 con 0x7f79b00af4a0 2022-07-20T02:31:17.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.608+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b00b2470 con 0x7f79b00af4a0 2022-07-20T02:31:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b00602b0 con 0x7f79b00af4a0 2022-07-20T02:31:17.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f79a8010070 con 0x7f79b00af4a0 2022-07-20T02:31:17.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79adffb700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 0x7f7990055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:17.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f79a806b4c0 con 0x7f79b00af4a0 2022-07-20T02:31:17.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79b4f2c700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 0x7f7990055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:17.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.612+0000 7f79b4f2c700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 0x7f7990055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f79a400d000 tx=0x7f79a4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:17.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.620+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79a8042450 con 0x7f79b00af4a0 2022-07-20T02:31:17.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:17.780+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7f79b00574e0 con 0x7f79b00af4a0 2022-07-20T02:31:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.456+0000 7f79adffb700 1 -- 172.21.15.26:0/3606618856 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v254) v1 ==== 142+0+0 (secure 0 0 0) 0x7f79a8045a70 con 0x7f79b00af4a0 2022-07-20T02:31:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.460+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 msgr2=0x7f7990055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.460+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 0x7f7990055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f79a400d000 tx=0x7f79a4009040).stop 2022-07-20T02:31:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.460+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 msgr2=0x7f79b0147230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.460+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f79a80011d0 tx=0x7f79a8007af0).stop 2022-07-20T02:31:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 shutdown_connections 2022-07-20T02:31:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79b00af4a0 0x7f79b0147230 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7990053a70 0x7f7990055f30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b00aeae0 0x7f79b0146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 --2- 172.21.15.26:0/3606618856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79b00aa070 0x7f79b00b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 >> 172.21.15.26:0/3606618856 conn(0x7f79b00b66e0 msgr2=0x7f79b014d620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 shutdown_connections 2022-07-20T02:31:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.464+0000 7f79b7190700 1 -- 172.21.15.26:0/3606618856 wait complete. 2022-07-20T02:31:18.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:31:18.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 Processor -- start 2022-07-20T02:31:18.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 -- start start 2022-07-20T02:31:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10880b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10880b2ca0 0x7f1088144840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10880b14b0 con 0x7f10880657b0 2022-07-20T02:31:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1088067620 con 0x7f10880b2ca0 2022-07-20T02:31:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880aecf0 con 0x7f1088067240 2022-07-20T02:31:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10880b2ca0 0x7f1088144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10880b2ca0 0x7f1088144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38986/0 (socket says 172.21.15.26:38986) 2022-07-20T02:31:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 -- 172.21.15.26:0/2773285663 learned_addr learned my addr 172.21.15.26:0/2773285663 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:18.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108659c700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10880b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 -- 172.21.15.26:0/2773285663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 msgr2=0x7f10880b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:18.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10880b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 -- 172.21.15.26:0/2773285663 >> v1:172.21.15.26:6789/0 conn(0x7f10880657b0 legacy=0x7f1088144d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 -- 172.21.15.26:0/2773285663 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880b1980 con 0x7f10880b2ca0 2022-07-20T02:31:18.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f1085d9b700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880b2ca0 0x7f1088144840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f107c002f70 tx=0x7f107c006eb0).ready entity=mon.0 client_cookie=5d8e57d2e2194c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:18.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108559a700 1 -- 172.21.15.26:0/2773285663 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f107c00d570 con 0x7f10880b2ca0 2022-07-20T02:31:18.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108559a700 1 -- 172.21.15.26:0/2773285663 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107c000e90 con 0x7f10880b2ca0 2022-07-20T02:31:18.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.608+0000 7f108559a700 1 -- 172.21.15.26:0/2773285663 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f107c00d900 con 0x7f10880b2ca0 2022-07-20T02:31:18.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- 172.21.15.26:0/2773285663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880b2ca0 msgr2=0x7f1088144840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880b2ca0 0x7f1088144840 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f107c002f70 tx=0x7f107c006eb0).stop 2022-07-20T02:31:18.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- 172.21.15.26:0/2773285663 shutdown_connections 2022-07-20T02:31:18.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880b2ca0 0x7f1088144840 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 --2- 172.21.15.26:0/2773285663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10880b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- 172.21.15.26:0/2773285663 >> 172.21.15.26:0/2773285663 conn(0x7f108805f840 msgr2=0x7f108805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:18.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- 172.21.15.26:0/2773285663 shutdown_connections 2022-07-20T02:31:18.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- 172.21.15.26:0/2773285663 wait complete. 2022-07-20T02:31:18.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 Processor -- start 2022-07-20T02:31:18.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.612+0000 7f108759e700 1 -- start start 2022-07-20T02:31:18.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10881450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10880b2ca0 0x7f1088141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10880ab580 con 0x7f10880657b0 2022-07-20T02:31:18.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10881455e0 con 0x7f10880b2ca0 2022-07-20T02:31:18.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880b0980 con 0x7f1088067240 2022-07-20T02:31:18.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38990/0 (socket says 172.21.15.26:38990) 2022-07-20T02:31:18.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 -- 172.21.15.26:0/2081229961 learned_addr learned my addr 172.21.15.26:0/2081229961 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:18.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f1086d9d700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10880b2ca0 0x7f1088141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f1085d9b700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10881450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 -- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 msgr2=0x7f10881450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10881450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 -- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10880b2ca0 msgr2=0x7f1088141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10880b2ca0 0x7f1088141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 -- 172.21.15.26:0/2081229961 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c00b1e0 con 0x7f10880657b0 2022-07-20T02:31:18.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108659c700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f107800c000 tx=0x7f1078006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:18.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10780104c0 con 0x7f10880657b0 2022-07-20T02:31:18.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1078010620 con 0x7f10880657b0 2022-07-20T02:31:18.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108805c5f0 con 0x7f10880657b0 2022-07-20T02:31:18.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1078010980 con 0x7f10880657b0 2022-07-20T02:31:18.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.616+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10880af020 con 0x7f10880657b0 2022-07-20T02:31:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10880afd80 con 0x7f10880657b0 2022-07-20T02:31:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f107801a030 con 0x7f10880657b0 2022-07-20T02:31:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f10777fe700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 0x7f1064055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f107806b9c0 con 0x7f10880657b0 2022-07-20T02:31:18.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f1085d9b700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 0x7f1064055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.620+0000 7f1085d9b700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 0x7f1064055e10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f107c01c000 tx=0x7f107c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.624+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1078042920 con 0x7f10880657b0 2022-07-20T02:31:18.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.756+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f10880b0b40 con 0x7f10880657b0 2022-07-20T02:31:18.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f10777fe700 1 -- 172.21.15.26:0/2081229961 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1078045f40 con 0x7f10880657b0 2022-07-20T02:31:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 msgr2=0x7f1064055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 0x7f1064055e10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f107c01c000 tx=0x7f107c015000).stop 2022-07-20T02:31:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 msgr2=0x7f10880b6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f107800c000 tx=0x7f1078006d90).stop 2022-07-20T02:31:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 shutdown_connections 2022-07-20T02:31:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1064053950 0x7f1064055e10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10880b2ca0 0x7f1088141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1088067240 0x7f10881450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 --2- 172.21.15.26:0/2081229961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10880657b0 0x7f10880b6370 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 >> 172.21.15.26:0/2081229961 conn(0x7f108805f840 msgr2=0x7f10880aa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 shutdown_connections 2022-07-20T02:31:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:18.760+0000 7f108759e700 1 -- 172.21.15.26:0/2081229961 wait complete. 2022-07-20T02:31:18.776 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:31:18.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:18.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:18.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:18.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:31:18.880 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:31:18.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:18.883 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:18.883 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:31:18.914 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:31:18.914 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:31:18.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:18.916 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:18.916 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:18.928 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:31:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:31:18.931 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:31:18.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:31:19.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 Processor -- start 2022-07-20T02:31:19.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 -- start start 2022-07-20T02:31:19.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5cb80b3260 0x7f5cb8060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb8095b10 0x7f5cb8095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb8067dc0 con 0x7f5cb80a95a0 2022-07-20T02:31:19.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb80b3640 con 0x7f5cb80b3260 2022-07-20T02:31:19.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbdbde700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80615c0 con 0x7f5cb8095b10 2022-07-20T02:31:19.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cb7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5cb80a95a0 0x7f5cb80676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38280/0 (socket says 172.21.15.26:38280) 2022-07-20T02:31:19.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cb7fff700 1 -- 172.21.15.26:0/76701230 learned_addr learned my addr 172.21.15.26:0/76701230 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:19.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/76701230 >> v2:172.21.15.120:3300/0 conn(0x7f5cb80b3260 0x7f5cb8060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbd3dd700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb8095b10 0x7f5cb8095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cb77fe700 1 -- 172.21.15.26:0/76701230 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704516668 0 0) 0x7f5cb8067dc0 con 0x7f5cb80a95a0 2022-07-20T02:31:19.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cb77fe700 1 -- 172.21.15.26:0/76701230 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca4003260 con 0x7f5cb80a95a0 2022-07-20T02:31:19.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbcbdc700 1 -- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb8095b10 msgr2=0x7f5cb8095f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:19.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb8095b10 0x7f5cb8095f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbcbdc700 1 -- 172.21.15.26:0/76701230 >> v1:172.21.15.26:6789/0 conn(0x7f5cb80a95a0 legacy=0x7f5cb80676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.048+0000 7f5cbcbdc700 1 -- 172.21.15.26:0/76701230 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8061860 con 0x7f5cb80b3260 2022-07-20T02:31:19.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb80b3260 0x7f5cb8060950 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f5ca8002f70 tx=0x7f5ca8006df0).ready entity=mon.1 client_cookie=3c9caf00b2d8dac3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cb77fe700 1 -- 172.21.15.26:0/76701230 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ca800d4a0 con 0x7f5cb80b3260 2022-07-20T02:31:19.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cb77fe700 1 -- 172.21.15.26:0/76701230 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca8000e90 con 0x7f5cb80b3260 2022-07-20T02:31:19.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cb77fe700 1 -- 172.21.15.26:0/76701230 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ca800d7f0 con 0x7f5cb80b3260 2022-07-20T02:31:19.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 -- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb80b3260 msgr2=0x7f5cb8060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb80b3260 0x7f5cb8060950 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f5ca8002f70 tx=0x7f5ca8006df0).stop 2022-07-20T02:31:19.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 -- 172.21.15.26:0/76701230 shutdown_connections 2022-07-20T02:31:19.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb8095b10 0x7f5cb8095f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/76701230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb80b3260 0x7f5cb8060950 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 -- 172.21.15.26:0/76701230 >> 172.21.15.26:0/76701230 conn(0x7f5cb80b67f0 msgr2=0x7f5cb80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:19.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 -- 172.21.15.26:0/76701230 shutdown_connections 2022-07-20T02:31:19.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.052+0000 7f5cbdbde700 1 -- 172.21.15.26:0/76701230 wait complete. 2022-07-20T02:31:19.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 Processor -- start 2022-07-20T02:31:19.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 -- start start 2022-07-20T02:31:19.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 0x7f5cb80651f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb80a95a0 0x7f5cb8064660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb809a410 con 0x7f5cb80b3260 2022-07-20T02:31:19.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80634d0 con 0x7f5cb8095b10 2022-07-20T02:31:19.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbdbde700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb8155750 con 0x7f5cb80a95a0 2022-07-20T02:31:19.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39008/0 (socket says 172.21.15.26:39008) 2022-07-20T02:31:19.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 -- 172.21.15.26:0/2208538421 learned_addr learned my addr 172.21.15.26:0/2208538421 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:19.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 0x7f5cb80651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cb7fff700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb80a95a0 0x7f5cb8064660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 -- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb80a95a0 msgr2=0x7f5cb8064660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb80a95a0 0x7f5cb8064660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 -- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 msgr2=0x7f5cb80651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 0x7f5cb80651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 -- 172.21.15.26:0/2208538421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca800b180 con 0x7f5cb80b3260 2022-07-20T02:31:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 0x7f5cb80651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.056+0000 7f5cbd3dd700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f5cb0006e00 tx=0x7f5cb00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb0012600 con 0x7f5cb80b3260 2022-07-20T02:31:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0012760 con 0x7f5cb80b3260 2022-07-20T02:31:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb80a9ca0 con 0x7f5cb80b3260 2022-07-20T02:31:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb0012ac0 con 0x7f5cb80b3260 2022-07-20T02:31:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb8064b50 con 0x7f5cb80b3260 2022-07-20T02:31:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5cb0022070 con 0x7f5cb80b3260 2022-07-20T02:31:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb80aa140 con 0x7f5cb80b3260 2022-07-20T02:31:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 0x7f5c98055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5cb006c2e0 con 0x7f5cb80b3260 2022-07-20T02:31:19.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 0x7f5c98055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.060+0000 7f5cbcbdc700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 0x7f5c98055f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5ca801a000 tx=0x7f5ca8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.068+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5cb00432a0 con 0x7f5cb80b3260 2022-07-20T02:31:19.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.200+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5cb80639b0 con 0x7f5cb80b3260 2022-07-20T02:31:19.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.200+0000 7f5cb5ffb700 1 -- 172.21.15.26:0/2208538421 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v69) v1 ==== 81+0+29268 (secure 0 0 0) 0x7f5cb00468c0 con 0x7f5cb80b3260 2022-07-20T02:31:19.200 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:19.200 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:31:19.201 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 69, 2022-07-20T02:31:19.201 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:31:19.201 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:31:19.201 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:31:16.432748+0000", 2022-07-20T02:31:19.202 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:31:19.202 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:31:19.202 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:31:19.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:31:19.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:31:19.203 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:31:19.203 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:31:19.203 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:31:19.204 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:31:19.204 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:19.205 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:31:19.205 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:31:19.205 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:31:19.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:31:19.206 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:31:19.206 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 12, 2022-07-20T02:31:19.206 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:31:19.207 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:31:19.207 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:31:19.207 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:31:19.207 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:31:19.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.209 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:31:19.209 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:31:19.210 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:31:19.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:19.211 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:19.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:19.212 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:19.213 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:19.214 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:19.214 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:19.216 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:19.217 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:31:19.217 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:31:19.217 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:31:19.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:31:19.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:31:19.219 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:19.219 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:19.220 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:19.220 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:19.220 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:19.221 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:19.221 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:19.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.222 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:31:19.222 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:19.223 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:19.223 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:19.223 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:19.224 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:19.224 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:19.224 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:19.225 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:19.225 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:19.225 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:19.226 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:19.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:19.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:19.228 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:19.229 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:19.230 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:19.231 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:19.231 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:31:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:31:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:31:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:31:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:31:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:31:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:31:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:31:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:31:19.246 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:31:19.246 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:31:19.247 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:31:19.247 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:31:19.248 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:31:19.248 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:31:19.248 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:31:19.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:31:19.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:31:19.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:31:19.249 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:31:19.250 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:31:19.250 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:31:19.251 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:31:19.251 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:31:19.251 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:31:19.252 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:31:19.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.252 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:31:19.253 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:31:19.253 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:31:19.254 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:31:19.254 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:31:19.255 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:31:19.255 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:31:19.256 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:31:19.256 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:31:19.257 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:31:19.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:31:19.258 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:31:19.259 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:31:19.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:31:19.260 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:31:19.261 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:31:19.261 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:31:19.261 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:31:19.262 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:31:19.262 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:31:19.263 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:31:19.263 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:31:19.264 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:31:19.264 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:31:19.264 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:31:19.265 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:31:19.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:31:19.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.266 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:31:19.267 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:31:19.267 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:31:19.268 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:31:19.268 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:31:19.269 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:31:19.270 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:31:19.270 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:31:19.271 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:31:19.271 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:31:19.272 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:31:19.272 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:31:19.273 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:31:19.273 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:31:19.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.274 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:19.275 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:31:19.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.275 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:31:19.276 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:31:19.276 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.277 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.277 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.278 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.279 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.279 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:19.279 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.280 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:31:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:31:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.286 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:31:19.288 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:31:19.291 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.301 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.306 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:31:19.306 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:31:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:31:19.312 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:31:19.315 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:31:19.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.317 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:31:19.318 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:31:19.318 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:31:19.319 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:31:19.321 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.321 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.322 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.322 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.327 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:31:19.327 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:31:19.328 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.328 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.328 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.329 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.329 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:31:19.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:31:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.334 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:31:19.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:31:19.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.339 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:31:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:31:19.342 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.344 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:31:19.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:31:19.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:31:19.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:31:19.352 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:31:19.353 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:31:19.353 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:31:19.354 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.354 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.355 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.357 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:31:19.357 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:31:19.358 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.358 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.359 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.359 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.360 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.360 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.360 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.361 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:19.361 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.362 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:31:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:31:19.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:31:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:31:19.374 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.376 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:31:19.378 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:31:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.381 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:31:19.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:31:19.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:31:19.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.387 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:31:19.387 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:31:19.388 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:31:19.388 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:31:19.389 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.389 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.389 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:31:19.392 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:31:19.392 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.392 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.393 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.393 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.394 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.394 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.395 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.395 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 51, 2022-07-20T02:31:19.396 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.396 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:31:19.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:31:19.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.406 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:31:19.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:31:19.417 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.419 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:31:19.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:31:19.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.425 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:31:19.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:31:19.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:31:19.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.433 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:31:19.433 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:31:19.434 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:31:19.434 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:31:19.435 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.435 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.435 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.437 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:31:19.437 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:31:19.438 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.438 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.438 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.439 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.439 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.440 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.440 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.441 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:31:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:31:19.447 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.448 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:31:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:31:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:31:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:31:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:31:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:31:19.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:31:19.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.468 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:31:19.469 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:31:19.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:31:19.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:31:19.469 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.470 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.470 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.471 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:31:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:31:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.473 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.473 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.473 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.474 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.474 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:19.474 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.474 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.475 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:31:19.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:31:19.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.478 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:31:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:31:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:31:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:31:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:31:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:31:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:31:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:31:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:31:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:31:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:31:19.489 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.489 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.489 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.491 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:31:19.492 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:31:19.492 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.492 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.493 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.493 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.495 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:31:19.495 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.495 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:31:19.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:31:19.499 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.501 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:31:19.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:31:19.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.505 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.506 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:31:19.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:31:19.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.512 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:31:19.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:31:19.525 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:31:19.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:31:19.527 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:31:19.527 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:31:19.527 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:31:19.527 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:31:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:31:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:31:19.530 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:31:19.530 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:31:19.531 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:31:19.531 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:31:19.533 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:31:19.534 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:31:19.534 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2022-07-20T02:31:19.534 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:31:19.535 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:31:19.535 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:31:19.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:31:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:31:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:31:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:31:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:31:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:31:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:31:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:31:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:31:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:31:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:31:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:31:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:31:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:31:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:31:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:31:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:31:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:31:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:31:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:31:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:31:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:31:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:31:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:31:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.555 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.555 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.555 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:31:19.556 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.557 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:31:19.557 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.558 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.558 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.558 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.559 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.559 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:31:19.560 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.561 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:31:19.561 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.562 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.562 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.563 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.563 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:31:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.565 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:31:19.565 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.566 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.566 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.567 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.567 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.567 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:31:19.568 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.569 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:31:19.569 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.569 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.569 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:31:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:31:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:31:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:31:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:31:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:31:19.578 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:31:19.578 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:31:19.578 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:31:19.579 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:31:19.579 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:31:19.580 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:31:19.580 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:31:19.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.581 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:31:19.581 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:31:19.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:31:19.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:31:19.583 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:31:19.583 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:31:19.584 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/2926574302": "2022-07-21T02:31:13.383005+0000", 2022-07-20T02:31:19.584 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/2926574302": "2022-07-21T02:31:13.383005+0000" 2022-07-20T02:31:19.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.585 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:31:19.586 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:31:19.586 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:31:19.587 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:31:19.587 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:31:19.588 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:31:19.588 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:31:19.589 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:31:19.589 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:31:19.590 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:31:19.590 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:31:19.591 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:31:19.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.592 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:31:19.592 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:31:19.592 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:31:19.593 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:31:19.593 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:31:19.594 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:31:19.594 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:31:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:31:19.596 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:31:19.596 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:31:19.597 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:31:19.597 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:31:19.598 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:31:19.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 msgr2=0x7f5c98055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 0x7f5c98055f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5ca801a000 tx=0x7f5ca8014000).stop 2022-07-20T02:31:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 msgr2=0x7f5cb80af5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f5cb0006e00 tx=0x7f5cb00091d0).stop 2022-07-20T02:31:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 shutdown_connections 2022-07-20T02:31:19.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cb80b3260 0x7f5cb80af5d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c98053a70 0x7f5c98055f30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cb80a95a0 0x7f5cb8064660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 --2- 172.21.15.26:0/2208538421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cb8095b10 0x7f5cb80651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 >> 172.21.15.26:0/2208538421 conn(0x7f5cb80b67f0 msgr2=0x7f5cb8145770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:19.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 shutdown_connections 2022-07-20T02:31:19.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.204+0000 7f5cbdbde700 1 -- 172.21.15.26:0/2208538421 wait complete. 2022-07-20T02:31:19.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:19.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 Processor -- start 2022-07-20T02:31:19.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- start start 2022-07-20T02:31:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2234061860 0x7f2234067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f22340980d0 0x7f2234142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223405ff70 con 0x7f22340b3260 2022-07-20T02:31:19.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340b3640 con 0x7f2234061860 2022-07-20T02:31:19.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340601a0 con 0x7f22340980d0 2022-07-20T02:31:19.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2234061860 0x7f2234067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2234061860 0x7f2234067900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39016/0 (socket says 172.21.15.26:39016) 2022-07-20T02:31:19.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 -- 172.21.15.26:0/2562446090 learned_addr learned my addr 172.21.15.26:0/2562446090 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:19.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22337fe700 1 --2- 172.21.15.26:0/2562446090 >> v2:172.21.15.120:3300/0 conn(0x7f22340980d0 0x7f2234142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 -- 172.21.15.26:0/2562446090 >> v2:172.21.15.120:3300/0 conn(0x7f22340980d0 msgr2=0x7f2234142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:19.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 --2- 172.21.15.26:0/2562446090 >> v2:172.21.15.120:3300/0 conn(0x7f22340980d0 0x7f2234142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 -- 172.21.15.26:0/2562446090 >> v1:172.21.15.120:6789/0 conn(0x7f22340b3260 legacy=0x7f22340979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:19.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 -- 172.21.15.26:0/2562446090 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340606e0 con 0x7f2234061860 2022-07-20T02:31:19.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f22327fc700 1 --2- 172.21.15.26:0/2562446090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f2234067900 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f2234097600 tx=0x7f2228006eb0).ready entity=mon.0 client_cookie=b0100b089a4b4bbe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2231ffb700 1 -- 172.21.15.26:0/2562446090 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f222800d5c0 con 0x7f2234061860 2022-07-20T02:31:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2231ffb700 1 -- 172.21.15.26:0/2562446090 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2228000e90 con 0x7f2234061860 2022-07-20T02:31:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2231ffb700 1 -- 172.21.15.26:0/2562446090 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f222800d910 con 0x7f2234061860 2022-07-20T02:31:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- 172.21.15.26:0/2562446090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 msgr2=0x7f2234067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 --2- 172.21.15.26:0/2562446090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f2234067900 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f2234097600 tx=0x7f2228006eb0).stop 2022-07-20T02:31:19.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- 172.21.15.26:0/2562446090 shutdown_connections 2022-07-20T02:31:19.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 --2- 172.21.15.26:0/2562446090 >> v2:172.21.15.120:3300/0 conn(0x7f22340980d0 0x7f2234142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 --2- 172.21.15.26:0/2562446090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f2234067900 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- 172.21.15.26:0/2562446090 >> 172.21.15.26:0/2562446090 conn(0x7f22340b67f0 msgr2=0x7f22340b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- 172.21.15.26:0/2562446090 shutdown_connections 2022-07-20T02:31:19.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.728+0000 7f2233fff700 1 -- 172.21.15.26:0/2562446090 wait complete. 2022-07-20T02:31:19.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 Processor -- start 2022-07-20T02:31:19.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 -- start start 2022-07-20T02:31:19.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22340980d0 0x7f223413fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22340b3260 0x7f2234152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b4d90 con 0x7f2234061860 2022-07-20T02:31:19.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b5030 con 0x7f22340980d0 2022-07-20T02:31:19.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2233fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b4480 con 0x7f22340b3260 2022-07-20T02:31:19.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39020/0 (socket says 172.21.15.26:39020) 2022-07-20T02:31:19.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f22337fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22340b3260 0x7f2234152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 -- 172.21.15.26:0/1412065159 learned_addr learned my addr 172.21.15.26:0/1412065159 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f22327fc700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22340980d0 0x7f223413fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 -- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22340b3260 msgr2=0x7f2234152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22340b3260 0x7f2234152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 -- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22340980d0 msgr2=0x7f223413fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22340980d0 0x7f223413fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 -- 172.21.15.26:0/1412065159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222800b1e0 con 0x7f2234061860 2022-07-20T02:31:19.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2232ffd700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f2224002f30 tx=0x7f2224006d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.732+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2224012410 con 0x7f2234061860 2022-07-20T02:31:19.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224012570 con 0x7f2234061860 2022-07-20T02:31:19.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22340b4920 con 0x7f2234061860 2022-07-20T02:31:19.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22240128d0 con 0x7f2234061860 2022-07-20T02:31:19.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223409bb40 con 0x7f2234061860 2022-07-20T02:31:19.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340b51f0 con 0x7f2234061860 2022-07-20T02:31:19.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f222400f030 con 0x7f2234061860 2022-07-20T02:31:19.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2223fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 0x7f2210055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:19.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f222406b910 con 0x7f2234061860 2022-07-20T02:31:19.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.736+0000 7f22327fc700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 0x7f2210055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:19.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.740+0000 7f22327fc700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 0x7f2210055fc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f222801c000 tx=0x7f2228015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:19.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.744+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2224042950 con 0x7f2234061860 2022-07-20T02:31:19.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.912+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22340b4640 con 0x7f2234061860 2022-07-20T02:31:19.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.912+0000 7f2223fff700 1 -- 172.21.15.26:0/1412065159 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f2224045f70 con 0x7f2234061860 2022-07-20T02:31:19.918 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:19.918 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/1006992740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1006992740},{"type":"v1","addr":"172.21.15.26:6837","nonce":1006992740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-20T02:31:19.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 msgr2=0x7f2210055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 0x7f2210055fc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f222801c000 tx=0x7f2228015000).stop 2022-07-20T02:31:19.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 msgr2=0x7f22340afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:19.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f2224002f30 tx=0x7f2224006d70).stop 2022-07-20T02:31:19.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 shutdown_connections 2022-07-20T02:31:19.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2210053b00 0x7f2210055fc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22340b3260 0x7f2234152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22340980d0 0x7f223413fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 --2- 172.21.15.26:0/1412065159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2234061860 0x7f22340afe00 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 >> 172.21.15.26:0/1412065159 conn(0x7f22340b67f0 msgr2=0x7f22340944a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:19.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 shutdown_connections 2022-07-20T02:31:19.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:19.916+0000 7f2233fff700 1 -- 172.21.15.26:0/1412065159 wait complete. 2022-07-20T02:31:19.922 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2022-07-20T02:31:19.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.032+0000 7f5ea9e27700 1 Processor -- start 2022-07-20T02:31:20.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.032+0000 7f5ea9e27700 1 -- start start 2022-07-20T02:31:20.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5ea40b2920 0x7f5ea4147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea40b1bc0 con 0x7f5ea4067190 2022-07-20T02:31:20.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea4147700 con 0x7f5ea4061050 2022-07-20T02:31:20.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea414b040 con 0x7f5ea40b2920 2022-07-20T02:31:20.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5ea40b2920 0x7f5ea4147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ea4067190 0x7f5ea40ab680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38310/0 (socket says 172.21.15.26:38310) 2022-07-20T02:31:20.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea3fff700 1 -- 172.21.15.26:0/2547140224 learned_addr learned my addr 172.21.15.26:0/2547140224 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:20.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973462934 0 0) 0x7f5ea40b1bc0 con 0x7f5ea4067190 2022-07-20T02:31:20.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e90003260 con 0x7f5ea4067190 2022-07-20T02:31:20.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898297451 0 0) 0x7f5ea4147700 con 0x7f5ea4061050 2022-07-20T02:31:20.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea37fe700 1 -- 172.21.15.26:0/2547140224 >> v1:172.21.15.120:6789/0 conn(0x7f5ea4061050 legacy=0x7f5ea4066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea37fe700 1 -- 172.21.15.26:0/2547140224 >> v1:172.21.15.26:6789/0 conn(0x7f5ea4067190 legacy=0x7f5ea40ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea37fe700 1 -- 172.21.15.26:0/2547140224 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4147930 con 0x7f5ea40b2920 2022-07-20T02:31:20.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea37fe700 1 --2- 172.21.15.26:0/2547140224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4147d90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f5e9c001800 tx=0x7f5e940091d0).ready entity=mon.0 client_cookie=4f875f93430b7350 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e9c001660 con 0x7f5ea40b2920 2022-07-20T02:31:20.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9400b580 con 0x7f5ea40b2920 2022-07-20T02:31:20.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea27fc700 1 -- 172.21.15.26:0/2547140224 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e9400b6f0 con 0x7f5ea40b2920 2022-07-20T02:31:20.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- 172.21.15.26:0/2547140224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 msgr2=0x7f5ea4147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/2547140224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4147d90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f5e9c001800 tx=0x7f5e940091d0).stop 2022-07-20T02:31:20.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- 172.21.15.26:0/2547140224 shutdown_connections 2022-07-20T02:31:20.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/2547140224 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4147d90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- 172.21.15.26:0/2547140224 >> 172.21.15.26:0/2547140224 conn(0x7f5ea40b60a0 msgr2=0x7f5ea40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:20.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- 172.21.15.26:0/2547140224 shutdown_connections 2022-07-20T02:31:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.036+0000 7f5ea9e27700 1 -- 172.21.15.26:0/2547140224 wait complete. 2022-07-20T02:31:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 Processor -- start 2022-07-20T02:31:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- start start 2022-07-20T02:31:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 0x7f5ea40659a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 0x7f5ea4148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40ab260 con 0x7f5ea40b2920 2022-07-20T02:31:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40aaa80 con 0x7f5ea4067190 2022-07-20T02:31:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40aad20 con 0x7f5ea4061050 2022-07-20T02:31:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39038/0 (socket says 172.21.15.26:39038) 2022-07-20T02:31:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 -- 172.21.15.26:0/1530131983 learned_addr learned my addr 172.21.15.26:0/1530131983 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea37fe700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 0x7f5ea40659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea2ffd700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 0x7f5ea4148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 -- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 msgr2=0x7f5ea40659a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 0x7f5ea40659a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 -- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 msgr2=0x7f5ea4148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 0x7f5ea4148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 -- 172.21.15.26:0/1530131983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94009470 con 0x7f5ea40b2920 2022-07-20T02:31:20.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea37fe700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 0x7f5ea40659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:20.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea2ffd700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 0x7f5ea4148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:20.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea3fff700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f5e9c001370 tx=0x7f5e9c001980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e9c007170 con 0x7f5ea40b2920 2022-07-20T02:31:20.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0072d0 con 0x7f5ea40b2920 2022-07-20T02:31:20.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea40aafc0 con 0x7f5ea40b2920 2022-07-20T02:31:20.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e9c004920 con 0x7f5ea40b2920 2022-07-20T02:31:20.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.040+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea4066b90 con 0x7f5ea40b2920 2022-07-20T02:31:20.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4065080 con 0x7f5ea40b2920 2022-07-20T02:31:20.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e9c004a80 con 0x7f5ea40b2920 2022-07-20T02:31:20.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea0ff9700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 0x7f5e84055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea37fe700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 0x7f5e84055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5e9c06b950 con 0x7f5ea40b2920 2022-07-20T02:31:20.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.044+0000 7f5ea37fe700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 0x7f5e84055fc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f5e94019000 tx=0x7f5ea414ac90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.052+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e9c0429f0 con 0x7f5ea40b2920 2022-07-20T02:31:20.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.212+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ea4064540 con 0x7f5ea40b2920 2022-07-20T02:31:20.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.212+0000 7f5ea0ff9700 1 -- 172.21.15.26:0/1530131983 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f5e9c046010 con 0x7f5ea40b2920 2022-07-20T02:31:20.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:20.221 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5438,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2110992247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2110992247},{"type":"v1","addr":"172.21.15.120:6833","nonce":2110992247}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/349069969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":349069969},{"type":"v1","addr":"172.21.15.26:6835","nonce":349069969}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":6207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/1006992740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1006992740},{"type":"v1","addr":"172.21.15.26:6837","nonce":1006992740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-20T02:31:20.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 msgr2=0x7f5e84055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 0x7f5e84055fc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f5e94019000 tx=0x7f5ea414ac90).stop 2022-07-20T02:31:20.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 msgr2=0x7f5ea4064850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f5e9c001370 tx=0x7f5e9c001980).stop 2022-07-20T02:31:20.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 shutdown_connections 2022-07-20T02:31:20.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ea40b2920 0x7f5ea4064850 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e84053b00 0x7f5e84055fc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ea4067190 0x7f5ea4148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 --2- 172.21.15.26:0/1530131983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ea4061050 0x7f5ea40659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 >> 172.21.15.26:0/1530131983 conn(0x7f5ea40b60a0 msgr2=0x7f5ea4147cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:20.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 shutdown_connections 2022-07-20T02:31:20.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.216+0000 7f5ea9e27700 1 -- 172.21.15.26:0/1530131983 wait complete. 2022-07-20T02:31:20.231 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2022-07-20T02:31:20.233 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:31:20.234 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:31:20.235 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:31:20.235 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:31:20.236 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:31:20.236 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:31:20.237 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:31:20.237 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:31:20.238 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:31:20.238 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:31:20.239 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:31:20.239 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:31:20.253 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:31:20.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:31:20.256 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:31:20.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:31:20.259 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:31:20.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:31:20.268 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:31:20.269 DEBUG:teuthology.parallel:result is None 2022-07-20T02:31:20.297 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:31:20.298 DEBUG:teuthology.parallel:result is None 2022-07-20T02:31:20.300 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:31:20.300 DEBUG:teuthology.parallel:result is None 2022-07-20T02:31:20.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:31:20.340 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:31:20.366 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:31:20.378 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:31:20.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 Processor -- start 2022-07-20T02:31:20.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 -- start start 2022-07-20T02:31:20.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2e813fd70 0x7fc2e8141a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80b62d0 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e81402f0 con 0x7fc2e80ac1b0 2022-07-20T02:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e8144290 con 0x7fc2e813fd70 2022-07-20T02:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2e80b5ef0 0x7fc2e80abaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38346/0 (socket says 172.21.15.26:38346) 2022-07-20T02:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.408+0000 7fc2e659c700 1 -- 172.21.15.26:0/2160092339 learned_addr learned my addr 172.21.15.26:0/2160092339 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e6d9d700 1 --2- 172.21.15.26:0/2160092339 >> v2:172.21.15.120:3300/0 conn(0x7fc2e813fd70 0x7fc2e8141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862408366 0 0) 0x7fc2e80b62d0 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c8003260 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644597964 0 0) 0x7fc2c8003260 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 >> v2:172.21.15.120:3300/0 conn(0x7fc2e813fd70 msgr2=0x7fc2e8141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 --2- 172.21.15.26:0/2160092339 >> v2:172.21.15.120:3300/0 conn(0x7fc2e813fd70 0x7fc2e8141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 >> v1:172.21.15.120:6789/0 conn(0x7fc2e80ac1b0 legacy=0x7fc2e8141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8140590 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fc2e0001a00 con 0x7fc2e80b5ef0 2022-07-20T02:31:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 >> v1:172.21.15.26:6789/0 conn(0x7fc2e80b5ef0 legacy=0x7fc2e80abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 0x7fc2c8003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2c8004500 0x7fc2c8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8004e70 0x7fc2c800d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800de50 con 0x7fc2c8004500 2022-07-20T02:31:20.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800dfc0 con 0x7fc2c8003750 2022-07-20T02:31:20.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800e180 con 0x7fc2c8004e70 2022-07-20T02:31:20.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e5d9b700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2c8004500 0x7fc2c8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 0x7fc2c8003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e6d9d700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8004e70 0x7fc2c800d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 -- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8004e70 msgr2=0x7fc2c800d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8004e70 0x7fc2c800d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 -- 172.21.15.26:0/2160092339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2c8004500 msgr2=0x7fc2c8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:20.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2c8004500 0x7fc2c8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 -- 172.21.15.26:0/2160092339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c8003260 con 0x7fc2c8003750 2022-07-20T02:31:20.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e5d9b700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2c8004500 0x7fc2c8004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:20.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e6d9d700 1 -- 172.21.15.26:0/2160092339 reap_dead start 2022-07-20T02:31:20.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e659c700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 0x7fc2c8003b30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc2e0001710 tx=0x7fc2e80a9490).ready entity=mon.1 client_cookie=f8878795b192c46d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2e000cbe0 con 0x7fc2c8003750 2022-07-20T02:31:20.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000cd40 con 0x7fc2c8003750 2022-07-20T02:31:20.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e559a700 1 -- 172.21.15.26:0/2160092339 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2e0004b50 con 0x7fc2c8003750 2022-07-20T02:31:20.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 -- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 msgr2=0x7fc2c8003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 0x7fc2c8003b30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc2e0001710 tx=0x7fc2e80a9490).stop 2022-07-20T02:31:20.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 -- 172.21.15.26:0/2160092339 shutdown_connections 2022-07-20T02:31:20.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 --2- 172.21.15.26:0/2160092339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2c8003750 0x7fc2c8003b30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 -- 172.21.15.26:0/2160092339 >> 172.21.15.26:0/2160092339 conn(0x7fc2e8060310 msgr2=0x7fc2e80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:20.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 -- 172.21.15.26:0/2160092339 shutdown_connections 2022-07-20T02:31:20.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.412+0000 7fc2e759e700 1 -- 172.21.15.26:0/2160092339 wait complete. 2022-07-20T02:31:20.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 Processor -- start 2022-07-20T02:31:20.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- start start 2022-07-20T02:31:20.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8003750 0x7fc2e813f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 0x7fc2e813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e813d1b0 con 0x7fc2e813cd00 2022-07-20T02:31:20.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80673c0 con 0x7fc2e8062bd0 2022-07-20T02:31:20.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8063260 con 0x7fc2c8003750 2022-07-20T02:31:20.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e5d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 0x7fc2e813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39082/0 (socket says 172.21.15.26:39082) 2022-07-20T02:31:20.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 -- 172.21.15.26:0/435173456 learned_addr learned my addr 172.21.15.26:0/435173456 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:20.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e659c700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8003750 0x7fc2e813f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 -- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8003750 msgr2=0x7fc2e813f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8003750 0x7fc2e813f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 -- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 msgr2=0x7fc2e813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:20.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 0x7fc2e813c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:20.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 -- 172.21.15.26:0/435173456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8063500 con 0x7fc2e813cd00 2022-07-20T02:31:20.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e5d9b700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 0x7fc2e813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:20.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e6d9d700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fc2d800b000 tx=0x7fc2d8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2d8014430 con 0x7fc2e813cd00 2022-07-20T02:31:20.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8014590 con 0x7fc2e813cd00 2022-07-20T02:31:20.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2d8014930 con 0x7fc2e813cd00 2022-07-20T02:31:20.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8066f10 con 0x7fc2e813cd00 2022-07-20T02:31:20.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8151db0 con 0x7fc2e813cd00 2022-07-20T02:31:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.416+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e8063690 con 0x7fc2e813cd00 2022-07-20T02:31:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.420+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2d801c030 con 0x7fc2e813cd00 2022-07-20T02:31:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.420+0000 7fc2df7fe700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 0x7fc2cc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.420+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fc2d806b7d0 con 0x7fc2e813cd00 2022-07-20T02:31:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.420+0000 7fc2e659c700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 0x7fc2cc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:20.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.420+0000 7fc2e659c700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 0x7fc2cc055e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc2e001d000 tx=0x7fc2e0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:20.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.424+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2d8042760 con 0x7fc2e813cd00 2022-07-20T02:31:20.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:20.564+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc2e8140830 con 0x7fc2e813cd00 2022-07-20T02:31:21.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.472+0000 7fc2df7fe700 1 -- 172.21.15.26:0/435173456 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v72) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc2d8045d80 con 0x7fc2e813cd00 2022-07-20T02:31:21.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 msgr2=0x7fc2cc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 0x7fc2cc055e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc2e001d000 tx=0x7fc2e0015000).stop 2022-07-20T02:31:21.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 msgr2=0x7fc2e81518c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fc2d800b000 tx=0x7fc2d8008040).stop 2022-07-20T02:31:21.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 shutdown_connections 2022-07-20T02:31:21.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e813cd00 0x7fc2e81518c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2cc053950 0x7fc2cc055e10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e8062bd0 0x7fc2e813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 --2- 172.21.15.26:0/435173456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2c8003750 0x7fc2e813f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.476+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 >> 172.21.15.26:0/435173456 conn(0x7fc2e8060310 msgr2=0x7fc2e80ae5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:21.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.480+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 shutdown_connections 2022-07-20T02:31:21.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.480+0000 7fc2e759e700 1 -- 172.21.15.26:0/435173456 wait complete. 2022-07-20T02:31:21.500 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:31:21.501 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:31:21.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:31:21.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 Processor -- start 2022-07-20T02:31:21.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 -- start start 2022-07-20T02:31:21.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 0x7fa5001474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 0x7fa5001479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000b62d0 con 0x7fa5000a8c40 2022-07-20T02:31:21.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa500147eb0 con 0x7fa5000a7fc0 2022-07-20T02:31:21.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50014aac0 con 0x7fa5000b5ef0 2022-07-20T02:31:21.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4ff7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa5000a8c40 0x7fa5000aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38372/0 (socket says 172.21.15.26:38372) 2022-07-20T02:31:21.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4ff7fe700 1 -- 172.21.15.26:0/4212378021 learned_addr learned my addr 172.21.15.26:0/4212378021 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:21.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 --2- 172.21.15.26:0/4212378021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 0x7fa5001474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fe7fc700 1 --2- 172.21.15.26:0/4212378021 >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 0x7fa5001479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fdffb700 1 -- 172.21.15.26:0/4212378021 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257047921 0 0) 0x7fa5000b62d0 con 0x7fa5000a8c40 2022-07-20T02:31:21.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 -- 172.21.15.26:0/4212378021 >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 msgr2=0x7fa5001479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:21.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 --2- 172.21.15.26:0/4212378021 >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 0x7fa5001479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 -- 172.21.15.26:0/4212378021 >> v1:172.21.15.26:6789/0 conn(0x7fa5000a8c40 legacy=0x7fa5000aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 -- 172.21.15.26:0/4212378021 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50014ad60 con 0x7fa5000b5ef0 2022-07-20T02:31:21.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fe7fc700 1 --2- 172.21.15.26:0/4212378021 >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 0x7fa5001479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:21.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4feffd700 1 --2- 172.21.15.26:0/4212378021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 0x7fa5001474d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa4e8002f70 tx=0x7fa4e8006df0).ready entity=mon.2 client_cookie=fdece1121da95d73 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:21.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fdffb700 1 -- 172.21.15.26:0/4212378021 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4e800d570 con 0x7fa5000b5ef0 2022-07-20T02:31:21.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fdffb700 1 -- 172.21.15.26:0/4212378021 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8000e90 con 0x7fa5000b5ef0 2022-07-20T02:31:21.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.628+0000 7fa4fdffb700 1 -- 172.21.15.26:0/4212378021 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4e800d8c0 con 0x7fa5000b5ef0 2022-07-20T02:31:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- 172.21.15.26:0/4212378021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 msgr2=0x7fa5001474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 --2- 172.21.15.26:0/4212378021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 0x7fa5001474d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa4e8002f70 tx=0x7fa4e8006df0).stop 2022-07-20T02:31:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- 172.21.15.26:0/4212378021 shutdown_connections 2022-07-20T02:31:21.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 --2- 172.21.15.26:0/4212378021 >> v2:172.21.15.120:3300/0 conn(0x7fa5000a7fc0 0x7fa5001479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 --2- 172.21.15.26:0/4212378021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000b5ef0 0x7fa5001474d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- 172.21.15.26:0/4212378021 >> 172.21.15.26:0/4212378021 conn(0x7fa500060310 msgr2=0x7fa5000b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:21.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- 172.21.15.26:0/4212378021 shutdown_connections 2022-07-20T02:31:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- 172.21.15.26:0/4212378021 wait complete. 2022-07-20T02:31:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 Processor -- start 2022-07-20T02:31:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.632+0000 7fa4fffff700 1 -- start start 2022-07-20T02:31:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 0x7fa50014db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000a8c40 0x7fa50014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5001497c0 con 0x7fa5000b5ef0 2022-07-20T02:31:21.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa500149a60 con 0x7fa5000a7fc0 2022-07-20T02:31:21.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa500149d00 con 0x7fa5000a8c40 2022-07-20T02:31:21.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39098/0 (socket says 172.21.15.26:39098) 2022-07-20T02:31:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 -- 172.21.15.26:0/3169136518 learned_addr learned my addr 172.21.15.26:0/3169136518 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4feffd700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 0x7fa50014db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fe7fc700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000a8c40 0x7fa50014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 -- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000a8c40 msgr2=0x7fa50014e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000a8c40 0x7fa50014e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 -- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 msgr2=0x7fa50014db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 0x7fa50014db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 -- 172.21.15.26:0/3169136518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8009470 con 0x7fa5000b5ef0 2022-07-20T02:31:21.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4feffd700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 0x7fa50014db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:21.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4ff7fe700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fa4f0000fe0 tx=0x7fa4f0007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:21.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4f00070d0 con 0x7fa5000b5ef0 2022-07-20T02:31:21.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f0007230 con 0x7fa5000b5ef0 2022-07-20T02:31:21.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e800b180 con 0x7fa5000b5ef0 2022-07-20T02:31:21.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4f0005610 con 0x7fa5000b5ef0 2022-07-20T02:31:21.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa500064b40 con 0x7fa5000b5ef0 2022-07-20T02:31:21.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.636+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50014b000 con 0x7fa5000b5ef0 2022-07-20T02:31:21.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.640+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa4f001a060 con 0x7fa5000b5ef0 2022-07-20T02:31:21.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.640+0000 7fa4f7fff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 0x7fa4e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:21.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.640+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa4f006b510 con 0x7fa5000b5ef0 2022-07-20T02:31:21.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.640+0000 7fa4feffd700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 0x7fa4e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:21.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.640+0000 7fa4feffd700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 0x7fa4e4055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fa4e8018000 tx=0x7fa4e8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:21.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.648+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4f0042610 con 0x7fa5000b5ef0 2022-07-20T02:31:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.812+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa5000574e0 con 0x7fa5000b5ef0 2022-07-20T02:31:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.812+0000 7fa4f7fff700 1 -- 172.21.15.26:0/3169136518 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fa4f0018070 con 0x7fa5000b5ef0 2022-07-20T02:31:21.816 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:21.816 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:31:21.817 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:31:21.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:31:21.820 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:31:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:31:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:31:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:31:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:31:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:31:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:31:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:31:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.834 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:31:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:31:21.835 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.838 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:31:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:31:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.845 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:31:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:31:21.846 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.847 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.850 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:31:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:31:21.851 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.854 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:31:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:31:21.855 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.856 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.862 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:31:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:31:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:31:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:31:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.865 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:31:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:31:21.867 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.868 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.869 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:31:21.869 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:31:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:31:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:31:21.870 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:31:21.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.871 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:31:21.871 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:31:21.872 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:31:21.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:31:21.872 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:31:21.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.873 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:31:21.873 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:31:21.874 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.874 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:31:21.874 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:31:21.874 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:31:21.875 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:31:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:31:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:31:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:31:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:31:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:31:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:31:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:31:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:31:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:31:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:31:21.880 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:31:21.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 msgr2=0x7fa4e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 0x7fa4e4055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fa4e8018000 tx=0x7fa4e8014000).stop 2022-07-20T02:31:21.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 msgr2=0x7fa50014d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fa4f0000fe0 tx=0x7fa4f0007aa0).stop 2022-07-20T02:31:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 shutdown_connections 2022-07-20T02:31:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5000b5ef0 0x7fa50014d350 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa4e4053a70 0x7fa4e4055f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5000a8c40 0x7fa50014e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 --2- 172.21.15.26:0/3169136518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5000a7fc0 0x7fa50014db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 >> 172.21.15.26:0/3169136518 conn(0x7fa500060310 msgr2=0x7fa500148830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 shutdown_connections 2022-07-20T02:31:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.816+0000 7fa4fffff700 1 -- 172.21.15.26:0/3169136518 wait complete. 2022-07-20T02:31:21.886 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:22.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.992+0000 7f6d84996700 1 Processor -- start 2022-07-20T02:31:22.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.992+0000 7f6d84996700 1 -- start start 2022-07-20T02:31:22.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6d8005cfd0 0x7f6d80143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6d80060f00 0x7f6d8013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d801474e0 con 0x7f6d80147000 2022-07-20T02:31:22.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d800b4e40 con 0x7f6d80060f00 2022-07-20T02:31:22.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8005d3b0 con 0x7f6d8005cfd0 2022-07-20T02:31:22.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6d80060f00 0x7f6d8013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6d80060f00 0x7f6d8013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39106/0 (socket says 172.21.15.26:39106) 2022-07-20T02:31:22.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 -- 172.21.15.26:0/1111696582 learned_addr learned my addr 172.21.15.26:0/1111696582 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:22.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7ed9d700 1 --2- 172.21.15.26:0/1111696582 >> v2:172.21.15.120:3300/0 conn(0x7f6d8005cfd0 0x7f6d80143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 -- 172.21.15.26:0/1111696582 >> v2:172.21.15.120:3300/0 conn(0x7f6d8005cfd0 msgr2=0x7f6d80143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:22.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 --2- 172.21.15.26:0/1111696582 >> v2:172.21.15.120:3300/0 conn(0x7f6d8005cfd0 0x7f6d80143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 -- 172.21.15.26:0/1111696582 >> v1:172.21.15.120:6789/0 conn(0x7f6d80147000 legacy=0x7f6d800b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 -- 172.21.15.26:0/1111696582 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d801431e0 con 0x7f6d80060f00 2022-07-20T02:31:22.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7f59e700 1 --2- 172.21.15.26:0/1111696582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d80060f00 0x7f6d8013c9b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f6d780032f0 tx=0x7f6d78009290).ready entity=mon.0 client_cookie=9575bcd573ac3b11 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7dd9b700 1 -- 172.21.15.26:0/1111696582 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7800b870 con 0x7f6d80060f00 2022-07-20T02:31:22.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7dd9b700 1 -- 172.21.15.26:0/1111696582 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7800b9d0 con 0x7f6d80060f00 2022-07-20T02:31:22.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d7dd9b700 1 -- 172.21.15.26:0/1111696582 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7800bb40 con 0x7f6d80060f00 2022-07-20T02:31:22.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- 172.21.15.26:0/1111696582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d80060f00 msgr2=0x7f6d8013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 --2- 172.21.15.26:0/1111696582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d80060f00 0x7f6d8013c9b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f6d780032f0 tx=0x7f6d78009290).stop 2022-07-20T02:31:22.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- 172.21.15.26:0/1111696582 shutdown_connections 2022-07-20T02:31:22.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 --2- 172.21.15.26:0/1111696582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d80060f00 0x7f6d8013c9b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 --2- 172.21.15.26:0/1111696582 >> v2:172.21.15.120:3300/0 conn(0x7f6d8005cfd0 0x7f6d80143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- 172.21.15.26:0/1111696582 >> 172.21.15.26:0/1111696582 conn(0x7f6d800620c0 msgr2=0x7f6d8005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- 172.21.15.26:0/1111696582 shutdown_connections 2022-07-20T02:31:22.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:21.996+0000 7f6d84996700 1 -- 172.21.15.26:0/1111696582 wait complete. 2022-07-20T02:31:22.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 Processor -- start 2022-07-20T02:31:22.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- start start 2022-07-20T02:31:22.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d80060f00 0x7f6d800b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80147000 0x7f6d800b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80142ea0 con 0x7f6d800b7b10 2022-07-20T02:31:22.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80152d80 con 0x7f6d80147000 2022-07-20T02:31:22.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b3580 con 0x7f6d80060f00 2022-07-20T02:31:22.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39112/0 (socket says 172.21.15.26:39112) 2022-07-20T02:31:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 -- 172.21.15.26:0/3203176403 learned_addr learned my addr 172.21.15.26:0/3203176403 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7ed9d700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d80060f00 0x7f6d800b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7e59c700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80147000 0x7f6d800b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 -- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d80060f00 msgr2=0x7f6d800b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d80060f00 0x7f6d800b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 -- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80147000 msgr2=0x7f6d800b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80147000 0x7f6d800b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 -- 172.21.15.26:0/3203176403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78009530 con 0x7f6d800b7b10 2022-07-20T02:31:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d7f59e700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f6d780032f0 tx=0x7f6d78016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7800b870 con 0x7f6d800b7b10 2022-07-20T02:31:22.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78021440 con 0x7f6d800b7b10 2022-07-20T02:31:22.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d80140f00 con 0x7f6d800b7b10 2022-07-20T02:31:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d780217a0 con 0x7f6d800b7b10 2022-07-20T02:31:22.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d80152f90 con 0x7f6d800b7b10 2022-07-20T02:31:22.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.000+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d80153150 con 0x7f6d800b7b10 2022-07-20T02:31:22.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.004+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d7800f070 con 0x7f6d800b7b10 2022-07-20T02:31:22.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.004+0000 7f6d77fff700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 0x7f6d64055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.004+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f6d7806d510 con 0x7f6d800b7b10 2022-07-20T02:31:22.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.004+0000 7f6d7ed9d700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 0x7f6d64055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.004+0000 7f6d7ed9d700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 0x7f6d64055f30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6d6800b000 tx=0x7f6d68006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.008+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d78044610 con 0x7f6d800b7b10 2022-07-20T02:31:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.128+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d80143480 con 0x7f6d800b7b10 2022-07-20T02:31:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d77fff700 1 -- 172.21.15.26:0/3203176403 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+13463 (secure 0 0 0) 0x7f6d78015020 con 0x7f6d800b7b10 2022-07-20T02:31:22.132 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:22.133 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:21.466012+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:22.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 msgr2=0x7f6d64055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 0x7f6d64055f30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6d6800b000 tx=0x7f6d68006df0).stop 2022-07-20T02:31:22.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 msgr2=0x7f6d800b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f6d780032f0 tx=0x7f6d78016000).stop 2022-07-20T02:31:22.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 shutdown_connections 2022-07-20T02:31:22.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d800b7b10 0x7f6d800b3090 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d64053a70 0x7f6d64055f30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80147000 0x7f6d800b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 --2- 172.21.15.26:0/3203176403 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d80060f00 0x7f6d800b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 >> 172.21.15.26:0/3203176403 conn(0x7f6d8005cfd0 msgr2=0x7f6d80144320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 shutdown_connections 2022-07-20T02:31:22.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.132+0000 7f6d84996700 1 -- 172.21.15.26:0/3203176403 wait complete. 2022-07-20T02:31:22.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:31:22.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:31:22.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 Processor -- start 2022-07-20T02:31:22.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 -- start start 2022-07-20T02:31:22.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 0x7fa9d80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9d80b3380 0x7fa9d814b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d8064880 con 0x7fa9d8064450 2022-07-20T02:31:22.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d80b85f0 con 0x7fa9d80b3380 2022-07-20T02:31:22.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b80b0 con 0x7fa9d80670c0 2022-07-20T02:31:22.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d5d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9d8064450 0x7fa9d8064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38400/0 (socket says 172.21.15.26:38400) 2022-07-20T02:31:22.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d5d9b700 1 -- 172.21.15.26:0/763280051 learned_addr learned my addr 172.21.15.26:0/763280051 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:22.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.272+0000 7fa9d659c700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 0x7fa9d80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 --2- 172.21.15.26:0/763280051 >> v2:172.21.15.120:3300/0 conn(0x7fa9d80b3380 0x7fa9d814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d559a700 1 -- 172.21.15.26:0/763280051 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054973452 0 0) 0x7fa9d8064880 con 0x7fa9d8064450 2022-07-20T02:31:22.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d559a700 1 -- 172.21.15.26:0/763280051 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b8003260 con 0x7fa9d8064450 2022-07-20T02:31:22.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 -- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 msgr2=0x7fa9d80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 0x7fa9d80b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 -- 172.21.15.26:0/763280051 >> v1:172.21.15.26:6789/0 conn(0x7fa9d8064450 legacy=0x7fa9d8064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 -- 172.21.15.26:0/763280051 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80b8350 con 0x7fa9d80b3380 2022-07-20T02:31:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d659c700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 0x7fa9d80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d6d9d700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80b3380 0x7fa9d814b840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa9d00032f0 tx=0x7fa9d00091d0).ready entity=mon.1 client_cookie=80ac3a1ad386aab0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d559a700 1 -- 172.21.15.26:0/763280051 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d000b600 con 0x7fa9d80b3380 2022-07-20T02:31:22.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d559a700 1 -- 172.21.15.26:0/763280051 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d000b760 con 0x7fa9d80b3380 2022-07-20T02:31:22.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d559a700 1 -- 172.21.15.26:0/763280051 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d000b8d0 con 0x7fa9d80b3380 2022-07-20T02:31:22.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 -- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80b3380 msgr2=0x7fa9d814b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80b3380 0x7fa9d814b840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa9d00032f0 tx=0x7fa9d00091d0).stop 2022-07-20T02:31:22.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 -- 172.21.15.26:0/763280051 shutdown_connections 2022-07-20T02:31:22.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80b3380 0x7fa9d814b840 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 --2- 172.21.15.26:0/763280051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d80670c0 0x7fa9d80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 -- 172.21.15.26:0/763280051 >> 172.21.15.26:0/763280051 conn(0x7fa9d805f840 msgr2=0x7fa9d805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 -- 172.21.15.26:0/763280051 shutdown_connections 2022-07-20T02:31:22.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.276+0000 7fa9d759e700 1 -- 172.21.15.26:0/763280051 wait complete. 2022-07-20T02:31:22.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 Processor -- start 2022-07-20T02:31:22.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 -- start start 2022-07-20T02:31:22.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 0x7fa9d8142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d8153820 0x7fa9d8153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8143630 con 0x7fa9d80afe00 2022-07-20T02:31:22.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80af910 con 0x7fa9d80670c0 2022-07-20T02:31:22.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b0e20 con 0x7fa9d8153820 2022-07-20T02:31:22.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39122/0 (socket says 172.21.15.26:39122) 2022-07-20T02:31:22.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 -- 172.21.15.26:0/226110868 learned_addr learned my addr 172.21.15.26:0/226110868 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:22.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d6d9d700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d8153820 0x7fa9d8153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d659c700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 0x7fa9d8142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 -- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d8153820 msgr2=0x7fa9d8153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d8153820 0x7fa9d8153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 -- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 msgr2=0x7fa9d8142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 0x7fa9d8142c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 -- 172.21.15.26:0/226110868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c800b180 con 0x7fa9d80afe00 2022-07-20T02:31:22.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d659c700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 0x7fa9d8142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:22.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.280+0000 7fa9d5d9b700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fa9c0001110 tx=0x7fa9c0007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c00070d0 con 0x7fa9d80afe00 2022-07-20T02:31:22.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c0007230 con 0x7fa9d80afe00 2022-07-20T02:31:22.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d0009470 con 0x7fa9d80afe00 2022-07-20T02:31:22.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c0005610 con 0x7fa9d80afe00 2022-07-20T02:31:22.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d8154190 con 0x7fa9d80afe00 2022-07-20T02:31:22.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d8064ab0 con 0x7fa9d80afe00 2022-07-20T02:31:22.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9c0010070 con 0x7fa9d80afe00 2022-07-20T02:31:22.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 0x7fa9bc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa9c006b500 con 0x7fa9d80afe00 2022-07-20T02:31:22.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9d659c700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 0x7fa9bc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.284+0000 7fa9d659c700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 0x7fa9bc055ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa9c800b000 tx=0x7fa9c8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.292+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9c0042500 con 0x7fa9d80afe00 2022-07-20T02:31:22.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.416+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa9d80574e0 con 0x7fa9d80afe00 2022-07-20T02:31:22.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/226110868 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa9c0045b20 con 0x7fa9d80afe00 2022-07-20T02:31:22.420 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:31:22.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 msgr2=0x7fa9bc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 0x7fa9bc055ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa9c800b000 tx=0x7fa9c8008040).stop 2022-07-20T02:31:22.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 msgr2=0x7fa9d8143140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fa9c0001110 tx=0x7fa9c0007b40).stop 2022-07-20T02:31:22.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 shutdown_connections 2022-07-20T02:31:22.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d80afe00 0x7fa9d8143140 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a10 0x7fa9bc055ed0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d8153820 0x7fa9d8153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 --2- 172.21.15.26:0/226110868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d80670c0 0x7fa9d8142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 >> 172.21.15.26:0/226110868 conn(0x7fa9d805f840 msgr2=0x7fa9d805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 shutdown_connections 2022-07-20T02:31:22.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.420+0000 7fa9d759e700 1 -- 172.21.15.26:0/226110868 wait complete. 2022-07-20T02:31:22.435 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:31:22.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 Processor -- start 2022-07-20T02:31:22.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 -- start start 2022-07-20T02:31:22.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 0x7ff658147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 0x7ff658147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65805d4d0 con 0x7ff6580b1200 2022-07-20T02:31:22.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6580a9ff0 con 0x7ff65805c720 2022-07-20T02:31:22.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65c994700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff658067e60 con 0x7ff6580afe40 2022-07-20T02:31:22.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff656d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 0x7ff658147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65759e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff6580b1200 0x7ff6580abbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38412/0 (socket says 172.21.15.26:38412) 2022-07-20T02:31:22.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65759e700 1 -- 172.21.15.26:0/4160943284 learned_addr learned my addr 172.21.15.26:0/4160943284 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:22.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 --2- 172.21.15.26:0/4160943284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 0x7ff658147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160100319 0 0) 0x7ff65805d4d0 con 0x7ff6580b1200 2022-07-20T02:31:22.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638003260 con 0x7ff6580b1200 2022-07-20T02:31:22.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 -- 172.21.15.26:0/4160943284 >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 msgr2=0x7ff658147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:22.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 --2- 172.21.15.26:0/4160943284 >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 0x7ff658147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 -- 172.21.15.26:0/4160943284 >> v1:172.21.15.26:6789/0 conn(0x7ff6580b1200 legacy=0x7ff6580abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1775107471 0 0) 0x7ff638003260 con 0x7ff6580b1200 2022-07-20T02:31:22.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 -- 172.21.15.26:0/4160943284 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658067bc0 con 0x7ff6580afe40 2022-07-20T02:31:22.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff656d9d700 1 --2- 172.21.15.26:0/4160943284 >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 0x7ff658147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:22.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff65659c700 1 --2- 172.21.15.26:0/4160943284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 0x7ff658147a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff650001800 tx=0x7ff6480091d0).ready entity=mon.2 client_cookie=26091d84a8be001d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff650001660 con 0x7ff6580afe40 2022-07-20T02:31:22.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64800f510 con 0x7ff6580afe40 2022-07-20T02:31:22.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.556+0000 7ff655d9b700 1 -- 172.21.15.26:0/4160943284 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64800f680 con 0x7ff6580afe40 2022-07-20T02:31:22.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- 172.21.15.26:0/4160943284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 msgr2=0x7ff658147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 --2- 172.21.15.26:0/4160943284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 0x7ff658147a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff650001800 tx=0x7ff6480091d0).stop 2022-07-20T02:31:22.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- 172.21.15.26:0/4160943284 shutdown_connections 2022-07-20T02:31:22.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 --2- 172.21.15.26:0/4160943284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580afe40 0x7ff658147a40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 --2- 172.21.15.26:0/4160943284 >> v2:172.21.15.26:3300/0 conn(0x7ff65805c720 0x7ff658147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- 172.21.15.26:0/4160943284 >> 172.21.15.26:0/4160943284 conn(0x7ff658060310 msgr2=0x7ff6580b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- 172.21.15.26:0/4160943284 shutdown_connections 2022-07-20T02:31:22.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- 172.21.15.26:0/4160943284 wait complete. 2022-07-20T02:31:22.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 Processor -- start 2022-07-20T02:31:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.560+0000 7ff65c994700 1 -- start start 2022-07-20T02:31:22.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6580afe40 0x7ff658156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580b1200 0x7ff6581572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6580ad5c0 con 0x7ff65805c720 2022-07-20T02:31:22.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658156a30 con 0x7ff6580afe40 2022-07-20T02:31:22.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff658063c60 con 0x7ff6580b1200 2022-07-20T02:31:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39140/0 (socket says 172.21.15.26:39140) 2022-07-20T02:31:22.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580b1200 0x7ff6581572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 -- 172.21.15.26:0/1497305214 learned_addr learned my addr 172.21.15.26:0/1497305214 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:22.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65659c700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6580afe40 0x7ff658156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 -- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580b1200 msgr2=0x7ff6581572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580b1200 0x7ff6581572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 -- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6580afe40 msgr2=0x7ff658156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6580afe40 0x7ff658156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 -- 172.21.15.26:0/1497305214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64000b180 con 0x7ff65805c720 2022-07-20T02:31:22.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff656d9d700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7ff658062750 tx=0x7ff640006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff640010590 con 0x7ff65805c720 2022-07-20T02:31:22.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6400106f0 con 0x7ff65805c720 2022-07-20T02:31:22.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff640010a50 con 0x7ff65805c720 2022-07-20T02:31:22.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff648009470 con 0x7ff65805c720 2022-07-20T02:31:22.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.564+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6580ad320 con 0x7ff65805c720 2022-07-20T02:31:22.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65815a0e0 con 0x7ff65805c720 2022-07-20T02:31:22.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff640010bb0 con 0x7ff65805c720 2022-07-20T02:31:22.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff64ffff700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 0x7ff63c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff64006c220 con 0x7ff65805c720 2022-07-20T02:31:22.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff65659c700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 0x7ff63c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.568+0000 7ff65659c700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 0x7ff63c055ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff64801b000 tx=0x7ff648016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.572+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff640069e90 con 0x7ff65805c720 2022-07-20T02:31:22.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff6580aa250 con 0x7ff65805c720 2022-07-20T02:31:22.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff64ffff700 1 -- 172.21.15.26:0/1497305214 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff640043280 con 0x7ff65805c720 2022-07-20T02:31:22.700 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:22.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 msgr2=0x7ff63c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 0x7ff63c055ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff64801b000 tx=0x7ff648016000).stop 2022-07-20T02:31:22.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 msgr2=0x7ff65814d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7ff658062750 tx=0x7ff640006df0).stop 2022-07-20T02:31:22.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 shutdown_connections 2022-07-20T02:31:22.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff63c0539e0 0x7ff63c055ea0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6580b1200 0x7ff6581572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.700+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6580afe40 0x7ff658156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.704+0000 7ff65c994700 1 --2- 172.21.15.26:0/1497305214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff65805c720 0x7ff65814d4c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.704+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 >> 172.21.15.26:0/1497305214 conn(0x7ff658060310 msgr2=0x7ff65814a430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.704+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 shutdown_connections 2022-07-20T02:31:22.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.704+0000 7ff65c994700 1 -- 172.21.15.26:0/1497305214 wait complete. 2022-07-20T02:31:22.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:31:22.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 Processor -- start 2022-07-20T02:31:22.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 -- start start 2022-07-20T02:31:22.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6408147000 0x7f6408143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64081474e0 con 0x7f640805cfd0 2022-07-20T02:31:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64080b4e40 con 0x7f6408060f00 2022-07-20T02:31:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f640d41f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640805d3b0 con 0x7f6408147000 2022-07-20T02:31:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f64077fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6408147000 0x7f6408143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f6407fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f640805cfd0 0x7f64080b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38424/0 (socket says 172.21.15.26:38424) 2022-07-20T02:31:22.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f6407fff700 1 -- 172.21.15.26:0/953403002 learned_addr learned my addr 172.21.15.26:0/953403002 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f64077fe700 1 -- 172.21.15.26:0/953403002 >> v1:172.21.15.120:6789/0 conn(0x7f6408060f00 legacy=0x7f640813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f64077fe700 1 -- 172.21.15.26:0/953403002 >> v1:172.21.15.26:6789/0 conn(0x7f640805cfd0 legacy=0x7f64080b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f64077fe700 1 -- 172.21.15.26:0/953403002 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64081431e0 con 0x7f6408147000 2022-07-20T02:31:22.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f6406ffd700 1 -- 172.21.15.26:0/953403002 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488456055 0 0) 0x7f64080b4e40 con 0x7f6408060f00 2022-07-20T02:31:22.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.824+0000 7f64077fe700 1 --2- 172.21.15.26:0/953403002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6408147000 0x7f6408143bc0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f63f8002f70 tx=0x7f63f8006df0).ready entity=mon.0 client_cookie=42bcd551ebbfa91c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f6406ffd700 1 -- 172.21.15.26:0/953403002 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63f800d610 con 0x7f6408147000 2022-07-20T02:31:22.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f6406ffd700 1 -- 172.21.15.26:0/953403002 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8000e90 con 0x7f6408147000 2022-07-20T02:31:22.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f6406ffd700 1 -- 172.21.15.26:0/953403002 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63f800d960 con 0x7f6408147000 2022-07-20T02:31:22.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- 172.21.15.26:0/953403002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6408147000 msgr2=0x7f6408143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 --2- 172.21.15.26:0/953403002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6408147000 0x7f6408143bc0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f63f8002f70 tx=0x7f63f8006df0).stop 2022-07-20T02:31:22.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- 172.21.15.26:0/953403002 shutdown_connections 2022-07-20T02:31:22.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 --2- 172.21.15.26:0/953403002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6408147000 0x7f6408143bc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- 172.21.15.26:0/953403002 >> 172.21.15.26:0/953403002 conn(0x7f64080620c0 msgr2=0x7f640805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- 172.21.15.26:0/953403002 shutdown_connections 2022-07-20T02:31:22.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- 172.21.15.26:0/953403002 wait complete. 2022-07-20T02:31:22.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 Processor -- start 2022-07-20T02:31:22.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- start start 2022-07-20T02:31:22.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 0x7f6408152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 0x7f64080b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64081522a0 con 0x7f64080aff20 2022-07-20T02:31:22.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408152490 con 0x7f6408060f00 2022-07-20T02:31:22.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640d41f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6408152730 con 0x7f6408147000 2022-07-20T02:31:22.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640cc1e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640cc1e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39154/0 (socket says 172.21.15.26:39154) 2022-07-20T02:31:22.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.828+0000 7f640cc1e700 1 -- 172.21.15.26:0/3618623166 learned_addr learned my addr 172.21.15.26:0/3618623166 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:22.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64077fe700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 0x7f64080b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f6407fff700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 0x7f6408152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 -- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 msgr2=0x7f64080b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 0x7f64080b7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 -- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 msgr2=0x7f6408152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 0x7f6408152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 -- 172.21.15.26:0/3618623166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f800b180 con 0x7f64080aff20 2022-07-20T02:31:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f6407fff700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 0x7f6408152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64077fe700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 0x7f64080b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640cc1e700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f6400001250 tx=0x7f6400007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6400010510 con 0x7f64080aff20 2022-07-20T02:31:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64081529d0 con 0x7f64080aff20 2022-07-20T02:31:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400010670 con 0x7f64080aff20 2022-07-20T02:31:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64000107e0 con 0x7f64080aff20 2022-07-20T02:31:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64080af1c0 con 0x7f64080aff20 2022-07-20T02:31:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6408152b60 con 0x7f64080aff20 2022-07-20T02:31:22.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6400010070 con 0x7f64080aff20 2022-07-20T02:31:22.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 0x7f63e8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f640006bb20 con 0x7f64080aff20 2022-07-20T02:31:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.832+0000 7f6407fff700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 0x7f63e8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:22.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.836+0000 7f6407fff700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 0x7f63e8055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f63fc00b000 tx=0x7f63fc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:22.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.840+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6400042b80 con 0x7f64080aff20 2022-07-20T02:31:22.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.968+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6408143480 con 0x7f64080aff20 2022-07-20T02:31:22.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.968+0000 7f64057fa700 1 -- 172.21.15.26:0/3618623166 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) v1 ==== 84+0+143 (secure 0 0 0) 0x7f64000461a0 con 0x7f64080aff20 2022-07-20T02:31:22.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:22.984 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:31:22.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:31:22.986 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:31:22.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:31:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:31:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:31:22.988 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:31:22.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:31:22.989 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:31:22.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 msgr2=0x7f63e8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 0x7f63e8055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f63fc00b000 tx=0x7f63fc009040).stop 2022-07-20T02:31:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 msgr2=0x7f64080b83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f6400001250 tx=0x7f6400007ed0).stop 2022-07-20T02:31:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 shutdown_connections 2022-07-20T02:31:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64080aff20 0x7f64080b83d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f63e8053b00 0x7f63e8055fc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6408147000 0x7f64080b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 --2- 172.21.15.26:0/3618623166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6408060f00 0x7f6408152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 >> 172.21.15.26:0/3618623166 conn(0x7f640805cfd0 msgr2=0x7f6408142a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 shutdown_connections 2022-07-20T02:31:23.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:22.972+0000 7f640d41f700 1 -- 172.21.15.26:0/3618623166 wait complete. 2022-07-20T02:31:23.001 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:31:23.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:31:23.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.120+0000 7f33b6e0f700 1 Processor -- start 2022-07-20T02:31:23.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.120+0000 7f33b6e0f700 1 -- start start 2022-07-20T02:31:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f33b00b7d80 0x7f33b00b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 0x7f33b00b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b00b2e40 con 0x7f33b0064220 2022-07-20T02:31:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b00617a0 con 0x7f33b00b7d80 2022-07-20T02:31:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b0060b20 con 0x7f33b00b3060 2022-07-20T02:31:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f33b00b7d80 0x7f33b00b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b660e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f33b0064220 0x7f33b0146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38440/0 (socket says 172.21.15.26:38440) 2022-07-20T02:31:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b660e700 1 -- 172.21.15.26:0/3944969938 learned_addr learned my addr 172.21.15.26:0/3944969938 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b560c700 1 --2- 172.21.15.26:0/3944969938 >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 0x7f33b00b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b4e0b700 1 -- 172.21.15.26:0/3944969938 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915321668 0 0) 0x7f33b00b2e40 con 0x7f33b0064220 2022-07-20T02:31:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b4e0b700 1 -- 172.21.15.26:0/3944969938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f339c003260 con 0x7f33b0064220 2022-07-20T02:31:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 -- 172.21.15.26:0/3944969938 >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 msgr2=0x7f33b00b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 --2- 172.21.15.26:0/3944969938 >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 0x7f33b00b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 -- 172.21.15.26:0/3944969938 >> v1:172.21.15.26:6789/0 conn(0x7f33b0064220 legacy=0x7f33b0146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 -- 172.21.15.26:0/3944969938 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0060dc0 con 0x7f33b00b7d80 2022-07-20T02:31:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b560c700 1 --2- 172.21.15.26:0/3944969938 >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 0x7f33b00b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:23.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b5e0d700 1 --2- 172.21.15.26:0/3944969938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b00b4020 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f33a4002f70 tx=0x7f33a4006df0).ready entity=mon.0 client_cookie=fc6e72fe7cc94e22 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b4e0b700 1 -- 172.21.15.26:0/3944969938 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a400d570 con 0x7f33b00b7d80 2022-07-20T02:31:23.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b4e0b700 1 -- 172.21.15.26:0/3944969938 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a4000e90 con 0x7f33b00b7d80 2022-07-20T02:31:23.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b4e0b700 1 -- 172.21.15.26:0/3944969938 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a400d8c0 con 0x7f33b00b7d80 2022-07-20T02:31:23.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3944969938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 msgr2=0x7f33b00b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3944969938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b00b4020 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f33a4002f70 tx=0x7f33a4006df0).stop 2022-07-20T02:31:23.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3944969938 shutdown_connections 2022-07-20T02:31:23.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3944969938 >> v2:172.21.15.120:3300/0 conn(0x7f33b00b3060 0x7f33b00b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3944969938 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b00b4020 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3944969938 >> 172.21.15.26:0/3944969938 conn(0x7f33b005f840 msgr2=0x7f33b005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:23.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3944969938 shutdown_connections 2022-07-20T02:31:23.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.124+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3944969938 wait complete. 2022-07-20T02:31:23.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 Processor -- start 2022-07-20T02:31:23.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- start start 2022-07-20T02:31:23.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b0064220 0x7f33b01400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b00b3060 0x7f33b013ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0145c60 con 0x7f33b00b7d80 2022-07-20T02:31:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0148c60 con 0x7f33b00b3060 2022-07-20T02:31:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b01474e0 con 0x7f33b0064220 2022-07-20T02:31:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39166/0 (socket says 172.21.15.26:39166) 2022-07-20T02:31:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 -- 172.21.15.26:0/3689656990 learned_addr learned my addr 172.21.15.26:0/3689656990 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b5e0d700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b0064220 0x7f33b01400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b560c700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b00b3060 0x7f33b013ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 -- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b0064220 msgr2=0x7f33b01400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b0064220 0x7f33b01400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 -- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b00b3060 msgr2=0x7f33b013ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b00b3060 0x7f33b013ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 -- 172.21.15.26:0/3689656990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a400b180 con 0x7f33b00b7d80 2022-07-20T02:31:23.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b660e700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f33b0147670 tx=0x7f33a8007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a80070d0 con 0x7f33b00b7d80 2022-07-20T02:31:23.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8007230 con 0x7f33b00b7d80 2022-07-20T02:31:23.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b0147240 con 0x7f33b00b7d80 2022-07-20T02:31:23.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a8005610 con 0x7f33b00b7d80 2022-07-20T02:31:23.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.128+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b01476f0 con 0x7f33b00b7d80 2022-07-20T02:31:23.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f33a8010070 con 0x7f33b00b7d80 2022-07-20T02:31:23.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b0155650 con 0x7f33b00b7d80 2022-07-20T02:31:23.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33aeffd700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 0x7f3390055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f33a806b530 con 0x7f33b00b7d80 2022-07-20T02:31:23.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33b5e0d700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 0x7f3390055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.132+0000 7f33b5e0d700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 0x7f3390055f30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f33a401a000 tx=0x7f33a4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.136+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33a80424a0 con 0x7f33b00b7d80 2022-07-20T02:31:23.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.256+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f33b0061060 con 0x7f33b00b7d80 2022-07-20T02:31:23.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.500+0000 7f33aeffd700 1 -- 172.21.15.26:0/3689656990 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v73) v1 ==== 133+0+0 (secure 0 0 0) 0x7f33a8045ac0 con 0x7f33b00b7d80 2022-07-20T02:31:23.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 msgr2=0x7f3390055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 0x7f3390055f30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f33a401a000 tx=0x7f33a4014000).stop 2022-07-20T02:31:23.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 msgr2=0x7f33b013f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f33b0147670 tx=0x7f33a8007af0).stop 2022-07-20T02:31:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 shutdown_connections 2022-07-20T02:31:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33b00b7d80 0x7f33b013f220 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3390053a70 0x7f3390055f30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33b00b3060 0x7f33b013ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 --2- 172.21.15.26:0/3689656990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33b0064220 0x7f33b01400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 >> 172.21.15.26:0/3689656990 conn(0x7f33b005f840 msgr2=0x7f33b00b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:23.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 shutdown_connections 2022-07-20T02:31:23.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.504+0000 7f33b6e0f700 1 -- 172.21.15.26:0/3689656990 wait complete. 2022-07-20T02:31:23.513 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:31:23.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:31:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 Processor -- start 2022-07-20T02:31:23.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 -- start start 2022-07-20T02:31:23.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3d08145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d080b25b0 con 0x7f3d080a9dc0 2022-07-20T02:31:23.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d08144720 con 0x7f3d080aa5c0 2022-07-20T02:31:23.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0e884700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080b0900 con 0x7f3d080aa9a0 2022-07-20T02:31:23.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d077fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3d080a9dc0 0x7f3d0813e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38454/0 (socket says 172.21.15.26:38454) 2022-07-20T02:31:23.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d077fe700 1 -- 172.21.15.26:0/3358687969 learned_addr learned my addr 172.21.15.26:0/3358687969 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:23.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0ce21700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3d08145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269740424 0 0) 0x7f3d080b25b0 con 0x7f3d080a9dc0 2022-07-20T02:31:23.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cec003260 con 0x7f3d080a9dc0 2022-07-20T02:31:23.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985757999 0 0) 0x7f3d08144720 con 0x7f3d080aa5c0 2022-07-20T02:31:23.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d080b25b0 con 0x7f3d080aa5c0 2022-07-20T02:31:23.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840074740 0 0) 0x7f3cec003260 con 0x7f3d080a9dc0 2022-07-20T02:31:23.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 msgr2=0x7f3d08145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:23.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3d08145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 >> v1:172.21.15.120:6789/0 conn(0x7f3d080aa5c0 legacy=0x7f3d08141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d081449c0 con 0x7f3d080a9dc0 2022-07-20T02:31:23.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f3cf0001660 con 0x7f3d080a9dc0 2022-07-20T02:31:23.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 >> v1:172.21.15.26:6789/0 conn(0x7f3d080a9dc0 legacy=0x7f3d0813e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cec003910 0x7f3cec003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cec004500 0x7f3cec004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3cec004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec00b750 con 0x7f3cec003910 2022-07-20T02:31:23.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec00b8c0 con 0x7f3cec004500 2022-07-20T02:31:23.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec00ba80 con 0x7f3d080aa9a0 2022-07-20T02:31:23.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d077fe700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cec004500 0x7f3cec004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cec003910 0x7f3cec003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.648+0000 7f3d0ce21700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3cec004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 -- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 msgr2=0x7f3cec004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:23.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080aa9a0 0x7f3cec004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 -- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cec004500 msgr2=0x7f3cec004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:23.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cec004500 0x7f3cec004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 -- 172.21.15.26:0/3358687969 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec003260 con 0x7f3cec003910 2022-07-20T02:31:23.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cec003910 0x7f3cec003d10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f3cf8001110 tx=0x7f3cf8007a00).ready entity=mon.0 client_cookie=831d442860587d98 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8007d10 con 0x7f3cec003910 2022-07-20T02:31:23.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8007030 con 0x7f3cec003910 2022-07-20T02:31:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d06ffd700 1 -- 172.21.15.26:0/3358687969 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8007390 con 0x7f3cec003910 2022-07-20T02:31:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- 172.21.15.26:0/3358687969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cec003910 msgr2=0x7f3cec003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3358687969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cec003910 0x7f3cec003d10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f3cf8001110 tx=0x7f3cf8007a00).stop 2022-07-20T02:31:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0ce21700 1 -- 172.21.15.26:0/3358687969 reap_dead start 2022-07-20T02:31:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- 172.21.15.26:0/3358687969 shutdown_connections 2022-07-20T02:31:23.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- 172.21.15.26:0/3358687969 >> 172.21.15.26:0/3358687969 conn(0x7f3d080b66e0 msgr2=0x7f3d080b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- 172.21.15.26:0/3358687969 shutdown_connections 2022-07-20T02:31:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- 172.21.15.26:0/3358687969 wait complete. 2022-07-20T02:31:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 Processor -- start 2022-07-20T02:31:23.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.652+0000 7f3d0e884700 1 -- start start 2022-07-20T02:31:23.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080a9730 0x7f3d08153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0809fd30 0x7f3d08154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d081545f0 con 0x7f3d08063c60 2022-07-20T02:31:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d08140a20 con 0x7f3d0809fd30 2022-07-20T02:31:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d08064040 con 0x7f3d080a9730 2022-07-20T02:31:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39184/0 (socket says 172.21.15.26:39184) 2022-07-20T02:31:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 -- 172.21.15.26:0/3558741672 learned_addr learned my addr 172.21.15.26:0/3558741672 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0ce21700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0809fd30 0x7f3d08154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d077fe700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080a9730 0x7f3d08153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 -- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080a9730 msgr2=0x7f3d08153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080a9730 0x7f3d08153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 -- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0809fd30 msgr2=0x7f3d08154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:23.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0809fd30 0x7f3d08154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:23.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 -- 172.21.15.26:0/3558741672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08150ee0 con 0x7f3d08063c60 2022-07-20T02:31:23.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d07fff700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f3cf8001110 tx=0x7f3cf8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8013810 con 0x7f3d08063c60 2022-07-20T02:31:23.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8013970 con 0x7f3d08063c60 2022-07-20T02:31:23.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0809f5f0 con 0x7f3d08063c60 2022-07-20T02:31:23.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8013cd0 con 0x7f3d08063c60 2022-07-20T02:31:23.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d08150b90 con 0x7f3d08063c60 2022-07-20T02:31:23.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.656+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d08151070 con 0x7f3d08063c60 2022-07-20T02:31:23.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.660+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cf8025030 con 0x7f3d08063c60 2022-07-20T02:31:23.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.660+0000 7f3d057fa700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 0x7f3ce0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:23.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.660+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f3cf806ce30 con 0x7f3d08063c60 2022-07-20T02:31:23.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.660+0000 7f3d077fe700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 0x7f3ce0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:23.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.660+0000 7f3d077fe700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 0x7f3ce0055e10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3cf0005000 tx=0x7f3cf0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:23.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.664+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cf8043e40 con 0x7f3d08063c60 2022-07-20T02:31:23.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:23.788+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f3d08151070 con 0x7f3d08063c60 2022-07-20T02:31:24.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.424+0000 7f3d057fa700 1 -- 172.21.15.26:0/3558741672 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v74) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3cf80473e0 con 0x7f3d08063c60 2022-07-20T02:31:24.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 msgr2=0x7f3ce0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 0x7f3ce0055e10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3cf0005000 tx=0x7f3cf0009040).stop 2022-07-20T02:31:24.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 msgr2=0x7f3d08153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f3cf8001110 tx=0x7f3cf8016000).stop 2022-07-20T02:31:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 shutdown_connections 2022-07-20T02:31:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3ce0053950 0x7f3ce0055e10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0809fd30 0x7f3d08154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d080a9730 0x7f3d08153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 --2- 172.21.15.26:0/3558741672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d08063c60 0x7f3d08153720 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.428+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 >> 172.21.15.26:0/3558741672 conn(0x7f3d080b66e0 msgr2=0x7f3d0805f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:24.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.432+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 shutdown_connections 2022-07-20T02:31:24.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.432+0000 7f3d0e884700 1 -- 172.21.15.26:0/3558741672 wait complete. 2022-07-20T02:31:24.435 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:31:24.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:31:24.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 Processor -- start 2022-07-20T02:31:24.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 -- start start 2022-07-20T02:31:24.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 0x7f12101474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100b62d0 con 0x7f12100a8c40 2022-07-20T02:31:24.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121014aac0 con 0x7f12100a7fc0 2022-07-20T02:31:24.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.576+0000 7f1215a01700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121014ad60 con 0x7f12100b5ef0 2022-07-20T02:31:24.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215200700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f12100a8c40 0x7f1210151dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38472/0 (socket says 172.21.15.26:38472) 2022-07-20T02:31:24.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215200700 1 -- 172.21.15.26:0/91162135 learned_addr learned my addr 172.21.15.26:0/91162135 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:24.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f12149ff700 1 --2- 172.21.15.26:0/91162135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 0x7f12101474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332438164 0 0) 0x7f12100b62d0 con 0x7f12100a8c40 2022-07-20T02:31:24.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f0003260 con 0x7f12100a8c40 2022-07-20T02:31:24.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418723339 0 0) 0x7f121014aac0 con 0x7f12100a7fc0 2022-07-20T02:31:24.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12100b62d0 con 0x7f12100a7fc0 2022-07-20T02:31:24.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641980111 0 0) 0x7f11f0003260 con 0x7f12100a8c40 2022-07-20T02:31:24.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f12149ff700 1 -- 172.21.15.26:0/91162135 >> v1:172.21.15.120:6789/0 conn(0x7f12100a7fc0 legacy=0x7f12100aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f12149ff700 1 -- 172.21.15.26:0/91162135 >> v1:172.21.15.26:6789/0 conn(0x7f12100a8c40 legacy=0x7f1210151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f12149ff700 1 -- 172.21.15.26:0/91162135 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121014b000 con 0x7f12100b5ef0 2022-07-20T02:31:24.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f12149ff700 1 --2- 172.21.15.26:0/91162135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 0x7f12101474d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1200001110 tx=0x7f12080091d0).ready entity=mon.2 client_cookie=88eddc194c668dd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1200001660 con 0x7f12100b5ef0 2022-07-20T02:31:24.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120800f510 con 0x7f12100b5ef0 2022-07-20T02:31:24.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f120f7fe700 1 -- 172.21.15.26:0/91162135 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f120800f680 con 0x7f12100b5ef0 2022-07-20T02:31:24.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 -- 172.21.15.26:0/91162135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 msgr2=0x7f12101474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 --2- 172.21.15.26:0/91162135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 0x7f12101474d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1200001110 tx=0x7f12080091d0).stop 2022-07-20T02:31:24.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 -- 172.21.15.26:0/91162135 shutdown_connections 2022-07-20T02:31:24.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 --2- 172.21.15.26:0/91162135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100b5ef0 0x7f12101474d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 -- 172.21.15.26:0/91162135 >> 172.21.15.26:0/91162135 conn(0x7f1210060310 msgr2=0x7f12100b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:24.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 -- 172.21.15.26:0/91162135 shutdown_connections 2022-07-20T02:31:24.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.580+0000 7f1215a01700 1 -- 172.21.15.26:0/91162135 wait complete. 2022-07-20T02:31:24.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 Processor -- start 2022-07-20T02:31:24.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 -- start start 2022-07-20T02:31:24.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 0x7f121005c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12100b5ef0 0x7f1210145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210140800 con 0x7f12100a8c40 2022-07-20T02:31:24.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210140a60 con 0x7f12100b5ef0 2022-07-20T02:31:24.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121014ab50 con 0x7f12100a7fc0 2022-07-20T02:31:24.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f12149ff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 0x7f121005c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39198/0 (socket says 172.21.15.26:39198) 2022-07-20T02:31:24.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215200700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12100b5ef0 0x7f1210145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 -- 172.21.15.26:0/1359479121 learned_addr learned my addr 172.21.15.26:0/1359479121 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:24.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f12149ff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 0x7f121005c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60712/0 (socket says 172.21.15.26:60712) 2022-07-20T02:31:24.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 -- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 msgr2=0x7f121005c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 0x7f121005c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 -- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12100b5ef0 msgr2=0x7f1210145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12100b5ef0 0x7f1210145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 -- 172.21.15.26:0/1359479121 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1208009470 con 0x7f12100a8c40 2022-07-20T02:31:24.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120ffff700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f1210140480 tx=0x7f11f8007b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11f800ce40 con 0x7f12100a8c40 2022-07-20T02:31:24.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f800f470 con 0x7f12100a8c40 2022-07-20T02:31:24.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121014a8f0 con 0x7f12100a8c40 2022-07-20T02:31:24.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.584+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11f800f7d0 con 0x7f12100a8c40 2022-07-20T02:31:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121005cbe0 con 0x7f12100a8c40 2022-07-20T02:31:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1210147d70 con 0x7f12100a8c40 2022-07-20T02:31:24.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11f8019030 con 0x7f12100a8c40 2022-07-20T02:31:24.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f120dffb700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 0x7f11f4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f12149ff700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 0x7f11f4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.588+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f11f806b2e0 con 0x7f12100a8c40 2022-07-20T02:31:24.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.592+0000 7f12149ff700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 0x7f11f4055ff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f120801b000 tx=0x7f1208016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.596+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1210147d70 con 0x7f12100a8c40 2022-07-20T02:31:24.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.756+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1210147d70 con 0x7f12100a8c40 2022-07-20T02:31:24.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.768+0000 7f120dffb700 1 -- 172.21.15.26:0/1359479121 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v36) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1210147d70 con 0x7f12100a8c40 2022-07-20T02:31:24.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 msgr2=0x7f11f4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 0x7f11f4055ff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f120801b000 tx=0x7f1208016000).stop 2022-07-20T02:31:24.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 msgr2=0x7f1210148030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f1210140480 tx=0x7f11f8007b30).stop 2022-07-20T02:31:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 shutdown_connections 2022-07-20T02:31:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12100a8c40 0x7f1210148030 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11f4053b30 0x7f11f4055ff0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12100b5ef0 0x7f1210145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 --2- 172.21.15.26:0/1359479121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12100a7fc0 0x7f121005c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 >> 172.21.15.26:0/1359479121 conn(0x7f1210060310 msgr2=0x7f12100b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:24.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 shutdown_connections 2022-07-20T02:31:24.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.772+0000 7f1215a01700 1 -- 172.21.15.26:0/1359479121 wait complete. 2022-07-20T02:31:24.780 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 13 and data pool 14 2022-07-20T02:31:24.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:24.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 Processor -- start 2022-07-20T02:31:24.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 -- start start 2022-07-20T02:31:24.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fae280add30 0x7fae280b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae280af6d0 0x7fae280afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280aa1e0 con 0x7fae28060950 2022-07-20T02:31:24.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280ae110 con 0x7fae280add30 2022-07-20T02:31:24.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e8bc700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280a9f40 con 0x7fae280af6d0 2022-07-20T02:31:24.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fae280add30 0x7fae280b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fae280add30 0x7fae280b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39214/0 (socket says 172.21.15.26:39214) 2022-07-20T02:31:24.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 -- 172.21.15.26:0/3162967737 learned_addr learned my addr 172.21.15.26:0/3162967737 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:24.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2e0bb700 1 --2- 172.21.15.26:0/3162967737 >> v2:172.21.15.120:3300/0 conn(0x7fae280af6d0 0x7fae280afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 -- 172.21.15.26:0/3162967737 >> v2:172.21.15.120:3300/0 conn(0x7fae280af6d0 msgr2=0x7fae280afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 --2- 172.21.15.26:0/3162967737 >> v2:172.21.15.120:3300/0 conn(0x7fae280af6d0 0x7fae280afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.912+0000 7fae2d8ba700 1 -- 172.21.15.26:0/3162967737 >> v1:172.21.15.120:6789/0 conn(0x7fae28060950 legacy=0x7fae28060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2d8ba700 1 -- 172.21.15.26:0/3162967737 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280b1110 con 0x7fae280add30 2022-07-20T02:31:24.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2d8ba700 1 --2- 172.21.15.26:0/3162967737 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280add30 0x7fae280b2ab0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fae24002f70 tx=0x7fae24006f40).ready entity=mon.0 client_cookie=c25f705286140737 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2c8b8700 1 -- 172.21.15.26:0/3162967737 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2400dc70 con 0x7fae280add30 2022-07-20T02:31:24.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2c8b8700 1 -- 172.21.15.26:0/3162967737 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae24000e90 con 0x7fae280add30 2022-07-20T02:31:24.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2c8b8700 1 -- 172.21.15.26:0/3162967737 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2400be90 con 0x7fae280add30 2022-07-20T02:31:24.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- 172.21.15.26:0/3162967737 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280add30 msgr2=0x7fae280b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/3162967737 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280add30 0x7fae280b2ab0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fae24002f70 tx=0x7fae24006f40).stop 2022-07-20T02:31:24.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- 172.21.15.26:0/3162967737 shutdown_connections 2022-07-20T02:31:24.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/3162967737 >> v2:172.21.15.120:3300/0 conn(0x7fae280af6d0 0x7fae280afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/3162967737 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280add30 0x7fae280b2ab0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- 172.21.15.26:0/3162967737 >> 172.21.15.26:0/3162967737 conn(0x7fae280b3050 msgr2=0x7fae280b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:24.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- 172.21.15.26:0/3162967737 shutdown_connections 2022-07-20T02:31:24.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- 172.21.15.26:0/3162967737 wait complete. 2022-07-20T02:31:24.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 Processor -- start 2022-07-20T02:31:24.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- start start 2022-07-20T02:31:24.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae28060950 0x7fae2814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae280af6d0 0x7fae280b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2814f190 con 0x7fae280acda0 2022-07-20T02:31:24.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280a2240 con 0x7fae280af6d0 2022-07-20T02:31:24.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e8bc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280a18a0 con 0x7fae28060950 2022-07-20T02:31:24.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.916+0000 7fae2e0bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2d8ba700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae28060950 0x7fae2814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39224/0 (socket says 172.21.15.26:39224) 2022-07-20T02:31:24.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 -- 172.21.15.26:0/1448048844 learned_addr learned my addr 172.21.15.26:0/1448048844 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:24.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2d0b9700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae280af6d0 0x7fae280b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 -- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae28060950 msgr2=0x7fae2814eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae28060950 0x7fae2814eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 -- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae280af6d0 msgr2=0x7fae280b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae280af6d0 0x7fae280b59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 -- 172.21.15.26:0/1448048844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2400b460 con 0x7fae280acda0 2022-07-20T02:31:24.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e0bb700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7fae2000a000 tx=0x7fae20008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae200127e0 con 0x7fae280acda0 2022-07-20T02:31:24.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae20012940 con 0x7fae280acda0 2022-07-20T02:31:24.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae20012ca0 con 0x7fae280acda0 2022-07-20T02:31:24.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae280a1b00 con 0x7fae280acda0 2022-07-20T02:31:24.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae280b5ea0 con 0x7fae280acda0 2022-07-20T02:31:24.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae280b13b0 con 0x7fae280acda0 2022-07-20T02:31:24.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae20013480 con 0x7fae280acda0 2022-07-20T02:31:24.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 0x7fae08055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:24.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.920+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fae2006c340 con 0x7fae280acda0 2022-07-20T02:31:24.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.924+0000 7fae2d8ba700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 0x7fae08055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:24.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.924+0000 7fae2d8ba700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 0x7fae08055ea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fae2401e000 tx=0x7fae280ae860).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:24.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:24.928+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae200430a0 con 0x7fae280acda0 2022-07-20T02:31:25.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.056+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae280574e0 con 0x7fae280acda0 2022-07-20T02:31:25.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.056+0000 7fae16ffd700 1 -- 172.21.15.26:0/1448048844 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16335 (secure 0 0 0) 0x7fae20012e00 con 0x7fae280acda0 2022-07-20T02:31:25.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:25.060 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":75,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:24.761211+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 msgr2=0x7fae08055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 0x7fae08055ea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fae2401e000 tx=0x7fae280ae860).stop 2022-07-20T02:31:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 msgr2=0x7fae281516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7fae2000a000 tx=0x7fae20008040).stop 2022-07-20T02:31:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 shutdown_connections 2022-07-20T02:31:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fae280acda0 0x7fae281516f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fae080539e0 0x7fae08055ea0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae280af6d0 0x7fae280b59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 --2- 172.21.15.26:0/1448048844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae28060950 0x7fae2814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 >> 172.21.15.26:0/1448048844 conn(0x7fae280b3050 msgr2=0x7fae28141ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 shutdown_connections 2022-07-20T02:31:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.060+0000 7fae2e8bc700 1 -- 172.21.15.26:0/1448048844 wait complete. 2022-07-20T02:31:25.071 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:25.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.176+0000 7f8ccaf69700 1 Processor -- start 2022-07-20T02:31:25.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.176+0000 7f8ccaf69700 1 -- start start 2022-07-20T02:31:25.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc40b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cc4067440 0x7f8cc41433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc4147080 con 0x7f8cc40b2f20 2022-07-20T02:31:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc4143000 con 0x7f8cc4067440 2022-07-20T02:31:25.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc41472b0 con 0x7f8cc40957f0 2022-07-20T02:31:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cc4067440 0x7f8cc41433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cc4067440 0x7f8cc41433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39232/0 (socket says 172.21.15.26:39232) 2022-07-20T02:31:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 -- 172.21.15.26:0/1612655946 learned_addr learned my addr 172.21.15.26:0/1612655946 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:25.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cc9766700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc40b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 -- 172.21.15.26:0/1612655946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 msgr2=0x7f8cc40b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc40b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 -- 172.21.15.26:0/1612655946 >> v1:172.21.15.120:6789/0 conn(0x7f8cc40b2f20 legacy=0x7f8cc40950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 -- 172.21.15.26:0/1612655946 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40a9920 con 0x7f8cc4067440 2022-07-20T02:31:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cca768700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc41433f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f8cbc0032f0 tx=0x7f8cbc009290).ready entity=mon.0 client_cookie=31c6decba77f3fae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cc8f65700 1 -- 172.21.15.26:0/1612655946 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cbc00b8a0 con 0x7f8cc4067440 2022-07-20T02:31:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cc8f65700 1 -- 172.21.15.26:0/1612655946 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc00ba00 con 0x7f8cc4067440 2022-07-20T02:31:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8cc8f65700 1 -- 172.21.15.26:0/1612655946 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cbc00bb70 con 0x7f8cc4067440 2022-07-20T02:31:25.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- 172.21.15.26:0/1612655946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 msgr2=0x7f8cc41433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc41433f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f8cbc0032f0 tx=0x7f8cbc009290).stop 2022-07-20T02:31:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- 172.21.15.26:0/1612655946 shutdown_connections 2022-07-20T02:31:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc41433f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/1612655946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc40b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- 172.21.15.26:0/1612655946 >> 172.21.15.26:0/1612655946 conn(0x7f8cc40b67f0 msgr2=0x7f8cc40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- 172.21.15.26:0/1612655946 shutdown_connections 2022-07-20T02:31:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.180+0000 7f8ccaf69700 1 -- 172.21.15.26:0/1612655946 wait complete. 2022-07-20T02:31:25.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 Processor -- start 2022-07-20T02:31:25.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- start start 2022-07-20T02:31:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc4098ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc40b2f20 0x7f8cc40993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40ae560 con 0x7f8cc4067440 2022-07-20T02:31:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40af850 con 0x7f8cc40b2f20 2022-07-20T02:31:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40aecd0 con 0x7f8cc40957f0 2022-07-20T02:31:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9766700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc4098ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cca768700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc40b2f20 0x7f8cc40993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39236/0 (socket says 172.21.15.26:39236) 2022-07-20T02:31:25.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 -- 172.21.15.26:0/715082589 learned_addr learned my addr 172.21.15.26:0/715082589 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 -- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 msgr2=0x7f8cc4098ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc4098ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 -- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc40b2f20 msgr2=0x7f8cc40993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc40b2f20 0x7f8cc40993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 -- 172.21.15.26:0/715082589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc009530 con 0x7f8cc4067440 2022-07-20T02:31:25.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cc9f67700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f8cc41432d0 tx=0x7f8cc0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc0012400 con 0x7f8cc4067440 2022-07-20T02:31:25.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0012560 con 0x7f8cc4067440 2022-07-20T02:31:25.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc409b850 con 0x7f8cc4067440 2022-07-20T02:31:25.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc00128c0 con 0x7f8cc4067440 2022-07-20T02:31:25.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.184+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc40afc20 con 0x7f8cc4067440 2022-07-20T02:31:25.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc40a95a0 con 0x7f8cc4067440 2022-07-20T02:31:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8cc0019030 con 0x7f8cc4067440 2022-07-20T02:31:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8cb2ffd700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 0x7f8ca4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8cc006b400 con 0x7f8cc4067440 2022-07-20T02:31:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8cc9766700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 0x7f8ca4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.188+0000 7f8cc9766700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 0x7f8ca4055e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f8cb400c000 tx=0x7f8cb4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.196+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8cc00420b0 con 0x7f8cc4067440 2022-07-20T02:31:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.324+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cc40574e0 con 0x7f8cc4067440 2022-07-20T02:31:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.328+0000 7f8cb2ffd700 1 -- 172.21.15.26:0/715082589 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16335 (secure 0 0 0) 0x7f8cc00456d0 con 0x7f8cc4067440 2022-07-20T02:31:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:25.329 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":75,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:24.761211+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.328+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 msgr2=0x7f8ca4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.328+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 0x7f8ca4055e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f8cb400c000 tx=0x7f8cb4008040).stop 2022-07-20T02:31:25.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.328+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 msgr2=0x7f8cc4097600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.328+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f8cc41432d0 tx=0x7f8cc0007c50).stop 2022-07-20T02:31:25.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 shutdown_connections 2022-07-20T02:31:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8ca4053950 0x7f8ca4055e10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc40b2f20 0x7f8cc40993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cc40957f0 0x7f8cc4098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 --2- 172.21.15.26:0/715082589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cc4067440 0x7f8cc4097600 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 >> 172.21.15.26:0/715082589 conn(0x7f8cc40b67f0 msgr2=0x7f8cc41460a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 shutdown_connections 2022-07-20T02:31:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.332+0000 7f8ccaf69700 1 -- 172.21.15.26:0/715082589 wait complete. 2022-07-20T02:31:25.342 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:31:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 Processor -- start 2022-07-20T02:31:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 -- start start 2022-07-20T02:31:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 0x7f9dc80b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dc80b1850 0x7f9dc8142db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80b62d0 con 0x7f9dc80a97e0 2022-07-20T02:31:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc80a9cf0 con 0x7f9dc80b1850 2022-07-20T02:31:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.464+0000 7f9dcd841700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc80a9f90 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dc80b1850 0x7f9dc8142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dc80b1850 0x7f9dc8142db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39242/0 (socket says 172.21.15.26:39242) 2022-07-20T02:31:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 -- 172.21.15.26:0/3613595089 learned_addr learned my addr 172.21.15.26:0/3613595089 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 0x7f9dc80b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 -- 172.21.15.26:0/3613595089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 msgr2=0x7f9dc80b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 0x7f9dc80b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 -- 172.21.15.26:0/3613595089 >> v1:172.21.15.120:6789/0 conn(0x7f9dc80a97e0 legacy=0x7f9dc81426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dc77fe700 1 -- 172.21.15.26:0/3613595089 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414229848 0 0) 0x7f9dc80b62d0 con 0x7f9dc80a97e0 2022-07-20T02:31:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 -- 172.21.15.26:0/3613595089 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc80aa230 con 0x7f9dc80b1850 2022-07-20T02:31:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 0x7f9dc80b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd040700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80b1850 0x7f9dc8142db0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f9dc00032f0 tx=0x7f9dc0009290).ready entity=mon.0 client_cookie=20ab24e40559fab7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dc77fe700 1 -- 172.21.15.26:0/3613595089 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000b8a0 con 0x7f9dc80b1850 2022-07-20T02:31:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dc77fe700 1 -- 172.21.15.26:0/3613595089 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc000ba00 con 0x7f9dc80b1850 2022-07-20T02:31:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dc77fe700 1 -- 172.21.15.26:0/3613595089 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000bb70 con 0x7f9dc80b1850 2022-07-20T02:31:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 -- 172.21.15.26:0/3613595089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80b1850 msgr2=0x7f9dc8142db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80b1850 0x7f9dc8142db0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f9dc00032f0 tx=0x7f9dc0009290).stop 2022-07-20T02:31:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 -- 172.21.15.26:0/3613595089 shutdown_connections 2022-07-20T02:31:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80b1850 0x7f9dc8142db0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3613595089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b5ef0 0x7f9dc80b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 -- 172.21.15.26:0/3613595089 >> 172.21.15.26:0/3613595089 conn(0x7f9dc8060310 msgr2=0x7f9dc80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 -- 172.21.15.26:0/3613595089 shutdown_connections 2022-07-20T02:31:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.468+0000 7f9dcd841700 1 -- 172.21.15.26:0/3613595089 wait complete. 2022-07-20T02:31:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 Processor -- start 2022-07-20T02:31:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- start start 2022-07-20T02:31:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b1850 0x7f9dc81339e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81343c0 con 0x7f9dc80a97e0 2022-07-20T02:31:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc8136f60 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc8137200 con 0x7f9dc80b1850 2022-07-20T02:31:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcc83f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58944/0 (socket says 172.21.15.26:58944) 2022-07-20T02:31:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 -- 172.21.15.26:0/3186465198 learned_addr learned my addr 172.21.15.26:0/3186465198 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39248/0 (socket says 172.21.15.26:39248) 2022-07-20T02:31:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dc7fff700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b1850 0x7f9dc81339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 -- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b1850 msgr2=0x7f9dc81339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b1850 0x7f9dc81339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 -- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 msgr2=0x7f9dc813a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 -- 172.21.15.26:0/3186465198 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc00b180 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd040700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f9dc00032f0 tx=0x7f9dc0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000bb70 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0021630 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc0009530 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000b8a0 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.472+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc80656d0 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc8136d20 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9dc0027020 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dc5ffb700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 0x7f9da8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 0x7f9da8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9dc006d460 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.476+0000 7f9dcc83f700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 0x7f9da8055e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f9dbc00b000 tx=0x7f9dbc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.488+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9dc8136d20 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.652+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9dc8136d20 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.780+0000 7f9dc5ffb700 1 -- 172.21.15.26:0/3186465198 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v37) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9dc8136d20 con 0x7f9dc80b5ef0 2022-07-20T02:31:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.780+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 msgr2=0x7f9da8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.780+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 0x7f9da8055e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f9dbc00b000 tx=0x7f9dbc006df0).stop 2022-07-20T02:31:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.780+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 msgr2=0x7f9dc8133ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.780+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f9dc00032f0 tx=0x7f9dc0016000).stop 2022-07-20T02:31:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 shutdown_connections 2022-07-20T02:31:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9da8053950 0x7f9da8055e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc80b5ef0 0x7f9dc8133ed0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc80b1850 0x7f9dc81339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 --2- 172.21.15.26:0/3186465198 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dc80a97e0 0x7f9dc813a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 >> 172.21.15.26:0/3186465198 conn(0x7f9dc8060310 msgr2=0x7f9dc80ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 shutdown_connections 2022-07-20T02:31:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.784+0000 7f9dcd841700 1 -- 172.21.15.26:0/3186465198 wait complete. 2022-07-20T02:31:25.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:25.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 Processor -- start 2022-07-20T02:31:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 -- start start 2022-07-20T02:31:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d640b3060 0x7f5d640b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d64064220 0x7f5d641434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d640b2e40 con 0x7f5d640b7d80 2022-07-20T02:31:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d640b4c20 con 0x7f5d640b3060 2022-07-20T02:31:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.916+0000 7f5d6b4e4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d640617a0 con 0x7f5d64064220 2022-07-20T02:31:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6a4e2700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5d640b7d80 0x7f5d640b4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38536/0 (socket says 172.21.15.26:38536) 2022-07-20T02:31:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6a4e2700 1 -- 172.21.15.26:0/3326083886 learned_addr learned my addr 172.21.15.26:0/3326083886 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d640b3060 0x7f5d640b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6ace3700 1 --2- 172.21.15.26:0/3326083886 >> v2:172.21.15.120:3300/0 conn(0x7f5d64064220 0x7f5d641434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d694e0700 1 -- 172.21.15.26:0/3326083886 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696374225 0 0) 0x7f5d640b2e40 con 0x7f5d640b7d80 2022-07-20T02:31:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d694e0700 1 -- 172.21.15.26:0/3326083886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d50003260 con 0x7f5d640b7d80 2022-07-20T02:31:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 -- 172.21.15.26:0/3326083886 >> v2:172.21.15.120:3300/0 conn(0x7f5d64064220 msgr2=0x7f5d641434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 --2- 172.21.15.26:0/3326083886 >> v2:172.21.15.120:3300/0 conn(0x7f5d64064220 0x7f5d641434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 -- 172.21.15.26:0/3326083886 >> v1:172.21.15.26:6789/0 conn(0x7f5d640b7d80 legacy=0x7f5d640b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 -- 172.21.15.26:0/3326083886 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64060b20 con 0x7f5d640b3060 2022-07-20T02:31:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d69ce1700 1 --2- 172.21.15.26:0/3326083886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b3060 0x7f5d640b4730 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f5d640618e0 tx=0x7f5d58006df0).ready entity=mon.0 client_cookie=bdaa01c8d2112ed7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d694e0700 1 -- 172.21.15.26:0/3326083886 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5800d4f0 con 0x7f5d640b3060 2022-07-20T02:31:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d694e0700 1 -- 172.21.15.26:0/3326083886 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d58000e90 con 0x7f5d640b3060 2022-07-20T02:31:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d694e0700 1 -- 172.21.15.26:0/3326083886 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5800d840 con 0x7f5d640b3060 2022-07-20T02:31:25.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/3326083886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b3060 msgr2=0x7f5d640b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/3326083886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b3060 0x7f5d640b4730 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f5d640618e0 tx=0x7f5d58006df0).stop 2022-07-20T02:31:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/3326083886 shutdown_connections 2022-07-20T02:31:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/3326083886 >> v2:172.21.15.120:3300/0 conn(0x7f5d64064220 0x7f5d641434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/3326083886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b3060 0x7f5d640b4730 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/3326083886 >> 172.21.15.26:0/3326083886 conn(0x7f5d6405f840 msgr2=0x7f5d6405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/3326083886 shutdown_connections 2022-07-20T02:31:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/3326083886 wait complete. 2022-07-20T02:31:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 Processor -- start 2022-07-20T02:31:25.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.920+0000 7f5d6b4e4700 1 -- start start 2022-07-20T02:31:25.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 0x7f5d64155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d640b3060 0x7f5d64148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d641525d0 con 0x7f5d640b7d80 2022-07-20T02:31:25.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d640ad4c0 con 0x7f5d640b3060 2022-07-20T02:31:25.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d64152830 con 0x7f5d64064220 2022-07-20T02:31:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39266/0 (socket says 172.21.15.26:39266) 2022-07-20T02:31:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 -- 172.21.15.26:0/2805294142 learned_addr learned my addr 172.21.15.26:0/2805294142 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d69ce1700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d640b3060 0x7f5d64148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6a4e2700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 0x7f5d64155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 -- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 msgr2=0x7f5d64155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 0x7f5d64155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 -- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d640b3060 msgr2=0x7f5d64148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d640b3060 0x7f5d64148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 -- 172.21.15.26:0/2805294142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5800b180 con 0x7f5d640b7d80 2022-07-20T02:31:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6a4e2700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 0x7f5d64155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6ace3700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f5d5c006e00 tx=0x7f5d5c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5c00c850 con 0x7f5d640b7d80 2022-07-20T02:31:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c00c9b0 con 0x7f5d640b7d80 2022-07-20T02:31:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d640ad760 con 0x7f5d640b7d80 2022-07-20T02:31:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5c00cd10 con 0x7f5d640b7d80 2022-07-20T02:31:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d640ac720 con 0x7f5d640b7d80 2022-07-20T02:31:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d641529f0 con 0x7f5d640b7d80 2022-07-20T02:31:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d5c0123f0 con 0x7f5d640b7d80 2022-07-20T02:31:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 0x7f5d44055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5d5c06c3c0 con 0x7f5d640b7d80 2022-07-20T02:31:25.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.924+0000 7f5d6a4e2700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 0x7f5d44055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:25.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.928+0000 7f5d6a4e2700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 0x7f5d44055f60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5d5400b000 tx=0x7f5d54009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:25.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:25.932+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d5c043120 con 0x7f5d640b7d80 2022-07-20T02:31:26.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.096+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d64060dc0 con 0x7f5d640b7d80 2022-07-20T02:31:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.096+0000 7f5d637fe700 1 -- 172.21.15.26:0/2805294142 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f5d5c046740 con 0x7f5d640b7d80 2022-07-20T02:31:26.105 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:26.105 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 msgr2=0x7f5d44055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 0x7f5d44055f60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5d5400b000 tx=0x7f5d54009040).stop 2022-07-20T02:31:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 msgr2=0x7f5d64152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f5d5c006e00 tx=0x7f5d5c0091d0).stop 2022-07-20T02:31:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 shutdown_connections 2022-07-20T02:31:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d640b7d80 0x7f5d64152cd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d44053aa0 0x7f5d44055f60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d640b3060 0x7f5d64148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 --2- 172.21.15.26:0/2805294142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d64064220 0x7f5d64155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 >> 172.21.15.26:0/2805294142 conn(0x7f5d6405f840 msgr2=0x7f5d640b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 shutdown_connections 2022-07-20T02:31:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.100+0000 7f5d6b4e4700 1 -- 172.21.15.26:0/2805294142 wait complete. 2022-07-20T02:31:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:26.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:26.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 Processor -- start 2022-07-20T02:31:26.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 -- start start 2022-07-20T02:31:26.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6840b5ef0 0x7fd6841474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 0x7fd6840af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6840b62d0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68414aac0 con 0x7fd6840b5ef0 2022-07-20T02:31:26.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68414ad60 con 0x7fd6840a8c40 2022-07-20T02:31:26.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6840b5ef0 0x7fd6841474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd682d9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 0x7fd6840af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6840b5ef0 0x7fd6841474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39272/0 (socket says 172.21.15.26:39272) 2022-07-20T02:31:26.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd682d9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 0x7fd6840af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:58966/0 (socket says 172.21.15.26:58966) 2022-07-20T02:31:26.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 -- 172.21.15.26:0/1464797053 learned_addr learned my addr 172.21.15.26:0/1464797053 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:26.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 -- 172.21.15.26:0/1464797053 >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 msgr2=0x7fd6840af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:26.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 --2- 172.21.15.26:0/1464797053 >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 0x7fd6840af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 -- 172.21.15.26:0/1464797053 >> v1:172.21.15.120:6789/0 conn(0x7fd6840a7fc0 legacy=0x7fd6840aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 -- 172.21.15.26:0/1464797053 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68414b000 con 0x7fd6840b5ef0 2022-07-20T02:31:26.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.248+0000 7fd68259c700 1 --2- 172.21.15.26:0/1464797053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840b5ef0 0x7fd6841474d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fd678002f70 tx=0x7fd678006eb0).ready entity=mon.0 client_cookie=63f70877cebf47d3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68159a700 1 -- 172.21.15.26:0/1464797053 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd67800d5c0 con 0x7fd6840b5ef0 2022-07-20T02:31:26.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68159a700 1 -- 172.21.15.26:0/1464797053 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678000e90 con 0x7fd6840b5ef0 2022-07-20T02:31:26.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68159a700 1 -- 172.21.15.26:0/1464797053 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd67800d950 con 0x7fd6840b5ef0 2022-07-20T02:31:26.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- 172.21.15.26:0/1464797053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840b5ef0 msgr2=0x7fd6841474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 --2- 172.21.15.26:0/1464797053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840b5ef0 0x7fd6841474d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fd678002f70 tx=0x7fd678006eb0).stop 2022-07-20T02:31:26.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- 172.21.15.26:0/1464797053 shutdown_connections 2022-07-20T02:31:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 --2- 172.21.15.26:0/1464797053 >> v2:172.21.15.120:3300/0 conn(0x7fd6840a8c40 0x7fd6840af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 --2- 172.21.15.26:0/1464797053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840b5ef0 0x7fd6841474d0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- 172.21.15.26:0/1464797053 >> 172.21.15.26:0/1464797053 conn(0x7fd684060310 msgr2=0x7fd6840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- 172.21.15.26:0/1464797053 shutdown_connections 2022-07-20T02:31:26.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- 172.21.15.26:0/1464797053 wait complete. 2022-07-20T02:31:26.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 Processor -- start 2022-07-20T02:31:26.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.252+0000 7fd68359e700 1 -- start start 2022-07-20T02:31:26.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6840a8c40 0x7fd6840ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 0x7fd6840abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6841477c0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684146c60 con 0x7fd6840b5ef0 2022-07-20T02:31:26.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd684146f00 con 0x7fd6840a8c40 2022-07-20T02:31:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39276/0 (socket says 172.21.15.26:39276) 2022-07-20T02:31:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 -- 172.21.15.26:0/2802011461 learned_addr learned my addr 172.21.15.26:0/2802011461 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd682d9d700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 0x7fd6840abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd681d9b700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6840a8c40 0x7fd6840ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 -- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6840a8c40 msgr2=0x7fd6840ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6840a8c40 0x7fd6840ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 -- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 msgr2=0x7fd6840abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 0x7fd6840abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 -- 172.21.15.26:0/2802011461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67800b1e0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd682d9d700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 0x7fd6840abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68259c700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fd678002f70 tx=0x7fd678015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd67800d940 con 0x7fd6840a7fc0 2022-07-20T02:31:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678011c30 con 0x7fd6840a7fc0 2022-07-20T02:31:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6841471a0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd67801e460 con 0x7fd6840a7fc0 2022-07-20T02:31:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6840aadb0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.256+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6840aa210 con 0x7fd6840a7fc0 2022-07-20T02:31:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.260+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd67801a040 con 0x7fd6840a7fc0 2022-07-20T02:31:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.260+0000 7fd6737fe700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 0x7fd660055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.260+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd67806cf40 con 0x7fd6840a7fc0 2022-07-20T02:31:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.260+0000 7fd681d9b700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 0x7fd660055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.260+0000 7fd681d9b700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 0x7fd660055f60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd67400d000 tx=0x7fd674000f40).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.268+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd678043d30 con 0x7fd6840a7fc0 2022-07-20T02:31:26.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.388+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd68414b2a0 con 0x7fd6840a7fc0 2022-07-20T02:31:26.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd6737fe700 1 -- 172.21.15.26:0/2802011461 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16322 (secure 0 0 0) 0x7fd678023020 con 0x7fd6840a7fc0 2022-07-20T02:31:26.397 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:26.398 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":76,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:25.765356+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:26.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 msgr2=0x7fd660055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 0x7fd660055f60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd67400d000 tx=0x7fd674000f40).stop 2022-07-20T02:31:26.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 msgr2=0x7fd68414a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fd678002f70 tx=0x7fd678015000).stop 2022-07-20T02:31:26.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 shutdown_connections 2022-07-20T02:31:26.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd660053aa0 0x7fd660055f60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6840b5ef0 0x7fd6840abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6840a8c40 0x7fd6840ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 --2- 172.21.15.26:0/2802011461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6840a7fc0 0x7fd68414a3b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 >> 172.21.15.26:0/2802011461 conn(0x7fd684060310 msgr2=0x7fd6841448d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 shutdown_connections 2022-07-20T02:31:26.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.392+0000 7fd68359e700 1 -- 172.21.15.26:0/2802011461 wait complete. 2022-07-20T02:31:26.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:31:26.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.528+0000 7f51c759e700 1 Processor -- start 2022-07-20T02:31:26.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.528+0000 7f51c759e700 1 -- start start 2022-07-20T02:31:26.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51c805cfd0 0x7f51c8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 0x7f51c80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c81474e0 con 0x7f51c8060f00 2022-07-20T02:31:26.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c805d3b0 con 0x7f51c805cfd0 2022-07-20T02:31:26.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c81431e0 con 0x7f51c8147000 2022-07-20T02:31:26.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f51c805cfd0 0x7f51c8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c6d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f51c8060f00 0x7f51c813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38568/0 (socket says 172.21.15.26:38568) 2022-07-20T02:31:26.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c6d9d700 1 -- 172.21.15.26:0/722719860 learned_addr learned my addr 172.21.15.26:0/722719860 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:26.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c5d9b700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 0x7f51c80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c559a700 1 -- 172.21.15.26:0/722719860 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393488184 0 0) 0x7f51c81474e0 con 0x7f51c8060f00 2022-07-20T02:31:26.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c559a700 1 -- 172.21.15.26:0/722719860 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51a8003260 con 0x7f51c8060f00 2022-07-20T02:31:26.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 -- 172.21.15.26:0/722719860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 msgr2=0x7f51c80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:26.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 0x7f51c80b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 -- 172.21.15.26:0/722719860 >> v1:172.21.15.26:6789/0 conn(0x7f51c8060f00 legacy=0x7f51c813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 -- 172.21.15.26:0/722719860 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c8143480 con 0x7f51c805cfd0 2022-07-20T02:31:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c5d9b700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 0x7f51c80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c659c700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c805cfd0 0x7f51c8143bc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f51b8002f70 tx=0x7f51b8006df0).ready entity=mon.0 client_cookie=ffba60afcb5ddb53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c559a700 1 -- 172.21.15.26:0/722719860 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51b800d570 con 0x7f51c805cfd0 2022-07-20T02:31:26.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c559a700 1 -- 172.21.15.26:0/722719860 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b8000e90 con 0x7f51c805cfd0 2022-07-20T02:31:26.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c559a700 1 -- 172.21.15.26:0/722719860 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51b800d8c0 con 0x7f51c805cfd0 2022-07-20T02:31:26.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- 172.21.15.26:0/722719860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c805cfd0 msgr2=0x7f51c8143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c805cfd0 0x7f51c8143bc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f51b8002f70 tx=0x7f51b8006df0).stop 2022-07-20T02:31:26.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- 172.21.15.26:0/722719860 shutdown_connections 2022-07-20T02:31:26.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8147000 0x7f51c80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 --2- 172.21.15.26:0/722719860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c805cfd0 0x7f51c8143bc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- 172.21.15.26:0/722719860 >> 172.21.15.26:0/722719860 conn(0x7f51c80620c0 msgr2=0x7f51c805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- 172.21.15.26:0/722719860 shutdown_connections 2022-07-20T02:31:26.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- 172.21.15.26:0/722719860 wait complete. 2022-07-20T02:31:26.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 Processor -- start 2022-07-20T02:31:26.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.532+0000 7f51c759e700 1 -- start start 2022-07-20T02:31:26.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8060f00 0x7f51c8141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51c8147000 0x7f51c80b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c8141410 con 0x7f51c80b28c0 2022-07-20T02:31:26.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c81416b0 con 0x7f51c8147000 2022-07-20T02:31:26.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c815a130 con 0x7f51c8060f00 2022-07-20T02:31:26.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39294/0 (socket says 172.21.15.26:39294) 2022-07-20T02:31:26.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 -- 172.21.15.26:0/200950350 learned_addr learned my addr 172.21.15.26:0/200950350 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:26.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c659c700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8060f00 0x7f51c8141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c5d9b700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51c8147000 0x7f51c80b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 -- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8060f00 msgr2=0x7f51c8141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8060f00 0x7f51c8141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 -- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51c8147000 msgr2=0x7f51c80b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51c8147000 0x7f51c80b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 -- 172.21.15.26:0/200950350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b800b180 con 0x7f51c80b28c0 2022-07-20T02:31:26.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c6d9d700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f51c0001110 tx=0x7f51c0007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51c000cea0 con 0x7f51c80b28c0 2022-07-20T02:31:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c00072e0 con 0x7f51c80b28c0 2022-07-20T02:31:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c80ae210 con 0x7f51c80b28c0 2022-07-20T02:31:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51c0007490 con 0x7f51c80b28c0 2022-07-20T02:31:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c8141120 con 0x7f51c80b28c0 2022-07-20T02:31:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c8141870 con 0x7f51c80b28c0 2022-07-20T02:31:26.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51c0010070 con 0x7f51c80b28c0 2022-07-20T02:31:26.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 0x7f51ac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.536+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f51c006b6b0 con 0x7f51c80b28c0 2022-07-20T02:31:26.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.540+0000 7f51c659c700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 0x7f51ac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.540+0000 7f51c659c700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 0x7f51ac055ea0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f51b8018000 tx=0x7f51b8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.544+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f51c00423d0 con 0x7f51c80b28c0 2022-07-20T02:31:26.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.708+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f51c815a2f0 con 0x7f51c80b28c0 2022-07-20T02:31:26.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.716+0000 7f51bf7fe700 1 -- 172.21.15.26:0/200950350 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7f51c00459f0 con 0x7f51c80b28c0 2022-07-20T02:31:26.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 msgr2=0x7f51ac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 0x7f51ac055ea0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f51b8018000 tx=0x7f51b8014000).stop 2022-07-20T02:31:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 msgr2=0x7f51c815a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f51c0001110 tx=0x7f51c0007af0).stop 2022-07-20T02:31:26.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 shutdown_connections 2022-07-20T02:31:26.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f51c80b28c0 0x7f51c815a5d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ac0539e0 0x7f51ac055ea0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51c8147000 0x7f51c80b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 --2- 172.21.15.26:0/200950350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51c8060f00 0x7f51c8141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 >> 172.21.15.26:0/200950350 conn(0x7f51c805cfd0 msgr2=0x7f51c80b97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 shutdown_connections 2022-07-20T02:31:26.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.720+0000 7f51c759e700 1 -- 172.21.15.26:0/200950350 wait complete. 2022-07-20T02:31:26.737 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:31:26.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:31:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.884+0000 7fa274ad9700 1 Processor -- start 2022-07-20T02:31:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.884+0000 7fa274ad9700 1 -- start start 2022-07-20T02:31:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700b5ef0 0x7fa2701474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa2700a7fc0 0x7fa2701479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2700b62d0 con 0x7fa2700a8c40 2022-07-20T02:31:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa270147eb0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27014aac0 con 0x7fa2700b5ef0 2022-07-20T02:31:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa2700a7fc0 0x7fa2701479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa2700a7fc0 0x7fa2701479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39300/0 (socket says 172.21.15.26:39300) 2022-07-20T02:31:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa2700a8c40 0x7fa2700aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38582/0 (socket says 172.21.15.26:38582) 2022-07-20T02:31:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 -- 172.21.15.26:0/1845166552 learned_addr learned my addr 172.21.15.26:0/1845166552 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26f7fe700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700b5ef0 0x7fa2701474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 -- 172.21.15.26:0/1845166552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700b5ef0 msgr2=0x7fa2701474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700b5ef0 0x7fa2701474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 -- 172.21.15.26:0/1845166552 >> v1:172.21.15.26:6789/0 conn(0x7fa2700a8c40 legacy=0x7fa2700aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 -- 172.21.15.26:0/1845166552 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27014ad60 con 0x7fa2700a7fc0 2022-07-20T02:31:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26effd700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa2701479c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fa260002f70 tx=0x7fa260006eb0).ready entity=mon.0 client_cookie=fac99e462a49a743 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26e7fc700 1 -- 172.21.15.26:0/1845166552 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa26000d570 con 0x7fa2700a7fc0 2022-07-20T02:31:26.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26e7fc700 1 -- 172.21.15.26:0/1845166552 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa260000e90 con 0x7fa2700a7fc0 2022-07-20T02:31:26.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa26e7fc700 1 -- 172.21.15.26:0/1845166552 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa26000d900 con 0x7fa2700a7fc0 2022-07-20T02:31:26.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- 172.21.15.26:0/1845166552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 msgr2=0x7fa2701479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa2701479c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fa260002f70 tx=0x7fa260006eb0).stop 2022-07-20T02:31:26.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- 172.21.15.26:0/1845166552 shutdown_connections 2022-07-20T02:31:26.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa2701479c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 --2- 172.21.15.26:0/1845166552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700b5ef0 0x7fa2701474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- 172.21.15.26:0/1845166552 >> 172.21.15.26:0/1845166552 conn(0x7fa270060310 msgr2=0x7fa2700b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:26.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- 172.21.15.26:0/1845166552 shutdown_connections 2022-07-20T02:31:26.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 -- 172.21.15.26:0/1845166552 wait complete. 2022-07-20T02:31:26.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.888+0000 7fa274ad9700 1 Processor -- start 2022-07-20T02:31:26.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- start start 2022-07-20T02:31:26.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700a8c40 0x7fa27005c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 0x7fa27015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa270146910 con 0x7fa2700a7fc0 2022-07-20T02:31:26.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa270064120 con 0x7fa2700b5ef0 2022-07-20T02:31:26.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa270067830 con 0x7fa2700a8c40 2022-07-20T02:31:26.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39304/0 (socket says 172.21.15.26:39304) 2022-07-20T02:31:26.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 -- 172.21.15.26:0/3256117546 learned_addr learned my addr 172.21.15.26:0/3256117546 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:26.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26effd700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700a8c40 0x7fa27005c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26ffff700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 0x7fa27015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 -- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700a8c40 msgr2=0x7fa27005c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700a8c40 0x7fa27005c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 -- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 msgr2=0x7fa27015a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:26.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 0x7fa27015a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:26.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 -- 172.21.15.26:0/3256117546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26000b1e0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26ffff700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 0x7fa27015a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:26.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26f7fe700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fa25800c000 tx=0x7fa258006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa258010780 con 0x7fa2700a7fc0 2022-07-20T02:31:26.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2580108e0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa270067ac0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa258010c40 con 0x7fa2700a7fc0 2022-07-20T02:31:26.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.892+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27015a2f0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27015a4f0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa25801c460 con 0x7fa2700a7fc0 2022-07-20T02:31:26.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa26cff9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 0x7fa254055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:26.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa25806c5b0 con 0x7fa2700a7fc0 2022-07-20T02:31:26.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa26effd700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 0x7fa254055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.896+0000 7fa26effd700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 0x7fa254055e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fa26001a000 tx=0x7fa260015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:26.904+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa27015a4f0 con 0x7fa2700a7fc0 2022-07-20T02:31:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.076+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa27015a4f0 con 0x7fa2700a7fc0 2022-07-20T02:31:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa26cff9700 1 -- 172.21.15.26:0/3256117546 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa27015a4f0 con 0x7fa2700a7fc0 2022-07-20T02:31:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 msgr2=0x7fa254055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 0x7fa254055e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fa26001a000 tx=0x7fa260015000).stop 2022-07-20T02:31:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 msgr2=0x7fa270067340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fa25800c000 tx=0x7fa258006df0).stop 2022-07-20T02:31:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 shutdown_connections 2022-07-20T02:31:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa254053950 0x7fa254055e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2700b5ef0 0x7fa27015a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2700a8c40 0x7fa27005c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 --2- 172.21.15.26:0/3256117546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa2700a7fc0 0x7fa270067340 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 >> 172.21.15.26:0/3256117546 conn(0x7fa270060310 msgr2=0x7fa270149f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 shutdown_connections 2022-07-20T02:31:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.084+0000 7fa274ad9700 1 -- 172.21.15.26:0/3256117546 wait complete. 2022-07-20T02:31:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:31:27.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:27.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 Processor -- start 2022-07-20T02:31:27.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 -- start start 2022-07-20T02:31:27.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ff00b5ef0 0x7f5ff01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff00b62d0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff014aac0 con 0x7f5ff00a8c40 2022-07-20T02:31:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff72e7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff014ad60 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff5ae4700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ff00a7fc0 0x7f5ff00aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38592/0 (socket says 172.21.15.26:38592) 2022-07-20T02:31:27.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff5ae4700 1 -- 172.21.15.26:0/564592820 learned_addr learned my addr 172.21.15.26:0/564592820 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:27.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff62e5700 1 --2- 172.21.15.26:0/564592820 >> v2:172.21.15.120:3300/0 conn(0x7f5ff00b5ef0 0x7f5ff01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020902202 0 0) 0x7f5ff00b62d0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fdc003260 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679787985 0 0) 0x7f5ff014aac0 con 0x7f5ff00a8c40 2022-07-20T02:31:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff00b62d0 con 0x7f5ff00a8c40 2022-07-20T02:31:27.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff62e5700 1 -- 172.21.15.26:0/564592820 >> v1:172.21.15.120:6789/0 conn(0x7f5ff00a8c40 legacy=0x7f5ff0151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff62e5700 1 -- 172.21.15.26:0/564592820 >> v1:172.21.15.26:6789/0 conn(0x7f5ff00a7fc0 legacy=0x7f5ff00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff62e5700 1 -- 172.21.15.26:0/564592820 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff014b000 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff62e5700 1 --2- 172.21.15.26:0/564592820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff01474d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5fec002f70 tx=0x7f5fec006df0).ready entity=mon.1 client_cookie=2ce8e3f1a7f729ac server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fec00d540 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec000e90 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.220+0000 7f5ff52e3700 1 -- 172.21.15.26:0/564592820 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fec00d890 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- 172.21.15.26:0/564592820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 msgr2=0x7f5ff01474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/564592820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff01474d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5fec002f70 tx=0x7f5fec006df0).stop 2022-07-20T02:31:27.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- 172.21.15.26:0/564592820 shutdown_connections 2022-07-20T02:31:27.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/564592820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff01474d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- 172.21.15.26:0/564592820 >> 172.21.15.26:0/564592820 conn(0x7f5ff0060310 msgr2=0x7f5ff00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:27.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- 172.21.15.26:0/564592820 shutdown_connections 2022-07-20T02:31:27.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- 172.21.15.26:0/564592820 wait complete. 2022-07-20T02:31:27.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 Processor -- start 2022-07-20T02:31:27.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- start start 2022-07-20T02:31:27.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00a8c40 0x7f5ff0062b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff0063070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff01599b0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00abd20 con 0x7f5ff00b5ef0 2022-07-20T02:31:27.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff72e7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00ab4d0 con 0x7f5ff00a8c40 2022-07-20T02:31:27.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff62e5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff5ae4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00a8c40 0x7f5ff0062b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff62e5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39320/0 (socket says 172.21.15.26:39320) 2022-07-20T02:31:27.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff62e5700 1 -- 172.21.15.26:0/3817243883 learned_addr learned my addr 172.21.15.26:0/3817243883 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:27.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.224+0000 7f5ff6ae6700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff0063070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 -- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00a8c40 msgr2=0x7f5ff0062b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00a8c40 0x7f5ff0062b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 -- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 msgr2=0x7f5ff0063070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff0063070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 -- 172.21.15.26:0/3817243883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec00b180 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff62e5700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f5fec002f70 tx=0x7f5fec014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fec016070 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec013850 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff00abf50 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fec013bb0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff0149c40 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff00ac3f0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.228+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fec025030 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.232+0000 7f5fe77fe700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 0x7f5fd0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.232+0000 7f5ff5ae4700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 0x7f5fd0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.232+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5fec06cf80 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.232+0000 7f5ff5ae4700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 0x7f5fd0055fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5fe000d000 tx=0x7f5fe0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.236+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fec043c30 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.404+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff014b2a0 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5fe77fe700 1 -- 172.21.15.26:0/3817243883 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f5fec023090 con 0x7f5ff00a7fc0 2022-07-20T02:31:27.407 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:27.408 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:27.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 msgr2=0x7f5fd0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 0x7f5fd0055fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5fe000d000 tx=0x7f5fe0009040).stop 2022-07-20T02:31:27.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 msgr2=0x7f5ff0149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f5fec002f70 tx=0x7f5fec014000).stop 2022-07-20T02:31:27.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 shutdown_connections 2022-07-20T02:31:27.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fd0053b00 0x7f5fd0055fc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff00b5ef0 0x7f5ff0063070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00a8c40 0x7f5ff0062b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 --2- 172.21.15.26:0/3817243883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ff00a7fc0 0x7f5ff0149750 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 >> 172.21.15.26:0/3817243883 conn(0x7f5ff0060310 msgr2=0x7f5ff005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:27.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 shutdown_connections 2022-07-20T02:31:27.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.408+0000 7f5ff72e7700 1 -- 172.21.15.26:0/3817243883 wait complete. 2022-07-20T02:31:27.419 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:27.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:31:24.761580+0000', 'modified': '2022-07-20T02:31:25.765269+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6309': {'gid': 6309, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/489399789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 489399789}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 489399789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:31:27.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:31:27.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:31:27.846 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6309,"want_state":"up:active","state":"up:active","rank_uptime":3.0747362630000001,"mdsmap_epoch":37,"osdmap_epoch":76,"osdmap_epoch_barrier":75,"uptime":7.4580790959999996} 2022-07-20T02:31:27.847 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6309, "mdsmap_epoch": 37, "osdmap_epoch": 76, "osdmap_epoch_barrier": 75, "rank_uptime": 3.074736263, "state": "up:active", "uptime": 7.458079096, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:31:27.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:27.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 Processor -- start 2022-07-20T02:31:27.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 -- start start 2022-07-20T02:31:27.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec1474e0 con 0x7feaec147000 2022-07-20T02:31:27.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaec0b4e40 con 0x7feaec05cfd0 2022-07-20T02:31:27.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.948+0000 7feaf083d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaec05d3b0 con 0x7feaec060f00 2022-07-20T02:31:27.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaead9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60846/0 (socket says 172.21.15.26:60846) 2022-07-20T02:31:27.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaead9d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59018/0 (socket says 172.21.15.26:59018) 2022-07-20T02:31:27.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaea59c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7feaec147000 0x7feaec0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:52818/0 (socket says 172.21.15.26:52818) 2022-07-20T02:31:27.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 -- 172.21.15.26:0/3177286837 learned_addr learned my addr 172.21.15.26:0/3177286837 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:27.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 -- 172.21.15.26:0/3177286837 >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 msgr2=0x7feaec143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:27.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 --2- 172.21.15.26:0/3177286837 >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 -- 172.21.15.26:0/3177286837 >> v1:172.21.15.120:6789/0 conn(0x7feaec147000 legacy=0x7feaec0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feae9d9b700 1 -- 172.21.15.26:0/3177286837 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836997920 0 0) 0x7feaec1474e0 con 0x7feaec147000 2022-07-20T02:31:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 -- 172.21.15.26:0/3177286837 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec1431e0 con 0x7feaec060f00 2022-07-20T02:31:27.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaead9d700 1 --2- 172.21.15.26:0/3177286837 >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:27.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaeb59e700 1 --2- 172.21.15.26:0/3177286837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7feae40032f0 tx=0x7feae4009290).ready entity=mon.2 client_cookie=50d16d53877cae22 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feae9d9b700 1 -- 172.21.15.26:0/3177286837 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feae400b8a0 con 0x7feaec060f00 2022-07-20T02:31:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feae9d9b700 1 -- 172.21.15.26:0/3177286837 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae400ba00 con 0x7feaec060f00 2022-07-20T02:31:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feae9d9b700 1 -- 172.21.15.26:0/3177286837 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feae400bb70 con 0x7feaec060f00 2022-07-20T02:31:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 -- 172.21.15.26:0/3177286837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 msgr2=0x7feaec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 --2- 172.21.15.26:0/3177286837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7feae40032f0 tx=0x7feae4009290).stop 2022-07-20T02:31:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 -- 172.21.15.26:0/3177286837 shutdown_connections 2022-07-20T02:31:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 --2- 172.21.15.26:0/3177286837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec060f00 0x7feaec13c9b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 --2- 172.21.15.26:0/3177286837 >> v2:172.21.15.120:3300/0 conn(0x7feaec05cfd0 0x7feaec143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 -- 172.21.15.26:0/3177286837 >> 172.21.15.26:0/3177286837 conn(0x7feaec0620c0 msgr2=0x7feaec05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 -- 172.21.15.26:0/3177286837 shutdown_connections 2022-07-20T02:31:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.952+0000 7feaf083d700 1 -- 172.21.15.26:0/3177286837 wait complete. 2022-07-20T02:31:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 Processor -- start 2022-07-20T02:31:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 -- start start 2022-07-20T02:31:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 0x7feaec1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec0aefb0 0x7feaec0b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec1455c0 con 0x7feaec147000 2022-07-20T02:31:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec0aeb30 con 0x7feaec060f00 2022-07-20T02:31:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaf083d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaec0ae2f0 con 0x7feaec0aefb0 2022-07-20T02:31:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaea59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaea59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39334/0 (socket says 172.21.15.26:39334) 2022-07-20T02:31:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaea59c700 1 -- 172.21.15.26:0/4006698716 learned_addr learned my addr 172.21.15.26:0/4006698716 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaead9d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 0x7feaec1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.956+0000 7feaeb59e700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec0aefb0 0x7feaec0b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 -- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec0aefb0 msgr2=0x7feaec0b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec0aefb0 0x7feaec0b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 -- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 msgr2=0x7feaec1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 0x7feaec1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 -- 172.21.15.26:0/4006698716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc00b180 con 0x7feaec147000 2022-07-20T02:31:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaead9d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 0x7feaec1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaea59c700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7feaec143110 tx=0x7feae0007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feae00070d0 con 0x7feaec147000 2022-07-20T02:31:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae0007230 con 0x7feaec147000 2022-07-20T02:31:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae4009530 con 0x7feaec147000 2022-07-20T02:31:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feae0005740 con 0x7feaec147000 2022-07-20T02:31:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaec0ae890 con 0x7feaec147000 2022-07-20T02:31:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaec0adff0 con 0x7feaec147000 2022-07-20T02:31:27.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feae001a060 con 0x7feaec147000 2022-07-20T02:31:27.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 0x7feac8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:27.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7feae006b420 con 0x7feaec147000 2022-07-20T02:31:27.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.960+0000 7feaead9d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 0x7feac8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:27.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.964+0000 7feaead9d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 0x7feac8055f60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7feadc00b000 tx=0x7feadc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:27.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:27.968+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feae00421e0 con 0x7feaec147000 2022-07-20T02:31:28.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.092+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaec143480 con 0x7feaec147000 2022-07-20T02:31:28.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feadbfff700 1 -- 172.21.15.26:0/4006698716 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16322 (secure 0 0 0) 0x7feae0018020 con 0x7feaec147000 2022-07-20T02:31:28.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:28.103 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":76,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:25.765356+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:28.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 msgr2=0x7feac8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 0x7feac8055f60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7feadc00b000 tx=0x7feadc006df0).stop 2022-07-20T02:31:28.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 msgr2=0x7feaec0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7feaec143110 tx=0x7feae0007b10).stop 2022-07-20T02:31:28.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 shutdown_connections 2022-07-20T02:31:28.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec147000 0x7feaec0b74f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feac8053aa0 0x7feac8055f60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaec0aefb0 0x7feaec0b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 --2- 172.21.15.26:0/4006698716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaec060f00 0x7feaec1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 >> 172.21.15.26:0/4006698716 conn(0x7feaec05cfd0 msgr2=0x7feaec13bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 shutdown_connections 2022-07-20T02:31:28.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.096+0000 7feaf083d700 1 -- 172.21.15.26:0/4006698716 wait complete. 2022-07-20T02:31:28.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:31:23.315398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:31:23.847170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:31:28.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:31:28.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 Processor -- start 2022-07-20T02:31:28.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 -- start start 2022-07-20T02:31:28.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95f80b5ef0 0x7f95f81474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a7fc0 0x7f95f81479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f80b62d0 con 0x7f95f80a8c40 2022-07-20T02:31:28.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f8147eb0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd92f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f814aac0 con 0x7f95f80a7fc0 2022-07-20T02:31:28.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd12e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f95f80a8c40 0x7f95f80aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38624/0 (socket says 172.21.15.26:38624) 2022-07-20T02:31:28.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fd12e700 1 -- 172.21.15.26:0/735432636 learned_addr learned my addr 172.21.15.26:0/735432636 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:28.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 --2- 172.21.15.26:0/735432636 >> v2:172.21.15.120:3300/0 conn(0x7f95f80b5ef0 0x7f95f81474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f7fff700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a7fc0 0x7f95f81479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f77fe700 1 -- 172.21.15.26:0/735432636 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159072875 0 0) 0x7f95f80b62d0 con 0x7f95f80a8c40 2022-07-20T02:31:28.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f77fe700 1 -- 172.21.15.26:0/735432636 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d8003260 con 0x7f95f80a8c40 2022-07-20T02:31:28.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 -- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a7fc0 msgr2=0x7f95f81479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:28.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a7fc0 0x7f95f81479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 -- 172.21.15.26:0/735432636 >> v1:172.21.15.26:6789/0 conn(0x7f95f80a8c40 legacy=0x7f95f80aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 -- 172.21.15.26:0/735432636 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f814ad60 con 0x7f95f80b5ef0 2022-07-20T02:31:28.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95fc92d700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80b5ef0 0x7f95f81474d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f95e0002f70 tx=0x7f95e0006df0).ready entity=mon.1 client_cookie=e618c8f0b200e52d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f77fe700 1 -- 172.21.15.26:0/735432636 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95e000d4a0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f77fe700 1 -- 172.21.15.26:0/735432636 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e0000e90 con 0x7f95f80b5ef0 2022-07-20T02:31:28.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.248+0000 7f95f77fe700 1 -- 172.21.15.26:0/735432636 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95e000d7f0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80b5ef0 msgr2=0x7f95f81474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80b5ef0 0x7f95f81474d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f95e0002f70 tx=0x7f95e0006df0).stop 2022-07-20T02:31:28.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- 172.21.15.26:0/735432636 shutdown_connections 2022-07-20T02:31:28.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a7fc0 0x7f95f81479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- 172.21.15.26:0/735432636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80b5ef0 0x7f95f81474d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- 172.21.15.26:0/735432636 >> 172.21.15.26:0/735432636 conn(0x7f95f8060310 msgr2=0x7f95f80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- 172.21.15.26:0/735432636 shutdown_connections 2022-07-20T02:31:28.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- 172.21.15.26:0/735432636 wait complete. 2022-07-20T02:31:28.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 Processor -- start 2022-07-20T02:31:28.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- start start 2022-07-20T02:31:28.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 0x7f95f80abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a8c40 0x7f95f8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8156670 con 0x7f95f80b5ef0 2022-07-20T02:31:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8156860 con 0x7f95f80a7fc0 2022-07-20T02:31:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.252+0000 7f95fd92f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8156b00 con 0x7f95f80a8c40 2022-07-20T02:31:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39350/0 (socket says 172.21.15.26:39350) 2022-07-20T02:31:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 -- 172.21.15.26:0/165516655 learned_addr learned my addr 172.21.15.26:0/165516655 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95f7fff700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a8c40 0x7f95f8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fc92d700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 0x7f95f80abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 -- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a8c40 msgr2=0x7f95f8062a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a8c40 0x7f95f8062a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 -- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 msgr2=0x7f95f80abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 0x7f95f80abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 -- 172.21.15.26:0/165516655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e000b180 con 0x7f95f80b5ef0 2022-07-20T02:31:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fc92d700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 0x7f95f80abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd12e700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f95f0001490 tx=0x7f95f0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95f0005ba0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f0005d00 con 0x7f95f80b5ef0 2022-07-20T02:31:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f8156da0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95f00072c0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f81574e0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.256+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f814b000 con 0x7f95f80b5ef0 2022-07-20T02:31:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.260+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f95f0010070 con 0x7f95f80b5ef0 2022-07-20T02:31:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.260+0000 7f95f5ffb700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 0x7f95dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.260+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f95f006ba60 con 0x7f95f80b5ef0 2022-07-20T02:31:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.260+0000 7f95fc92d700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 0x7f95dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.260+0000 7f95fc92d700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 0x7f95dc055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f95e001a000 tx=0x7f95e0014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.264+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95f0042710 con 0x7f95f80b5ef0 2022-07-20T02:31:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.392+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f95f80574e0 con 0x7f95f80b5ef0 2022-07-20T02:31:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.392+0000 7f95f5ffb700 1 -- 172.21.15.26:0/165516655 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v76) v1 ==== 110+0+10 (secure 0 0 0) 0x7f95f0045d30 con 0x7f95f80b5ef0 2022-07-20T02:31:28.398 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:31:28.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 msgr2=0x7f95dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 0x7f95dc055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f95e001a000 tx=0x7f95e0014000).stop 2022-07-20T02:31:28.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 msgr2=0x7f95f80aa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f95f0001490 tx=0x7f95f0007e20).stop 2022-07-20T02:31:28.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 shutdown_connections 2022-07-20T02:31:28.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95f80b5ef0 0x7f95f80aa080 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f95dc053b00 0x7f95dc055fc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95f80a8c40 0x7f95f8062a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 --2- 172.21.15.26:0/165516655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95f80a7fc0 0x7f95f80abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 >> 172.21.15.26:0/165516655 conn(0x7f95f8060310 msgr2=0x7f95f80a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 shutdown_connections 2022-07-20T02:31:28.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.396+0000 7f95fd92f700 1 -- 172.21.15.26:0/165516655 wait complete. 2022-07-20T02:31:28.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:31:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.528+0000 7fb899629700 1 Processor -- start 2022-07-20T02:31:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.528+0000 7fb899629700 1 -- start start 2022-07-20T02:31:28.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb894062710 0x7fb8941431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb8940b5b70 0x7fb8941403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940ba060 con 0x7fb8940b9c80 2022-07-20T02:31:28.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89412bee0 con 0x7fb8940b5b70 2022-07-20T02:31:28.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894061f80 con 0x7fb894062710 2022-07-20T02:31:28.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb892ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb8940b9c80 0x7fb8940b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38634/0 (socket says 172.21.15.26:38634) 2022-07-20T02:31:28.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb892ffd700 1 -- 172.21.15.26:0/2184341530 learned_addr learned my addr 172.21.15.26:0/2184341530 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:28.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 --2- 172.21.15.26:0/2184341530 >> v2:172.21.15.26:3300/0 conn(0x7fb8940b5b70 0x7fb8941403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8927fc700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb894062710 0x7fb8941431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 -- 172.21.15.26:0/2184341530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb894062710 msgr2=0x7fb8941431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:28.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb894062710 0x7fb8941431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 -- 172.21.15.26:0/2184341530 >> v1:172.21.15.26:6789/0 conn(0x7fb8940b9c80 legacy=0x7fb8940b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 -- 172.21.15.26:0/2184341530 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894062220 con 0x7fb8940b5b70 2022-07-20T02:31:28.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb8937fe700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb8941403e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7fb88c0032f0 tx=0x7fb88c0091d0).ready entity=mon.0 client_cookie=581d2ab0e5484b25 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb891ffb700 1 -- 172.21.15.26:0/2184341530 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c00b6c0 con 0x7fb8940b5b70 2022-07-20T02:31:28.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb891ffb700 1 -- 172.21.15.26:0/2184341530 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c00b820 con 0x7fb8940b5b70 2022-07-20T02:31:28.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb891ffb700 1 -- 172.21.15.26:0/2184341530 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c00b990 con 0x7fb8940b5b70 2022-07-20T02:31:28.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- 172.21.15.26:0/2184341530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 msgr2=0x7fb8941403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb8941403e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7fb88c0032f0 tx=0x7fb88c0091d0).stop 2022-07-20T02:31:28.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- 172.21.15.26:0/2184341530 shutdown_connections 2022-07-20T02:31:28.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb8941403e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 --2- 172.21.15.26:0/2184341530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb894062710 0x7fb8941431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- 172.21.15.26:0/2184341530 >> 172.21.15.26:0/2184341530 conn(0x7fb894066f70 msgr2=0x7fb89405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- 172.21.15.26:0/2184341530 shutdown_connections 2022-07-20T02:31:28.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 -- 172.21.15.26:0/2184341530 wait complete. 2022-07-20T02:31:28.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.532+0000 7fb899629700 1 Processor -- start 2022-07-20T02:31:28.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- start start 2022-07-20T02:31:28.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894062710 0x7fb894141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8940b9c80 0x7fb8941563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894155a80 con 0x7fb8940b5b70 2022-07-20T02:31:28.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894141670 con 0x7fb894062710 2022-07-20T02:31:28.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894155cb0 con 0x7fb8940b9c80 2022-07-20T02:31:28.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39362/0 (socket says 172.21.15.26:39362) 2022-07-20T02:31:28.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 -- 172.21.15.26:0/2203339506 learned_addr learned my addr 172.21.15.26:0/2203339506 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8937fe700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8940b9c80 0x7fb8941563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb892ffd700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894062710 0x7fb894141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 -- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8940b9c80 msgr2=0x7fb8941563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8940b9c80 0x7fb8941563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 -- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894062710 msgr2=0x7fb894141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894062710 0x7fb894141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 -- 172.21.15.26:0/2203339506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c009470 con 0x7fb8940b5b70 2022-07-20T02:31:28.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb8927fc700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fb88800c000 tx=0x7fb888006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888010780 con 0x7fb8940b5b70 2022-07-20T02:31:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8880108e0 con 0x7fb8940b5b70 2022-07-20T02:31:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb894155f50 con 0x7fb8940b5b70 2022-07-20T02:31:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888010c40 con 0x7fb8940b5b70 2022-07-20T02:31:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.536+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8940b3b50 con 0x7fb8940b5b70 2022-07-20T02:31:28.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8940b6650 con 0x7fb8940b5b70 2022-07-20T02:31:28.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb88801c460 con 0x7fb8940b5b70 2022-07-20T02:31:28.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb883fff700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 0x7fb870055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb88806c3d0 con 0x7fb8940b5b70 2022-07-20T02:31:28.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb892ffd700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 0x7fb870055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.540+0000 7fb892ffd700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 0x7fb870055e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb88400d000 tx=0x7fb884007be0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.544+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb888043130 con 0x7fb8940b5b70 2022-07-20T02:31:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.668+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8940574e0 con 0x7fb8940b5b70 2022-07-20T02:31:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb883fff700 1 -- 172.21.15.26:0/2203339506 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v76) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb888046750 con 0x7fb8940b5b70 2022-07-20T02:31:28.745 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:28.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 msgr2=0x7fb870055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 0x7fb870055e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb88400d000 tx=0x7fb884007be0).stop 2022-07-20T02:31:28.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 msgr2=0x7fb894147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fb88800c000 tx=0x7fb888006df0).stop 2022-07-20T02:31:28.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 shutdown_connections 2022-07-20T02:31:28.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8940b5b70 0x7fb894147190 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb870053980 0x7fb870055e40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8940b9c80 0x7fb8941563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 --2- 172.21.15.26:0/2203339506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894062710 0x7fb894141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 >> 172.21.15.26:0/2203339506 conn(0x7fb894066f70 msgr2=0x7fb894142330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 shutdown_connections 2022-07-20T02:31:28.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.672+0000 7fb899629700 1 -- 172.21.15.26:0/2203339506 wait complete. 2022-07-20T02:31:28.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:31:28.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.884+0000 7f91e351c700 1 Processor -- start 2022-07-20T02:31:28.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.884+0000 7f91e351c700 1 -- start start 2022-07-20T02:31:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0add30 0x7f91dc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f91dc0af6d0 0x7f91dc0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0aa1e0 con 0x7f91dc060950 2022-07-20T02:31:28.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc0ae110 con 0x7f91dc0af6d0 2022-07-20T02:31:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0a9f40 con 0x7f91dc0add30 2022-07-20T02:31:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f91dc0af6d0 0x7f91dc0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f91dc0af6d0 0x7f91dc0afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39370/0 (socket says 172.21.15.26:39370) 2022-07-20T02:31:28.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 -- 172.21.15.26:0/3276888509 learned_addr learned my addr 172.21.15.26:0/3276888509 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:28.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e251a700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0add30 0x7f91dc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 -- 172.21.15.26:0/3276888509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0add30 msgr2=0x7f91dc0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:28.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0add30 0x7f91dc0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 -- 172.21.15.26:0/3276888509 >> v1:172.21.15.120:6789/0 conn(0x7f91dc060950 legacy=0x7f91dc060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 -- 172.21.15.26:0/3276888509 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0b1110 con 0x7f91dc0af6d0 2022-07-20T02:31:28.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e2d1b700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc0af6d0 0x7f91dc0afb60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f91d40032f0 tx=0x7f91d400b530).ready entity=mon.0 client_cookie=7a94342545880952 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e1518700 1 -- 172.21.15.26:0/3276888509 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d4010660 con 0x7f91dc0af6d0 2022-07-20T02:31:28.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e1518700 1 -- 172.21.15.26:0/3276888509 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d40107c0 con 0x7f91dc0af6d0 2022-07-20T02:31:28.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e1518700 1 -- 172.21.15.26:0/3276888509 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d4010b20 con 0x7f91dc0af6d0 2022-07-20T02:31:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- 172.21.15.26:0/3276888509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc0af6d0 msgr2=0x7f91dc0afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc0af6d0 0x7f91dc0afb60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f91d40032f0 tx=0x7f91d400b530).stop 2022-07-20T02:31:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- 172.21.15.26:0/3276888509 shutdown_connections 2022-07-20T02:31:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc0af6d0 0x7f91dc0afb60 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 --2- 172.21.15.26:0/3276888509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0add30 0x7f91dc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- 172.21.15.26:0/3276888509 >> 172.21.15.26:0/3276888509 conn(0x7f91dc0b3050 msgr2=0x7f91dc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:28.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- 172.21.15.26:0/3276888509 shutdown_connections 2022-07-20T02:31:28.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.888+0000 7f91e351c700 1 -- 172.21.15.26:0/3276888509 wait complete. 2022-07-20T02:31:28.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 Processor -- start 2022-07-20T02:31:28.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- start start 2022-07-20T02:31:28.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91dc0add30 0x7f91dc0656e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 0x7f91dc13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc065bd0 con 0x7f91dc060950 2022-07-20T02:31:28.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc065010 con 0x7f91dc0add30 2022-07-20T02:31:28.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc13d530 con 0x7f91dc0af6d0 2022-07-20T02:31:28.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39372/0 (socket says 172.21.15.26:39372) 2022-07-20T02:31:28.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 -- 172.21.15.26:0/2901056672 learned_addr learned my addr 172.21.15.26:0/2901056672 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:28.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e2d1b700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 0x7f91dc13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e1d19700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91dc0add30 0x7f91dc0656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 -- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 msgr2=0x7f91dc13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 0x7f91dc13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 -- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91dc0add30 msgr2=0x7f91dc0656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:28.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91dc0add30 0x7f91dc0656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:28.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 -- 172.21.15.26:0/2901056672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d400b7d0 con 0x7f91dc060950 2022-07-20T02:31:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e2d1b700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 0x7f91dc13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e251a700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f91dc14def0 tx=0x7f91d8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d8011780 con 0x7f91dc060950 2022-07-20T02:31:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d80118e0 con 0x7f91dc060950 2022-07-20T02:31:28.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d8011c40 con 0x7f91dc060950 2022-07-20T02:31:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc13d7d0 con 0x7f91dc060950 2022-07-20T02:31:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.892+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc140040 con 0x7f91dc060950 2022-07-20T02:31:28.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc0b13b0 con 0x7f91dc060950 2022-07-20T02:31:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f91d801c460 con 0x7f91dc060950 2022-07-20T02:31:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91cb7fe700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 0x7f91bc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:28.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f91d806c510 con 0x7f91dc060950 2022-07-20T02:31:28.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91e1d19700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 0x7f91bc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:28.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.896+0000 7f91e1d19700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 0x7f91bc055e10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f91cc00f000 tx=0x7f91cc007ac0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:28.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:28.900+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91d8043270 con 0x7f91dc060950 2022-07-20T02:31:29.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.032+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f91dc0574e0 con 0x7f91dc060950 2022-07-20T02:31:29.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.032+0000 7f91cb7fe700 1 -- 172.21.15.26:0/2901056672 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+10 (secure 0 0 0) 0x7f91d8046890 con 0x7f91dc060950 2022-07-20T02:31:29.033 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:29.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 msgr2=0x7f91bc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 0x7f91bc055e10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f91cc00f000 tx=0x7f91cc007ac0).stop 2022-07-20T02:31:29.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 msgr2=0x7f91dc1460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f91dc14def0 tx=0x7f91d8008040).stop 2022-07-20T02:31:29.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 shutdown_connections 2022-07-20T02:31:29.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91bc053950 0x7f91bc055e10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91dc0af6d0 0x7f91dc13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91dc0add30 0x7f91dc0656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 --2- 172.21.15.26:0/2901056672 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91dc060950 0x7f91dc1460b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 >> 172.21.15.26:0/2901056672 conn(0x7f91dc0b3050 msgr2=0x7f91dc0b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:29.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 shutdown_connections 2022-07-20T02:31:29.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.036+0000 7f91e351c700 1 -- 172.21.15.26:0/2901056672 wait complete. 2022-07-20T02:31:29.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:31:29.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 Processor -- start 2022-07-20T02:31:29.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 -- start start 2022-07-20T02:31:29.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb000b2ca0 0x7fcb001414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb000657b0 0x7fcb001419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb000b14b0 con 0x7fcb00067240 2022-07-20T02:31:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb00067620 con 0x7fcb000b2ca0 2022-07-20T02:31:29.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.160+0000 7fcb05712700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb000aecf0 con 0x7fcb000657b0 2022-07-20T02:31:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcafffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcb00067240 0x7fcb000b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38660/0 (socket says 172.21.15.26:38660) 2022-07-20T02:31:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcafffff700 1 -- 172.21.15.26:0/4112500247 learned_addr learned my addr 172.21.15.26:0/4112500247 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/4112500247 >> v2:172.21.15.26:3300/0 conn(0x7fcb000b2ca0 0x7fcb001414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb04f11700 1 --2- 172.21.15.26:0/4112500247 >> v2:172.21.15.120:3300/0 conn(0x7fcb000657b0 0x7fcb001419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaff7fe700 1 -- 172.21.15.26:0/4112500247 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599596346 0 0) 0x7fcb000b14b0 con 0x7fcb00067240 2022-07-20T02:31:29.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaff7fe700 1 -- 172.21.15.26:0/4112500247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae0003260 con 0x7fcb00067240 2022-07-20T02:31:29.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 -- 172.21.15.26:0/4112500247 >> v2:172.21.15.120:3300/0 conn(0x7fcb000657b0 msgr2=0x7fcb001419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:29.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/4112500247 >> v2:172.21.15.120:3300/0 conn(0x7fcb000657b0 0x7fcb001419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 -- 172.21.15.26:0/4112500247 >> v1:172.21.15.26:6789/0 conn(0x7fcb00067240 legacy=0x7fcb000b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 -- 172.21.15.26:0/4112500247 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb000b1980 con 0x7fcb000b2ca0 2022-07-20T02:31:29.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/4112500247 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb000b2ca0 0x7fcb001414c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7fcae8002f70 tx=0x7fcae8006df0).ready entity=mon.0 client_cookie=d534892b83645442 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaff7fe700 1 -- 172.21.15.26:0/4112500247 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcae800d4f0 con 0x7fcb000b2ca0 2022-07-20T02:31:29.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaff7fe700 1 -- 172.21.15.26:0/4112500247 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae8000e90 con 0x7fcb000b2ca0 2022-07-20T02:31:29.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcaff7fe700 1 -- 172.21.15.26:0/4112500247 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcae800d840 con 0x7fcb000b2ca0 2022-07-20T02:31:29.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 -- 172.21.15.26:0/4112500247 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb000b2ca0 msgr2=0x7fcb001414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 --2- 172.21.15.26:0/4112500247 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb000b2ca0 0x7fcb001414c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7fcae8002f70 tx=0x7fcae8006df0).stop 2022-07-20T02:31:29.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 -- 172.21.15.26:0/4112500247 shutdown_connections 2022-07-20T02:31:29.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 --2- 172.21.15.26:0/4112500247 >> v2:172.21.15.120:3300/0 conn(0x7fcb000657b0 0x7fcb001419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 --2- 172.21.15.26:0/4112500247 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb000b2ca0 0x7fcb001414c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 -- 172.21.15.26:0/4112500247 >> 172.21.15.26:0/4112500247 conn(0x7fcb0005f840 msgr2=0x7fcb0005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:29.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 -- 172.21.15.26:0/4112500247 shutdown_connections 2022-07-20T02:31:29.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.164+0000 7fcb05712700 1 -- 172.21.15.26:0/4112500247 wait complete. 2022-07-20T02:31:29.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 Processor -- start 2022-07-20T02:31:29.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- start start 2022-07-20T02:31:29.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb000657b0 0x7fcb000b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb000b2ca0 0x7fcb00153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb001542d0 con 0x7fcb00067240 2022-07-20T02:31:29.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb000ad070 con 0x7fcb000b2ca0 2022-07-20T02:31:29.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb000a0000 con 0x7fcb000657b0 2022-07-20T02:31:29.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39388/0 (socket says 172.21.15.26:39388) 2022-07-20T02:31:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 -- 172.21.15.26:0/1825211187 learned_addr learned my addr 172.21.15.26:0/1825211187 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:29.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcafffff700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb000657b0 0x7fcb000b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb04f11700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb000b2ca0 0x7fcb00153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 -- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb000657b0 msgr2=0x7fcb000b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb000657b0 0x7fcb000b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 -- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb000b2ca0 msgr2=0x7fcb00153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb000b2ca0 0x7fcb00153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 -- 172.21.15.26:0/1825211187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae800b180 con 0x7fcb00067240 2022-07-20T02:31:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcaf77fe700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fcae8002f70 tx=0x7fcae8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcae800de60 con 0x7fcb00067240 2022-07-20T02:31:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae801e400 con 0x7fcb00067240 2022-07-20T02:31:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcae801e760 con 0x7fcb00067240 2022-07-20T02:31:29.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb000ad2a0 con 0x7fcb00067240 2022-07-20T02:31:29.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb000a0500 con 0x7fcb00067240 2022-07-20T02:31:29.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.168+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb000aafb0 con 0x7fcb00067240 2022-07-20T02:31:29.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.172+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcae801c070 con 0x7fcb00067240 2022-07-20T02:31:29.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.172+0000 7fcafdffb700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 0x7fcae4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.172+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcae806cab0 con 0x7fcb00067240 2022-07-20T02:31:29.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.172+0000 7fcafffff700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 0x7fcae4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.172+0000 7fcafffff700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 0x7fcae4055e40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fcaf000b000 tx=0x7fcaf0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.180+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcae80437d0 con 0x7fcb00067240 2022-07-20T02:31:29.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.304+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb000574e0 con 0x7fcb00067240 2022-07-20T02:31:29.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcafdffb700 1 -- 172.21.15.26:0/1825211187 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcae8023020 con 0x7fcb00067240 2022-07-20T02:31:29.307 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:29.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 msgr2=0x7fcae4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 0x7fcae4055e40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fcaf000b000 tx=0x7fcaf0009040).stop 2022-07-20T02:31:29.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 msgr2=0x7fcb001538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fcae8002f70 tx=0x7fcae8014000).stop 2022-07-20T02:31:29.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 shutdown_connections 2022-07-20T02:31:29.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb00067240 0x7fcb001538f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcae4053980 0x7fcae4055e40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb000b2ca0 0x7fcb00153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 --2- 172.21.15.26:0/1825211187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb000657b0 0x7fcb000b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 >> 172.21.15.26:0/1825211187 conn(0x7fcb0005f840 msgr2=0x7fcb001446f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:29.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 shutdown_connections 2022-07-20T02:31:29.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.308+0000 7fcb05712700 1 -- 172.21.15.26:0/1825211187 wait complete. 2022-07-20T02:31:29.323 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:31:29.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:29.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 Processor -- start 2022-07-20T02:31:29.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 -- start start 2022-07-20T02:31:29.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2d405cfd0 0x7fc2d4143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2d4060f00 0x7fc2d413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d41474e0 con 0x7fc2d4147000 2022-07-20T02:31:29.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d40b4e40 con 0x7fc2d4060f00 2022-07-20T02:31:29.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d9515700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d405d3b0 con 0x7fc2d405cfd0 2022-07-20T02:31:29.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d8d14700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2d4060f00 0x7fc2d413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2d405cfd0 0x7fc2d4143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2d4147000 0x7fc2d40b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38676/0 (socket says 172.21.15.26:38676) 2022-07-20T02:31:29.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d37fe700 1 -- 172.21.15.26:0/4109065602 learned_addr learned my addr 172.21.15.26:0/4109065602 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:29.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d2ffd700 1 -- 172.21.15.26:0/4109065602 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909536328 0 0) 0x7fc2d41474e0 con 0x7fc2d4147000 2022-07-20T02:31:29.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 -- 172.21.15.26:0/4109065602 >> v2:172.21.15.26:3300/0 conn(0x7fc2d4060f00 msgr2=0x7fc2d413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:29.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 --2- 172.21.15.26:0/4109065602 >> v2:172.21.15.26:3300/0 conn(0x7fc2d4060f00 0x7fc2d413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 -- 172.21.15.26:0/4109065602 >> v1:172.21.15.26:6789/0 conn(0x7fc2d4147000 legacy=0x7fc2d40b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 -- 172.21.15.26:0/4109065602 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d41431e0 con 0x7fc2d405cfd0 2022-07-20T02:31:29.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d3fff700 1 --2- 172.21.15.26:0/4109065602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d405cfd0 0x7fc2d4143bc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc2cc0032f0 tx=0x7fc2cc0091d0).ready entity=mon.1 client_cookie=3472a60154177a08 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d2ffd700 1 -- 172.21.15.26:0/4109065602 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2cc00b680 con 0x7fc2d405cfd0 2022-07-20T02:31:29.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d2ffd700 1 -- 172.21.15.26:0/4109065602 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc00b7e0 con 0x7fc2d405cfd0 2022-07-20T02:31:29.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.428+0000 7fc2d2ffd700 1 -- 172.21.15.26:0/4109065602 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2cc00b950 con 0x7fc2d405cfd0 2022-07-20T02:31:29.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- 172.21.15.26:0/4109065602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d405cfd0 msgr2=0x7fc2d4143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 --2- 172.21.15.26:0/4109065602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d405cfd0 0x7fc2d4143bc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc2cc0032f0 tx=0x7fc2cc0091d0).stop 2022-07-20T02:31:29.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- 172.21.15.26:0/4109065602 shutdown_connections 2022-07-20T02:31:29.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 --2- 172.21.15.26:0/4109065602 >> v2:172.21.15.26:3300/0 conn(0x7fc2d4060f00 0x7fc2d413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 --2- 172.21.15.26:0/4109065602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d405cfd0 0x7fc2d4143bc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- 172.21.15.26:0/4109065602 >> 172.21.15.26:0/4109065602 conn(0x7fc2d40620c0 msgr2=0x7fc2d405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:29.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- 172.21.15.26:0/4109065602 shutdown_connections 2022-07-20T02:31:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- 172.21.15.26:0/4109065602 wait complete. 2022-07-20T02:31:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 Processor -- start 2022-07-20T02:31:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.432+0000 7fc2d9515700 1 -- start start 2022-07-20T02:31:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2d4060f00 0x7fc2d41462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d4147000 0x7fc2d41381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4138bd0 con 0x7fc2d40b7990 2022-07-20T02:31:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40b0060 con 0x7fc2d4147000 2022-07-20T02:31:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40b0290 con 0x7fc2d4060f00 2022-07-20T02:31:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39404/0 (socket says 172.21.15.26:39404) 2022-07-20T02:31:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 -- 172.21.15.26:0/1567029977 learned_addr learned my addr 172.21.15.26:0/1567029977 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d3fff700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2d4060f00 0x7fc2d41462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d37fe700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d4147000 0x7fc2d41381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 -- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2d4060f00 msgr2=0x7fc2d41462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2d4060f00 0x7fc2d41462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 -- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d4147000 msgr2=0x7fc2d41381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d4147000 0x7fc2d41381f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 -- 172.21.15.26:0/1567029977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc009470 con 0x7fc2d40b7990 2022-07-20T02:31:29.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d8d14700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc2d413c8d0 tx=0x7fc2c4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c4011750 con 0x7fc2d40b7990 2022-07-20T02:31:29.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c40118b0 con 0x7fc2d40b7990 2022-07-20T02:31:29.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c4011c10 con 0x7fc2d40b7990 2022-07-20T02:31:29.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d40b0530 con 0x7fc2d40b7990 2022-07-20T02:31:29.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d40ae620 con 0x7fc2d40b7990 2022-07-20T02:31:29.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.436+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d4152610 con 0x7fc2d40b7990 2022-07-20T02:31:29.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.440+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2c4011d70 con 0x7fc2d40b7990 2022-07-20T02:31:29.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.440+0000 7fc2d17fa700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 0x7fc2b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:29.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.440+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc2c406c340 con 0x7fc2d40b7990 2022-07-20T02:31:29.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.440+0000 7fc2d3fff700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 0x7fc2b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.440+0000 7fc2d3fff700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 0x7fc2b8055ea0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fc2cc01b000 tx=0x7fc2cc016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.447+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2c4043100 con 0x7fc2d40b7990 2022-07-20T02:31:29.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.603+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d4143480 con 0x7fc2d40b7990 2022-07-20T02:31:29.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.603+0000 7fc2d17fa700 1 -- 172.21.15.26:0/1567029977 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fc2c4046720 con 0x7fc2d40b7990 2022-07-20T02:31:29.609 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:29.609 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:29.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 msgr2=0x7fc2b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 0x7fc2b8055ea0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fc2cc01b000 tx=0x7fc2cc016000).stop 2022-07-20T02:31:29.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 msgr2=0x7fc2d41386e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:29.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc2d413c8d0 tx=0x7fc2c4006df0).stop 2022-07-20T02:31:29.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 shutdown_connections 2022-07-20T02:31:29.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2d40b7990 0x7fc2d41386e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2b80539e0 0x7fc2b8055ea0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2d4147000 0x7fc2d41381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 --2- 172.21.15.26:0/1567029977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2d4060f00 0x7fc2d41462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:29.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 >> 172.21.15.26:0/1567029977 conn(0x7fc2d405cfd0 msgr2=0x7fc2d40b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:29.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 shutdown_connections 2022-07-20T02:31:29.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:29.607+0000 7fc2d9515700 1 -- 172.21.15.26:0/1567029977 wait complete. 2022-07-20T02:31:29.617 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:29.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:31:24.761580+0000', 'modified': '2022-07-20T02:31:25.765269+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6309': {'gid': 6309, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/489399789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 489399789}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 489399789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:31:29.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:31:29.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:31:30.062 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6309,"want_state":"up:active","state":"up:active","rank_uptime":5.2903002910000003,"mdsmap_epoch":37,"osdmap_epoch":76,"osdmap_epoch_barrier":75,"uptime":9.6736350180000006} 2022-07-20T02:31:30.063 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6309, "mdsmap_epoch": 37, "osdmap_epoch": 76, "osdmap_epoch_barrier": 75, "rank_uptime": 5.290300291, "state": "up:active", "uptime": 9.673635018, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:31:30.064 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:31:30.064 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:31:30.064 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:31:30.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.065 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:30.075 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:31:30.146 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:31:30.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:31:30.160 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:31:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:31:30.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.165 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:31:30.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:31:30.215 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:30.240 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:31:30.270 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:31:30.271 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:31:30.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:31:30.316+0000 7ff27ce99080 -1 init, newargv = 0x563637e28960 newargc=9ceph-fuse[22829]: starting ceph client 2022-07-20T02:31:30.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:31:30.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22829]: starting fuse 2022-07-20T02:31:30.332 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:31:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:31:30.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.336 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:31:30.349 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:31:30.350 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:31:30.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.352 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:31:30.353 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:31:30.353 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:31:30.354 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:31:30.355 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:31:30.355 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:31:30.356 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:31:30.357 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:31:30.358 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:31:30.358 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:31:30.359 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:31:30.360 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:31:30.361 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:31:30.362 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:31:30.363 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:31:30.363 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:31:30.364 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:31:30.365 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:31:30.366 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:31:30.366 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:31:30.367 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:31:30.368 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:31:30.369 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:31:30.450 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22829.asok 2022-07-20T02:31:30.452 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22829.asok 2022-07-20T02:31:30.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.470 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22829.asok status 2022-07-20T02:31:30.592 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:31:30.593 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:31:30.593 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:31:30.593 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:31:30.593 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:31:30.594 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:31:30.594 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:31:30.594 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22829", 2022-07-20T02:31:30.594 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:31:30.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:31:30.595 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:31:30.595 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:31:30.595 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 6340, 2022-07-20T02:31:30.596 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:31:30.596 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:31:30.596 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:31:30.597 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 6340 2022-07-20T02:31:30.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:31:30.597 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:31:30.597 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:31:30.598 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:31:30.598 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 682748160 2022-07-20T02:31:30.598 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:31:30.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:31:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:31:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:31:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:31:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 682748160 2022-07-20T02:31:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:31:30.600 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.6340 172.21.15.159:0/682748160", 2022-07-20T02:31:30.600 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/682748160", 2022-07-20T02:31:30.600 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:31:30.601 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 37, 2022-07-20T02:31:30.601 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 76, 2022-07-20T02:31:30.601 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:31:30.601 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:31:30.602 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:31:30.602 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:31:30.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.604 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:30.613 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:30.622 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:31:30.624 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:31:30.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:30.625 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:30.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:31:30.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 Processor -- start 2022-07-20T02:31:30.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- start start 2022-07-20T02:31:30.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 0x7f548c12c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:30.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c061e00 con 0x7f548c1468c0 2022-07-20T02:31:30.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c12cc40 con 0x7f548c061a20 2022-07-20T02:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c147040 con 0x7f548c1432e0 2022-07-20T02:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f54915c4700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f548c1468c0 0x7f548c0b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38692/0 (socket says 172.21.15.26:38692) 2022-07-20T02:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f54915c4700 1 -- 172.21.15.26:0/4027907966 learned_addr learned my addr 172.21.15.26:0/4027907966 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548bfff700 1 --2- 172.21.15.26:0/4027907966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 0x7f548c12c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:30.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171266155 0 0) 0x7f548c061e00 con 0x7f548c1468c0 2022-07-20T02:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470003260 con 0x7f548c1468c0 2022-07-20T02:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756372972 0 0) 0x7f548c12cc40 con 0x7f548c061a20 2022-07-20T02:31:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c061e00 con 0x7f548c061a20 2022-07-20T02:31:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548bfff700 1 -- 172.21.15.26:0/4027907966 >> v1:172.21.15.120:6789/0 conn(0x7f548c061a20 legacy=0x7f548c12c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548bfff700 1 -- 172.21.15.26:0/4027907966 >> v1:172.21.15.26:6789/0 conn(0x7f548c1468c0 legacy=0x7f548c0b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548bfff700 1 -- 172.21.15.26:0/4027907966 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c1472e0 con 0x7f548c1432e0 2022-07-20T02:31:30.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548bfff700 1 --2- 172.21.15.26:0/4027907966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 0x7f548c12c750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f547c002f70 tx=0x7f547c006df0).ready entity=mon.2 client_cookie=c5f8982cefbe5f91 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:30.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f547c00d5c0 con 0x7f548c1432e0 2022-07-20T02:31:30.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c000e90 con 0x7f548c1432e0 2022-07-20T02:31:30.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f548b7fe700 1 -- 172.21.15.26:0/4027907966 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f547c00d910 con 0x7f548c1432e0 2022-07-20T02:31:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- 172.21.15.26:0/4027907966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 msgr2=0x7f548c12c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 --2- 172.21.15.26:0/4027907966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 0x7f548c12c750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f547c002f70 tx=0x7f547c006df0).stop 2022-07-20T02:31:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- 172.21.15.26:0/4027907966 shutdown_connections 2022-07-20T02:31:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 --2- 172.21.15.26:0/4027907966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c1432e0 0x7f548c12c750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.803+0000 7f5493027700 1 -- 172.21.15.26:0/4027907966 >> 172.21.15.26:0/4027907966 conn(0x7f548c05faf0 msgr2=0x7f548c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- 172.21.15.26:0/4027907966 shutdown_connections 2022-07-20T02:31:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- 172.21.15.26:0/4027907966 wait complete. 2022-07-20T02:31:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 Processor -- start 2022-07-20T02:31:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- start start 2022-07-20T02:31:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 0x7f548c0ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c1468c0 0x7f548c0b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c0b7e90 con 0x7f548c1432e0 2022-07-20T02:31:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c153120 con 0x7f548c1468c0 2022-07-20T02:31:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5493027700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1533c0 con 0x7f548c061a20 2022-07-20T02:31:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5490dc3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 0x7f548c0ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39416/0 (socket says 172.21.15.26:39416) 2022-07-20T02:31:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f54915c4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c1468c0 0x7f548c0b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 -- 172.21.15.26:0/808588514 learned_addr learned my addr 172.21.15.26:0/808588514 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f5490dc3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 0x7f548c0ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:60930/0 (socket says 172.21.15.26:60930) 2022-07-20T02:31:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 -- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 msgr2=0x7f548c0ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 0x7f548c0ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 -- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c1468c0 msgr2=0x7f548c0b93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c1468c0 0x7f548c0b93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 -- 172.21.15.26:0/808588514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c00b180 con 0x7f548c1432e0 2022-07-20T02:31:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.807+0000 7f548bfff700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f547c002f70 tx=0x7f547c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f547c016070 con 0x7f548c1432e0 2022-07-20T02:31:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c0137d0 con 0x7f548c1432e0 2022-07-20T02:31:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c0b7b80 con 0x7f548c1432e0 2022-07-20T02:31:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f547c013b30 con 0x7f548c1432e0 2022-07-20T02:31:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c0b8860 con 0x7f548c1432e0 2022-07-20T02:31:30.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c153580 con 0x7f548c1432e0 2022-07-20T02:31:30.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f547c025030 con 0x7f548c1432e0 2022-07-20T02:31:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 0x7f5464055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f547c06cf10 con 0x7f548c1432e0 2022-07-20T02:31:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.811+0000 7f5490dc3700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 0x7f5464055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.815+0000 7f5490dc3700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 0x7f5464055ed0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f547400d000 tx=0x7f5474000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.819+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f547c043cd0 con 0x7f548c1432e0 2022-07-20T02:31:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f548c147580 con 0x7f548c1432e0 2022-07-20T02:31:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5489ffb700 1 -- 172.21.15.26:0/808588514 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f547c0472f0 con 0x7f548c1432e0 2022-07-20T02:31:30.953 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:30.953 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":76,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:31:25.765356+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:31:30.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 msgr2=0x7f5464055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 0x7f5464055ed0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f547400d000 tx=0x7f5474000bc0).stop 2022-07-20T02:31:30.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 msgr2=0x7f548c0aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:30.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f547c002f70 tx=0x7f547c014000).stop 2022-07-20T02:31:30.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 shutdown_connections 2022-07-20T02:31:30.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f548c1432e0 0x7f548c0aed20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5464053a10 0x7f5464055ed0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c1468c0 0x7f548c0b93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 --2- 172.21.15.26:0/808588514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f548c061a20 0x7f548c0ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:30.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 >> 172.21.15.26:0/808588514 conn(0x7f548c05faf0 msgr2=0x7f548c145b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:30.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.947+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 shutdown_connections 2022-07-20T02:31:30.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:30.951+0000 7f5493027700 1 -- 172.21.15.26:0/808588514 wait complete. 2022-07-20T02:31:30.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:31:23.315398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:31:23.847170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:31:30.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:31:31.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 Processor -- start 2022-07-20T02:31:31.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 -- start start 2022-07-20T02:31:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e540b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4e54064220 0x7f4e541434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e540b2e40 con 0x7f4e540b7d80 2022-07-20T02:31:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e540b4c20 con 0x7f4e54064220 2022-07-20T02:31:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e53fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540617a0 con 0x7f4e540b3060 2022-07-20T02:31:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4e54064220 0x7f4e541434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4e54064220 0x7f4e541434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39426/0 (socket says 172.21.15.26:39426) 2022-07-20T02:31:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 -- 172.21.15.26:0/1770275647 learned_addr learned my addr 172.21.15.26:0/1770275647 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e527fc700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e540b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 -- 172.21.15.26:0/1770275647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 msgr2=0x7f4e540b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e540b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 -- 172.21.15.26:0/1770275647 >> v1:172.21.15.120:6789/0 conn(0x7f4e540b7d80 legacy=0x7f4e540b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.087+0000 7f4e537fe700 1 -- 172.21.15.26:0/1770275647 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54060b20 con 0x7f4e54064220 2022-07-20T02:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e537fe700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e54064220 0x7f4e541434f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f4e540618e0 tx=0x7f4e4c009290).ready entity=mon.0 client_cookie=a4d06738c9127614 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e51ffb700 1 -- 172.21.15.26:0/1770275647 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4c00b860 con 0x7f4e54064220 2022-07-20T02:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e51ffb700 1 -- 172.21.15.26:0/1770275647 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c00b9c0 con 0x7f4e54064220 2022-07-20T02:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e51ffb700 1 -- 172.21.15.26:0/1770275647 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4c00bb30 con 0x7f4e54064220 2022-07-20T02:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- 172.21.15.26:0/1770275647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e54064220 msgr2=0x7f4e541434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e54064220 0x7f4e541434f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f4e540618e0 tx=0x7f4e4c009290).stop 2022-07-20T02:31:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- 172.21.15.26:0/1770275647 shutdown_connections 2022-07-20T02:31:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e54064220 0x7f4e541434f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 --2- 172.21.15.26:0/1770275647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e540b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- 172.21.15.26:0/1770275647 >> 172.21.15.26:0/1770275647 conn(0x7f4e5405f840 msgr2=0x7f4e5405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- 172.21.15.26:0/1770275647 shutdown_connections 2022-07-20T02:31:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- 172.21.15.26:0/1770275647 wait complete. 2022-07-20T02:31:31.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 Processor -- start 2022-07-20T02:31:31.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.091+0000 7f4e53fff700 1 -- start start 2022-07-20T02:31:31.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e541429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 0x7f4e54142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540afd20 con 0x7f4e540b7d80 2022-07-20T02:31:31.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540affc0 con 0x7f4e54064220 2022-07-20T02:31:31.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540b0260 con 0x7f4e540b3060 2022-07-20T02:31:31.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 0x7f4e54142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e527fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e541429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59120/0 (socket says 172.21.15.26:59120) 2022-07-20T02:31:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e52ffd700 1 -- 172.21.15.26:0/2296524791 learned_addr learned my addr 172.21.15.26:0/2296524791 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 -- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 msgr2=0x7f4e541429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e541429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 -- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 msgr2=0x7f4e54145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 -- 172.21.15.26:0/2296524791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c009530 con 0x7f4e540b7d80 2022-07-20T02:31:31.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e52ffd700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:31.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e527fc700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e541429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:31.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e537fe700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 0x7f4e54142e90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f4e540618e0 tx=0x7f4e4c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4c0032f0 con 0x7f4e540b7d80 2022-07-20T02:31:31.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c021440 con 0x7f4e540b7d80 2022-07-20T02:31:31.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4c0217a0 con 0x7f4e540b7d80 2022-07-20T02:31:31.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e540b0500 con 0x7f4e540b7d80 2022-07-20T02:31:31.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e54143380 con 0x7f4e540b7d80 2022-07-20T02:31:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.095+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e540b0690 con 0x7f4e540b7d80 2022-07-20T02:31:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.099+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e4c00f070 con 0x7f4e540b7d80 2022-07-20T02:31:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.099+0000 7f4e43fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 0x7f4e30055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.099+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4e4c06d720 con 0x7f4e540b7d80 2022-07-20T02:31:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.099+0000 7f4e52ffd700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 0x7f4e30055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.099+0000 7f4e52ffd700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 0x7f4e30055f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f4e4400d000 tx=0x7f4e44009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.103+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e4c0444e0 con 0x7f4e540b7d80 2022-07-20T02:31:31.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.227+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4e54060dc0 con 0x7f4e540b7d80 2022-07-20T02:31:31.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.227+0000 7f4e43fff700 1 -- 172.21.15.26:0/2296524791 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v76) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4e4c015070 con 0x7f4e540b7d80 2022-07-20T02:31:31.230 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:31:31.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 msgr2=0x7f4e30055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 0x7f4e30055f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f4e4400d000 tx=0x7f4e44009040).stop 2022-07-20T02:31:31.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 msgr2=0x7f4e54142e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 0x7f4e54142e90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f4e540618e0 tx=0x7f4e4c016000).stop 2022-07-20T02:31:31.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 shutdown_connections 2022-07-20T02:31:31.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e540b7d80 0x7f4e54142e90 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e30053a70 0x7f4e30055f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e540b3060 0x7f4e541429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 --2- 172.21.15.26:0/2296524791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e54064220 0x7f4e54145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 >> 172.21.15.26:0/2296524791 conn(0x7f4e5405f840 msgr2=0x7f4e5405cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 shutdown_connections 2022-07-20T02:31:31.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.231+0000 7f4e53fff700 1 -- 172.21.15.26:0/2296524791 wait complete. 2022-07-20T02:31:31.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:31:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 Processor -- start 2022-07-20T02:31:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 -- start start 2022-07-20T02:31:31.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd63c140320 0x7fd63c140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd63c05cf80 0x7fd63c143040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c0ab530 con 0x7fd63c147020 2022-07-20T02:31:31.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c0b1ce0 con 0x7fd63c05cf80 2022-07-20T02:31:31.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.371+0000 7fd6428d2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c05cc90 con 0x7fd63c140320 2022-07-20T02:31:31.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd63c05cf80 0x7fd63c143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd63c05cf80 0x7fd63c143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39438/0 (socket says 172.21.15.26:39438) 2022-07-20T02:31:31.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6418d0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd63c140320 0x7fd63c140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 -- 172.21.15.26:0/407988981 learned_addr learned my addr 172.21.15.26:0/407988981 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 -- 172.21.15.26:0/407988981 >> v2:172.21.15.120:3300/0 conn(0x7fd63c140320 msgr2=0x7fd63c140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:31.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 --2- 172.21.15.26:0/407988981 >> v2:172.21.15.120:3300/0 conn(0x7fd63c140320 0x7fd63c140700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 -- 172.21.15.26:0/407988981 >> v1:172.21.15.120:6789/0 conn(0x7fd63c147020 legacy=0x7fd63c05c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 -- 172.21.15.26:0/407988981 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c0aab50 con 0x7fd63c05cf80 2022-07-20T02:31:31.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6420d1700 1 --2- 172.21.15.26:0/407988981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c05cf80 0x7fd63c143040 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fd6340032f0 tx=0x7fd634009290).ready entity=mon.0 client_cookie=eae48dbb5a9badc7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6408ce700 1 -- 172.21.15.26:0/407988981 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63400b870 con 0x7fd63c05cf80 2022-07-20T02:31:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6408ce700 1 -- 172.21.15.26:0/407988981 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63400b9d0 con 0x7fd63c05cf80 2022-07-20T02:31:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6408ce700 1 -- 172.21.15.26:0/407988981 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63400bb40 con 0x7fd63c05cf80 2022-07-20T02:31:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 -- 172.21.15.26:0/407988981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c05cf80 msgr2=0x7fd63c143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 --2- 172.21.15.26:0/407988981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c05cf80 0x7fd63c143040 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fd6340032f0 tx=0x7fd634009290).stop 2022-07-20T02:31:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 -- 172.21.15.26:0/407988981 shutdown_connections 2022-07-20T02:31:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 --2- 172.21.15.26:0/407988981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c05cf80 0x7fd63c143040 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 --2- 172.21.15.26:0/407988981 >> v2:172.21.15.120:3300/0 conn(0x7fd63c140320 0x7fd63c140700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 -- 172.21.15.26:0/407988981 >> 172.21.15.26:0/407988981 conn(0x7fd63c060310 msgr2=0x7fd63c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 -- 172.21.15.26:0/407988981 shutdown_connections 2022-07-20T02:31:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.375+0000 7fd6428d2700 1 -- 172.21.15.26:0/407988981 wait complete. 2022-07-20T02:31:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 Processor -- start 2022-07-20T02:31:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- start start 2022-07-20T02:31:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd63c05cf80 0x7fd63c13d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd63c147020 0x7fd63c1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c142960 con 0x7fd63c140320 2022-07-20T02:31:31.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c1413d0 con 0x7fd63c05cf80 2022-07-20T02:31:31.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c156910 con 0x7fd63c147020 2022-07-20T02:31:31.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39442/0 (socket says 172.21.15.26:39442) 2022-07-20T02:31:31.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 -- 172.21.15.26:0/1899395747 learned_addr learned my addr 172.21.15.26:0/1899395747 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6418d0700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd63c05cf80 0x7fd63c13d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6420d1700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd63c147020 0x7fd63c1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 -- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd63c147020 msgr2=0x7fd63c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd63c147020 0x7fd63c1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 -- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd63c05cf80 msgr2=0x7fd63c13d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd63c05cf80 0x7fd63c13d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 -- 172.21.15.26:0/1899395747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009530 con 0x7fd63c140320 2022-07-20T02:31:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6410cf700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fd63c147450 tx=0x7fd638007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd638007030 con 0x7fd63c140320 2022-07-20T02:31:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c142b90 con 0x7fd63c140320 2022-07-20T02:31:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c1437c0 con 0x7fd63c140320 2022-07-20T02:31:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638007190 con 0x7fd63c140320 2022-07-20T02:31:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd638006510 con 0x7fd63c140320 2022-07-20T02:31:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.379+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c1566e0 con 0x7fd63c140320 2022-07-20T02:31:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.383+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd638010030 con 0x7fd63c140320 2022-07-20T02:31:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.383+0000 7fd62effd700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 0x7fd61c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.383+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd63806bb60 con 0x7fd63c140320 2022-07-20T02:31:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.383+0000 7fd6418d0700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 0x7fd61c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.383+0000 7fd6418d0700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 0x7fd61c055ea0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd63000b000 tx=0x7fd630008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.391+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd638042880 con 0x7fd63c140320 2022-07-20T02:31:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.515+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd63c0ab530 con 0x7fd63c140320 2022-07-20T02:31:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd62effd700 1 -- 172.21.15.26:0/1899395747 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v76) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd63c0ab530 con 0x7fd63c140320 2022-07-20T02:31:31.522 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 msgr2=0x7fd61c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 0x7fd61c055ea0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd63000b000 tx=0x7fd630008040).stop 2022-07-20T02:31:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 msgr2=0x7fd63c140ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fd63c147450 tx=0x7fd638007c00).stop 2022-07-20T02:31:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 shutdown_connections 2022-07-20T02:31:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd63c140320 0x7fd63c140ee0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd61c0539e0 0x7fd61c055ea0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd63c147020 0x7fd63c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 --2- 172.21.15.26:0/1899395747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd63c05cf80 0x7fd63c13d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 >> 172.21.15.26:0/1899395747 conn(0x7fd63c060310 msgr2=0x7fd63c13e060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 shutdown_connections 2022-07-20T02:31:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.519+0000 7fd6428d2700 1 -- 172.21.15.26:0/1899395747 wait complete. 2022-07-20T02:31:31.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:31:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 Processor -- start 2022-07-20T02:31:31.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- start start 2022-07-20T02:31:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 0x7fdae8140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae8147a00 con 0x7fdae805cfd0 2022-07-20T02:31:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae805d3b0 con 0x7fdae8146e80 2022-07-20T02:31:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae81435b0 con 0x7fdae80b1160 2022-07-20T02:31:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae77fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdae805cfd0 0x7fdae81407c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38730/0 (socket says 172.21.15.26:38730) 2022-07-20T02:31:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae77fe700 1 -- 172.21.15.26:0/3646793736 learned_addr learned my addr 172.21.15.26:0/3646793736 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae6ffd700 1 --2- 172.21.15.26:0/3646793736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 0x7fdae8140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae67fc700 1 -- 172.21.15.26:0/3646793736 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605514905 0 0) 0x7fdae8147a00 con 0x7fdae805cfd0 2022-07-20T02:31:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae67fc700 1 -- 172.21.15.26:0/3646793736 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad4003260 con 0x7fdae805cfd0 2022-07-20T02:31:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae6ffd700 1 -- 172.21.15.26:0/3646793736 >> v1:172.21.15.120:6789/0 conn(0x7fdae8146e80 legacy=0x7fdae81472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae6ffd700 1 -- 172.21.15.26:0/3646793736 >> v1:172.21.15.26:6789/0 conn(0x7fdae805cfd0 legacy=0x7fdae81407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae6ffd700 1 -- 172.21.15.26:0/3646793736 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80b4bc0 con 0x7fdae80b1160 2022-07-20T02:31:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae6ffd700 1 --2- 172.21.15.26:0/3646793736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 0x7fdae8140ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdadc002f70 tx=0x7fdadc006df0).ready entity=mon.2 client_cookie=149e00674fe16b38 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae67fc700 1 -- 172.21.15.26:0/3646793736 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdadc00d570 con 0x7fdae80b1160 2022-07-20T02:31:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae67fc700 1 -- 172.21.15.26:0/3646793736 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc000e90 con 0x7fdae80b1160 2022-07-20T02:31:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdae67fc700 1 -- 172.21.15.26:0/3646793736 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdadc00d8c0 con 0x7fdae80b1160 2022-07-20T02:31:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- 172.21.15.26:0/3646793736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 msgr2=0x7fdae8140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 --2- 172.21.15.26:0/3646793736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 0x7fdae8140ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdadc002f70 tx=0x7fdadc006df0).stop 2022-07-20T02:31:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- 172.21.15.26:0/3646793736 shutdown_connections 2022-07-20T02:31:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 --2- 172.21.15.26:0/3646793736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae80b1160 0x7fdae8140ed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- 172.21.15.26:0/3646793736 >> 172.21.15.26:0/3646793736 conn(0x7fdae8066f70 msgr2=0x7fdae805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- 172.21.15.26:0/3646793736 shutdown_connections 2022-07-20T02:31:31.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.643+0000 7fdaee173700 1 -- 172.21.15.26:0/3646793736 wait complete. 2022-07-20T02:31:31.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 Processor -- start 2022-07-20T02:31:31.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 -- start start 2022-07-20T02:31:31.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae805cfd0 0x7fdae8139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae80b1160 0x7fdae8155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80b9720 con 0x7fdae8146e80 2022-07-20T02:31:31.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80b8f40 con 0x7fdae80b1160 2022-07-20T02:31:31.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.647+0000 7fdaee173700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80b91e0 con 0x7fdae805cfd0 2022-07-20T02:31:31.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae6ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae80b1160 0x7fdae8155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39460/0 (socket says 172.21.15.26:39460) 2022-07-20T02:31:31.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 -- 172.21.15.26:0/1293414333 learned_addr learned my addr 172.21.15.26:0/1293414333 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae77fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae805cfd0 0x7fdae8139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 -- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae805cfd0 msgr2=0x7fdae8139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae805cfd0 0x7fdae8139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 -- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae80b1160 msgr2=0x7fdae8155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae80b1160 0x7fdae8155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 -- 172.21.15.26:0/1293414333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc00b180 con 0x7fdae8146e80 2022-07-20T02:31:31.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae7fff700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fdae8060d70 tx=0x7fdae0007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae000ce80 con 0x7fdae8146e80 2022-07-20T02:31:31.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae000f470 con 0x7fdae8146e80 2022-07-20T02:31:31.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae000f7d0 con 0x7fdae8146e80 2022-07-20T02:31:31.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae80b9480 con 0x7fdae8146e80 2022-07-20T02:31:31.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae80b8bc0 con 0x7fdae8146e80 2022-07-20T02:31:31.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae80574e0 con 0x7fdae8146e80 2022-07-20T02:31:31.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.651+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdae0010070 con 0x7fdae8146e80 2022-07-20T02:31:31.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.655+0000 7fdae4ff9700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 0x7fdac8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.655+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdae006b420 con 0x7fdae8146e80 2022-07-20T02:31:31.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.655+0000 7fdae77fe700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 0x7fdac8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.655+0000 7fdae77fe700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 0x7fdac8055f60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fdad800d000 tx=0x7fdad8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.659+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdae00421e0 con 0x7fdae8146e80 2022-07-20T02:31:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.783+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdae8060030 con 0x7fdae8146e80 2022-07-20T02:31:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdae4ff9700 1 -- 172.21.15.26:0/1293414333 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdae0045800 con 0x7fdae8146e80 2022-07-20T02:31:31.793 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 msgr2=0x7fdac8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 0x7fdac8055f60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fdad800d000 tx=0x7fdad8009040).stop 2022-07-20T02:31:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 msgr2=0x7fdae81563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fdae8060d70 tx=0x7fdae0007f50).stop 2022-07-20T02:31:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 shutdown_connections 2022-07-20T02:31:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdae8146e80 0x7fdae81563e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdac8053aa0 0x7fdac8055f60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae80b1160 0x7fdae8155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 --2- 172.21.15.26:0/1293414333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdae805cfd0 0x7fdae8139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 >> 172.21.15.26:0/1293414333 conn(0x7fdae8066f70 msgr2=0x7fdae8143a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 shutdown_connections 2022-07-20T02:31:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.787+0000 7fdaee173700 1 -- 172.21.15.26:0/1293414333 wait complete. 2022-07-20T02:31:31.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:31:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.907+0000 7f646b04d700 1 Processor -- start 2022-07-20T02:31:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646b04d700 1 -- start start 2022-07-20T02:31:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646b04d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f646405c8c0 0x7f64640af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646b04d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64640b00e0 con 0x7f64640abd80 2022-07-20T02:31:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646b04d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646405cca0 con 0x7f64640b0690 2022-07-20T02:31:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646b04d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64640acf00 con 0x7f646405c8c0 2022-07-20T02:31:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a84c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64640abd80 0x7f646414bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38748/0 (socket says 172.21.15.26:38748) 2022-07-20T02:31:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a84c700 1 -- 172.21.15.26:0/3102550534 learned_addr learned my addr 172.21.15.26:0/3102550534 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f646405c8c0 0x7f64640af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363012439 0 0) 0x7f64640b00e0 con 0x7f64640abd80 2022-07-20T02:31:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c003260 con 0x7f64640abd80 2022-07-20T02:31:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110479450 0 0) 0x7f644c003260 con 0x7f64640abd80 2022-07-20T02:31:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f646405c8c0 msgr2=0x7f64640af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f646405c8c0 0x7f64640af150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 >> v1:172.21.15.120:6789/0 conn(0x7f64640b0690 legacy=0x7f64640ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64640afe40 con 0x7f64640abd80 2022-07-20T02:31:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f645c001660 con 0x7f64640abd80 2022-07-20T02:31:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 >> v1:172.21.15.26:6789/0 conn(0x7f64640abd80 legacy=0x7f646414bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f644c003780 0x7f644c003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c004500 0x7f644c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f644c004e70 0x7f644c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640acf00 con 0x7f644c003780 2022-07-20T02:31:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c00bca0 con 0x7f644c004500 2022-07-20T02:31:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c00be60 con 0x7f644c004e70 2022-07-20T02:31:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f646405c8c0 0x7f64640af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f644c003780 0x7f644c003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646984a700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c004500 0x7f644c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a84c700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f644c004e70 0x7f644c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 -- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f644c004e70 msgr2=0x7f644c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f644c004e70 0x7f644c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 -- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c004500 msgr2=0x7f644c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c004500 0x7f644c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 -- 172.21.15.26:0/3102550534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c003260 con 0x7f644c003780 2022-07-20T02:31:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f646a04b700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f644c003780 0x7f644c003b60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f645400a000 tx=0x7f6454008040).ready entity=mon.0 client_cookie=852a4f8e24f2fd2f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6454013670 con 0x7f644c003780 2022-07-20T02:31:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64540137d0 con 0x7f644c003780 2022-07-20T02:31:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.911+0000 7f6469049700 1 -- 172.21.15.26:0/3102550534 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6454013b30 con 0x7f644c003780 2022-07-20T02:31:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- 172.21.15.26:0/3102550534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f644c003780 msgr2=0x7f644c003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 --2- 172.21.15.26:0/3102550534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f644c003780 0x7f644c003b60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f645400a000 tx=0x7f6454008040).stop 2022-07-20T02:31:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646a84c700 1 -- 172.21.15.26:0/3102550534 reap_dead start 2022-07-20T02:31:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- 172.21.15.26:0/3102550534 shutdown_connections 2022-07-20T02:31:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- 172.21.15.26:0/3102550534 >> 172.21.15.26:0/3102550534 conn(0x7f6464060310 msgr2=0x7f64640b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- 172.21.15.26:0/3102550534 shutdown_connections 2022-07-20T02:31:31.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- 172.21.15.26:0/3102550534 wait complete. 2022-07-20T02:31:31.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 Processor -- start 2022-07-20T02:31:31.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- start start 2022-07-20T02:31:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c003780 0x7f646413dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6464153fd0 0x7f646413c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6464150190 con 0x7f6464145490 2022-07-20T02:31:31.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6464064050 con 0x7f644c003780 2022-07-20T02:31:31.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646b04d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64640642f0 con 0x7f6464153fd0 2022-07-20T02:31:31.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.915+0000 7f646984a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39478/0 (socket says 172.21.15.26:39478) 2022-07-20T02:31:31.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 -- 172.21.15.26:0/624848776 learned_addr learned my addr 172.21.15.26:0/624848776 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646a04b700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c003780 0x7f646413dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646a84c700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6464153fd0 0x7f646413c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 -- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6464153fd0 msgr2=0x7f646413c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6464153fd0 0x7f646413c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 -- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c003780 msgr2=0x7f646413dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:31.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c003780 0x7f646413dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 -- 172.21.15.26:0/624848776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464064590 con 0x7f6464145490 2022-07-20T02:31:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646984a700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f646000d000 tx=0x7f6460007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6460017070 con 0x7f6464145490 2022-07-20T02:31:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6460002210 con 0x7f6464145490 2022-07-20T02:31:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6464063cd0 con 0x7f6464145490 2022-07-20T02:31:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6460002570 con 0x7f6464145490 2022-07-20T02:31:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6464145b30 con 0x7f6464145490 2022-07-20T02:31:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64640aca50 con 0x7f6464145490 2022-07-20T02:31:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.919+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64600017d0 con 0x7f6464145490 2022-07-20T02:31:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.923+0000 7f64537fe700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 0x7f6444055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.923+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f646006c080 con 0x7f6464145490 2022-07-20T02:31:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.923+0000 7f646a04b700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 0x7f6444055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.923+0000 7f646a04b700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 0x7f6444055ff0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f645401b000 tx=0x7f6454008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:31.927+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6460042e40 con 0x7f6464145490 2022-07-20T02:31:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.055+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64640574e0 con 0x7f6464145490 2022-07-20T02:31:32.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.055+0000 7f64537fe700 1 -- 172.21.15.26:0/624848776 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6460046460 con 0x7f6464145490 2022-07-20T02:31:32.059 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:31:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 msgr2=0x7f6444055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 0x7f6444055ff0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f645401b000 tx=0x7f6454008040).stop 2022-07-20T02:31:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 msgr2=0x7f6464153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f646000d000 tx=0x7f6460007d80).stop 2022-07-20T02:31:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 shutdown_connections 2022-07-20T02:31:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6464145490 0x7f6464153ae0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6444053b30 0x7f6444055ff0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6464153fd0 0x7f646413c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 --2- 172.21.15.26:0/624848776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f644c003780 0x7f646413dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 >> 172.21.15.26:0/624848776 conn(0x7f6464060310 msgr2=0x7f64640b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:32.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 shutdown_connections 2022-07-20T02:31:32.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.059+0000 7f646b04d700 1 -- 172.21.15.26:0/624848776 wait complete. 2022-07-20T02:31:32.074 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:31:32.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.187+0000 7fa37caf6700 1 Processor -- start 2022-07-20T02:31:32.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.187+0000 7fa37caf6700 1 -- start start 2022-07-20T02:31:32.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa378147000 0x7fa378143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3781474e0 con 0x7fa37805cfd0 2022-07-20T02:31:32.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3780b4e40 con 0x7fa378147000 2022-07-20T02:31:32.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37805d3b0 con 0x7fa378060f00 2022-07-20T02:31:32.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa378147000 0x7fa378143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa378147000 0x7fa378143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39486/0 (socket says 172.21.15.26:39486) 2022-07-20T02:31:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 -- 172.21.15.26:0/4174167166 learned_addr learned my addr 172.21.15.26:0/4174167166 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:32.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa377fff700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 -- 172.21.15.26:0/4174167166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 msgr2=0x7fa37813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:32.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 -- 172.21.15.26:0/4174167166 >> v1:172.21.15.120:6789/0 conn(0x7fa37805cfd0 legacy=0x7fa3780b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 -- 172.21.15.26:0/4174167166 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3781431e0 con 0x7fa378147000 2022-07-20T02:31:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa376ffd700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa378143bc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fa368002f70 tx=0x7fa368006eb0).ready entity=mon.0 client_cookie=95b8e45be44f7bb7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa3767fc700 1 -- 172.21.15.26:0/4174167166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36800d570 con 0x7fa378147000 2022-07-20T02:31:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa3767fc700 1 -- 172.21.15.26:0/4174167166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368000e90 con 0x7fa378147000 2022-07-20T02:31:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa3767fc700 1 -- 172.21.15.26:0/4174167166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36800d900 con 0x7fa378147000 2022-07-20T02:31:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- 172.21.15.26:0/4174167166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 msgr2=0x7fa378143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa378143bc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fa368002f70 tx=0x7fa368006eb0).stop 2022-07-20T02:31:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- 172.21.15.26:0/4174167166 shutdown_connections 2022-07-20T02:31:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 --2- 172.21.15.26:0/4174167166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa378143bc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- 172.21.15.26:0/4174167166 >> 172.21.15.26:0/4174167166 conn(0x7fa3780620c0 msgr2=0x7fa37805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- 172.21.15.26:0/4174167166 shutdown_connections 2022-07-20T02:31:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.191+0000 7fa37caf6700 1 -- 172.21.15.26:0/4174167166 wait complete. 2022-07-20T02:31:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 Processor -- start 2022-07-20T02:31:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- start start 2022-07-20T02:31:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa37813b580 0x7fa3780ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37813cac0 con 0x7fa378147000 2022-07-20T02:31:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378153070 con 0x7fa37813b580 2022-07-20T02:31:32.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37813ccf0 con 0x7fa378060f00 2022-07-20T02:31:32.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39492/0 (socket says 172.21.15.26:39492) 2022-07-20T02:31:32.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 -- 172.21.15.26:0/727862648 learned_addr learned my addr 172.21.15.26:0/727862648 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:32.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa377fff700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa37813b580 0x7fa3780ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa3777fe700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 -- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 msgr2=0x7fa37813d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 -- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa37813b580 msgr2=0x7fa3780ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa37813b580 0x7fa3780ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 -- 172.21.15.26:0/727862648 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36800b1e0 con 0x7fa378147000 2022-07-20T02:31:32.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa3777fe700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:31:32.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa376ffd700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7fa368002f70 tx=0x7fa368015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:32.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa368011950 con 0x7fa378147000 2022-07-20T02:31:32.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368011ab0 con 0x7fa378147000 2022-07-20T02:31:32.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa368011e10 con 0x7fa378147000 2022-07-20T02:31:32.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa378153310 con 0x7fa378147000 2022-07-20T02:31:32.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.195+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3780b7990 con 0x7fa378147000 2022-07-20T02:31:32.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa378143480 con 0x7fa378147000 2022-07-20T02:31:32.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa36801a040 con 0x7fa378147000 2022-07-20T02:31:32.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa374ff9700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 0x7fa358055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:32.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa36806d010 con 0x7fa378147000 2022-07-20T02:31:32.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa3777fe700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 0x7fa358055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.199+0000 7fa3777fe700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 0x7fa358055fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fa36c00b000 tx=0x7fa36c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.203+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa368043dd0 con 0x7fa378147000 2022-07-20T02:31:32.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.367+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3780574e0 con 0x7fa378147000 2022-07-20T02:31:32.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.367+0000 7fa374ff9700 1 -- 172.21.15.26:0/727862648 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fa368023070 con 0x7fa378147000 2022-07-20T02:31:32.385 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 msgr2=0x7fa358055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 0x7fa358055fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fa36c00b000 tx=0x7fa36c009040).stop 2022-07-20T02:31:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 msgr2=0x7fa37813b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7fa368002f70 tx=0x7fa368015000).stop 2022-07-20T02:31:32.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 shutdown_connections 2022-07-20T02:31:32.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa378147000 0x7fa37813b090 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa358053b00 0x7fa358055fc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa37813b580 0x7fa3780ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 --2- 172.21.15.26:0/727862648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa378060f00 0x7fa37813d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:32.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 >> 172.21.15.26:0/727862648 conn(0x7fa37805cfd0 msgr2=0x7fa3780b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:32.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 shutdown_connections 2022-07-20T02:31:32.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:32.371+0000 7fa37caf6700 1 -- 172.21.15.26:0/727862648 wait complete. 2022-07-20T02:31:32.397 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:32.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:31:24.761580+0000', 'modified': '2022-07-20T02:31:25.765269+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6309': {'gid': 6309, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/489399789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 489399789}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 489399789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:31:32.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:31:32.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:31:32.823 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6309,"want_state":"up:active","state":"up:active","rank_uptime":8.0521227060000005,"mdsmap_epoch":37,"osdmap_epoch":76,"osdmap_epoch_barrier":75,"uptime":12.435463742} 2022-07-20T02:31:32.824 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6309, "mdsmap_epoch": 37, "osdmap_epoch": 76, "osdmap_epoch_barrier": 75, "rank_uptime": 8.052122706, "state": "up:active", "uptime": 12.435463742, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:31:32.825 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:31:32.825 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:31:32.826 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:31:32.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:32.827 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:32.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:31:32.922 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:31:32.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:31:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:31:32.941 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:31:32.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:32.942 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:31:32.993 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:31:32.993 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:32.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:31:33.013 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:31:33.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:31:33.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:31:33.063+0000 7f669a974080 -1 init, newargv = 0x55f9055b7860 newargc=9 2022-07-20T02:31:33.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22515]: starting ceph client 2022-07-20T02:31:33.071 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:31:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:31:33.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:33.074 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:31:33.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22515]: starting fuse 2022-07-20T02:31:34.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:31:34.201 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:31:34.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:31:34.260 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:31:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:31:34.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:34.262 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:31:34.268 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:31:34.269 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:31:34.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:34.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:31:34.271 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:31:34.272 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:31:34.273 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:31:34.274 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:31:34.275 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:31:34.275 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:31:34.276 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:31:34.277 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:31:34.278 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:31:34.279 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:31:34.279 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:31:34.280 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:31:34.281 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:31:34.282 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:31:34.283 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:31:34.283 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:31:34.285 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:31:34.286 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:31:34.287 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:31:34.288 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:31:34.289 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:31:34.290 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:31:34.355 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22515.asok 2022-07-20T02:31:34.364 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22515.asok 2022-07-20T02:31:34.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:34.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22515.asok status 2022-07-20T02:31:34.475 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:31:34.475 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:31:34.475 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:31:34.475 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:31:34.476 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:31:34.501 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:31:34.502 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:31:34.503 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22515", 2022-07-20T02:31:34.511 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:31:34.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:31:34.512 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:31:34.512 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:31:34.513 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 6519, 2022-07-20T02:31:34.513 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:31:34.513 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:31:34.522 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:31:34.522 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 6519 2022-07-20T02:31:34.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:31:34.523 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:31:34.523 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:31:34.523 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:31:34.523 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 4112403225 2022-07-20T02:31:34.524 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:31:34.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:31:34.524 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:31:34.525 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:31:34.526 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:31:34.526 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 4112403225 2022-07-20T02:31:34.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:31:34.527 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.6519 172.21.15.176:0/4112403225", 2022-07-20T02:31:34.528 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/4112403225", 2022-07-20T02:31:34.528 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:31:34.529 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 37, 2022-07-20T02:31:34.530 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 76, 2022-07-20T02:31:34.531 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:31:34.531 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:31:34.532 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:31:34.532 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:31:34.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:34.534 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:34.535 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:34.556 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:31:34.556 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:31:34.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:34.558 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:34.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:34.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 Processor -- start 2022-07-20T02:31:34.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 -- start start 2022-07-20T02:31:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7febdc060f00 0x7febdc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:34.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc1474e0 con 0x7febdc05cfd0 2022-07-20T02:31:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0b4e40 con 0x7febdc147000 2022-07-20T02:31:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe2615700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc05d3b0 con 0x7febdc060f00 2022-07-20T02:31:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe1613700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7febdc05cfd0 0x7febdc0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38778/0 (socket says 172.21.15.26:38778) 2022-07-20T02:31:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe1613700 1 -- 172.21.15.26:0/830698217 learned_addr learned my addr 172.21.15.26:0/830698217 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.739+0000 7febe1e14700 1 --2- 172.21.15.26:0/830698217 >> v2:172.21.15.120:3300/0 conn(0x7febdc060f00 0x7febdc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261188313 0 0) 0x7febdc0b4e40 con 0x7febdc147000 2022-07-20T02:31:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4003260 con 0x7febdc147000 2022-07-20T02:31:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292248260 0 0) 0x7febdc1474e0 con 0x7febdc05cfd0 2022-07-20T02:31:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc0b4e40 con 0x7febdc05cfd0 2022-07-20T02:31:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe1e14700 1 -- 172.21.15.26:0/830698217 >> v1:172.21.15.120:6789/0 conn(0x7febdc147000 legacy=0x7febdc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe1e14700 1 -- 172.21.15.26:0/830698217 >> v1:172.21.15.26:6789/0 conn(0x7febdc05cfd0 legacy=0x7febdc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe1e14700 1 -- 172.21.15.26:0/830698217 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc1431e0 con 0x7febdc060f00 2022-07-20T02:31:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe1e14700 1 --2- 172.21.15.26:0/830698217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc060f00 0x7febdc143bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7febd40032f0 tx=0x7febd40091d0).ready entity=mon.1 client_cookie=e0a681ff24aac948 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:34.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd400b600 con 0x7febdc060f00 2022-07-20T02:31:34.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd400b760 con 0x7febdc060f00 2022-07-20T02:31:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febcbfff700 1 -- 172.21.15.26:0/830698217 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd400b8d0 con 0x7febdc060f00 2022-07-20T02:31:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- 172.21.15.26:0/830698217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc060f00 msgr2=0x7febdc143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 --2- 172.21.15.26:0/830698217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc060f00 0x7febdc143bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7febd40032f0 tx=0x7febd40091d0).stop 2022-07-20T02:31:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- 172.21.15.26:0/830698217 shutdown_connections 2022-07-20T02:31:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 --2- 172.21.15.26:0/830698217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc060f00 0x7febdc143bc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- 172.21.15.26:0/830698217 >> 172.21.15.26:0/830698217 conn(0x7febdc0620c0 msgr2=0x7febdc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- 172.21.15.26:0/830698217 shutdown_connections 2022-07-20T02:31:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- 172.21.15.26:0/830698217 wait complete. 2022-07-20T02:31:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 Processor -- start 2022-07-20T02:31:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.743+0000 7febe2615700 1 -- start start 2022-07-20T02:31:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febdc060f00 0x7febdc152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 0x7febdc145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc140a50 con 0x7febdc147000 2022-07-20T02:31:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc146300 con 0x7febdc142230 2022-07-20T02:31:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc1427b0 con 0x7febdc060f00 2022-07-20T02:31:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39506/0 (socket says 172.21.15.26:39506) 2022-07-20T02:31:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 -- 172.21.15.26:0/2989112522 learned_addr learned my addr 172.21.15.26:0/2989112522 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe1e14700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 0x7febdc145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe1613700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febdc060f00 0x7febdc152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 -- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febdc060f00 msgr2=0x7febdc152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febdc060f00 0x7febdc152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 -- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 msgr2=0x7febdc145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 0x7febdc145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 -- 172.21.15.26:0/2989112522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4009470 con 0x7febdc147000 2022-07-20T02:31:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe1e14700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 0x7febdc145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:31:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe0e12700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7febcc001110 tx=0x7febcc007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc012400 con 0x7febdc147000 2022-07-20T02:31:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc012560 con 0x7febdc147000 2022-07-20T02:31:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc152270 con 0x7febdc147000 2022-07-20T02:31:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc0128c0 con 0x7febdc147000 2022-07-20T02:31:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc153140 con 0x7febdc147000 2022-07-20T02:31:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.747+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc143480 con 0x7febdc147000 2022-07-20T02:31:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.751+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7febcc019030 con 0x7febdc147000 2022-07-20T02:31:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.751+0000 7febca7fc700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 0x7febbc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.751+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7febcc06b420 con 0x7febdc147000 2022-07-20T02:31:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.751+0000 7febe1613700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 0x7febbc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.751+0000 7febe1613700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 0x7febbc055fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7febd800d000 tx=0x7febd8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.755+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febcc042160 con 0x7febdc147000 2022-07-20T02:31:34.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.915+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febdc0574e0 con 0x7febdc147000 2022-07-20T02:31:34.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.915+0000 7febca7fc700 1 -- 172.21.15.26:0/2989112522 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7febcc017020 con 0x7febdc147000 2022-07-20T02:31:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:34.929 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:34.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 msgr2=0x7febbc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 0x7febbc055fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7febd800d000 tx=0x7febd8009040).stop 2022-07-20T02:31:34.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 msgr2=0x7febdc140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7febcc001110 tx=0x7febcc007ad0).stop 2022-07-20T02:31:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 shutdown_connections 2022-07-20T02:31:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febdc147000 0x7febdc140f00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febbc053b00 0x7febbc055fc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febdc142230 0x7febdc145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 --2- 172.21.15.26:0/2989112522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febdc060f00 0x7febdc152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 >> 172.21.15.26:0/2989112522 conn(0x7febdc05cfd0 msgr2=0x7febdc0b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 shutdown_connections 2022-07-20T02:31:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:34.919+0000 7febe2615700 1 -- 172.21.15.26:0/2989112522 wait complete. 2022-07-20T02:31:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:34.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:31:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 Processor -- start 2022-07-20T02:31:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 -- start start 2022-07-20T02:31:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4fbc0b5ef0 0x7f4fbc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fbc0b62d0 con 0x7f4fbc0a7fc0 2022-07-20T02:31:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fbc14aac0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc24b2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc14ad60 con 0x7f4fbc0a8c40 2022-07-20T02:31:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc0caf700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4fbc0a7fc0 0x7f4fbc0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38792/0 (socket says 172.21.15.26:38792) 2022-07-20T02:31:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc0caf700 1 -- 172.21.15.26:0/659031608 learned_addr learned my addr 172.21.15.26:0/659031608 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:31:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/659031608 >> v2:172.21.15.26:3300/0 conn(0x7f4fbc0b5ef0 0x7f4fbc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc1cb1700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fb3fff700 1 -- 172.21.15.26:0/659031608 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575779195 0 0) 0x7f4fbc0b62d0 con 0x7f4fbc0a7fc0 2022-07-20T02:31:35.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fb3fff700 1 -- 172.21.15.26:0/659031608 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa0003260 con 0x7f4fbc0a7fc0 2022-07-20T02:31:35.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc14b0700 1 -- 172.21.15.26:0/659031608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 msgr2=0x7f4fbc0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:31:35.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc14b0700 1 -- 172.21.15.26:0/659031608 >> v1:172.21.15.26:6789/0 conn(0x7f4fbc0a7fc0 legacy=0x7f4fbc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.043+0000 7f4fc14b0700 1 -- 172.21.15.26:0/659031608 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc14b000 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc1474d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f4fac002f70 tx=0x7f4fac006df0).ready entity=mon.0 client_cookie=20f38285222e6e8b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:35.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fb3fff700 1 -- 172.21.15.26:0/659031608 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fac00d4a0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fb3fff700 1 -- 172.21.15.26:0/659031608 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac000e90 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fb3fff700 1 -- 172.21.15.26:0/659031608 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fac00d7f0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- 172.21.15.26:0/659031608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 msgr2=0x7f4fbc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc1474d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f4fac002f70 tx=0x7f4fac006df0).stop 2022-07-20T02:31:35.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- 172.21.15.26:0/659031608 shutdown_connections 2022-07-20T02:31:35.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/659031608 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc1474d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- 172.21.15.26:0/659031608 >> 172.21.15.26:0/659031608 conn(0x7f4fbc060310 msgr2=0x7f4fbc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:35.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- 172.21.15.26:0/659031608 shutdown_connections 2022-07-20T02:31:35.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- 172.21.15.26:0/659031608 wait complete. 2022-07-20T02:31:35.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 Processor -- start 2022-07-20T02:31:35.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.047+0000 7f4fc24b2700 1 -- start start 2022-07-20T02:31:35.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fbc0a7fc0 0x7f4fbc143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc1472f0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0ac3e0 con 0x7f4fbc0a7fc0 2022-07-20T02:31:35.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc1572f0 con 0x7f4fbc0a8c40 2022-07-20T02:31:35.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39522/0 (socket says 172.21.15.26:39522) 2022-07-20T02:31:35.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 -- 172.21.15.26:0/2026145714 learned_addr learned my addr 172.21.15.26:0/2026145714 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:31:35.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc0caf700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fbc0a7fc0 0x7f4fbc143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 -- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 msgr2=0x7f4fbc159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 -- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fbc0a7fc0 msgr2=0x7f4fbc143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fbc0a7fc0 0x7f4fbc143760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 -- 172.21.15.26:0/2026145714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac00b180 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc1cb1700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f4fb400c000 tx=0x7f4fb4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:35.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fb4011750 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb40118b0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fb4011c10 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fbc147520 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.051+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc143fe0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fbc14b2a0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4fb4011d70 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fb27fc700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 0x7f4fa4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:31:35.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4fb406c380 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 0x7f4fa4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:31:35.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.055+0000 7f4fc14b0700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 0x7f4fa4055fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f4fac01a000 tx=0x7f4fac014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:31:35.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.059+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4fb4043140 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fbc0574e0 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fb27fc700 1 -- 172.21.15.26:0/2026145714 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f4fb4046760 con 0x7f4fbc0b5ef0 2022-07-20T02:31:35.225 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:31:35.225 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:31:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 msgr2=0x7f4fa4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 0x7f4fa4055fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f4fac01a000 tx=0x7f4fac014000).stop 2022-07-20T02:31:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 msgr2=0x7f4fbc159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:31:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f4fb400c000 tx=0x7f4fb4006df0).stop 2022-07-20T02:31:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 shutdown_connections 2022-07-20T02:31:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fbc0b5ef0 0x7f4fbc159f50 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4fa4053b00 0x7f4fa4055fc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4fbc0a8c40 0x7f4fbc159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 --2- 172.21.15.26:0/2026145714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fbc0a7fc0 0x7f4fbc143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:31:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 >> 172.21.15.26:0/2026145714 conn(0x7f4fbc060310 msgr2=0x7f4fbc0a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:31:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 shutdown_connections 2022-07-20T02:31:35.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:31:35.223+0000 7f4fc24b2700 1 -- 172.21.15.26:0/2026145714 wait complete. 2022-07-20T02:31:35.239 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:31:35.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_cap_revoke_eviction_timeout 30.0 2022-07-20T02:31:35.428 INFO:teuthology.orchestra.run.smithi026.stdout:{"success":"mds_cap_revoke_eviction_timeout = '30.000000' "} 2022-07-20T02:31:35.429 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_cap_revoke_eviction_timeout = '30.000000' " } 2022-07-20T02:31:35.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:35.431 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:35.431 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:31:35.448 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:31:35.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:31:35.450 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:31:35.451 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:31:35.451 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:31:35.452 DEBUG:teuthology.orchestra.run.smithi159:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2022-07-20T02:31:35.453 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:31:35.454 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:31:35.455 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content2'"'"') 2022-07-20T02:31:35.455 DEBUG:teuthology.orchestra.run.smithi159:> while True: 2022-07-20T02:31:35.456 DEBUG:teuthology.orchestra.run.smithi159:> time.sleep(1) 2022-07-20T02:31:35.457 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:31:35.494 DEBUG:teuthology.orchestra.run.smithi159:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T02:31:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-20T02:31:35.524 INFO:teuthology.orchestra.run.smithi159.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2022-07-20T02:31:36.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T02:31:36.550 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T02:31:36.552 DEBUG:tasks.cephfs.mount:File background_file became visible from 0 after 1s 2022-07-20T02:31:36.553 DEBUG:teuthology.orchestra.run.smithi176:> sudo ls /home/ubuntu/cephtest/mnt.1/background_file 2022-07-20T02:31:36.598 INFO:teuthology.orchestra.run.smithi176.stdout:/home/ubuntu/cephtest/mnt.1/background_file 2022-07-20T02:31:36.599 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2022-07-20T02:31:36.599 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:31:36.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:31:36.616 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:31:36.617 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:31:36.645 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:31:36.646 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:31:36.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:31:36.647 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:31:36.647 DEBUG:teuthology.orchestra.run.smithi176:> import time 2022-07-20T02:31:36.648 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:31:36.648 DEBUG:teuthology.orchestra.run.smithi176:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2022-07-20T02:31:36.648 DEBUG:teuthology.orchestra.run.smithi176:> try: 2022-07-20T02:31:36.649 DEBUG:teuthology.orchestra.run.smithi176:> while True: 2022-07-20T02:31:36.649 DEBUG:teuthology.orchestra.run.smithi176:> os.write(fd, b'"'"'content'"'"') 2022-07-20T02:31:36.650 DEBUG:teuthology.orchestra.run.smithi176:> time.sleep(1) 2022-07-20T02:31:36.650 DEBUG:teuthology.orchestra.run.smithi176:> if not False: 2022-07-20T02:31:36.650 DEBUG:teuthology.orchestra.run.smithi176:> break 2022-07-20T02:31:36.651 DEBUG:teuthology.orchestra.run.smithi176:> except IOError as e: 2022-07-20T02:31:36.651 DEBUG:teuthology.orchestra.run.smithi176:> pass 2022-07-20T02:31:36.653 DEBUG:teuthology.orchestra.run.smithi176:> os.close(fd) 2022-07-20T02:31:36.653 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:31:47.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:47.192 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:47.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:31:47.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:16.583 INFO:tasks.cephfs.test_misc:cap_waiter waited 39.92083168029785s 2022-07-20T02:32:16.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist ls --format=json 2022-07-20T02:32:16.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 Processor -- start 2022-07-20T02:32:16.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 -- start start 2022-07-20T02:32:16.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffaf80b5ef0 0x7ffaf81474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ffaf80a7fc0 0x7ffaf81479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80b62d0 con 0x7ffaf80a8c40 2022-07-20T02:32:16.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf8147eb0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffafd4b1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf814aac0 con 0x7ffaf80b5ef0 2022-07-20T02:32:16.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ffaf80a7fc0 0x7ffaf81479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ffaf80a7fc0 0x7ffaf81479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39528/0 (socket says 172.21.15.26:39528) 2022-07-20T02:32:16.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 -- 172.21.15.26:0/2722066772 learned_addr learned my addr 172.21.15.26:0/2722066772 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf7fff700 1 --2- 172.21.15.26:0/2722066772 >> v2:172.21.15.120:3300/0 conn(0x7ffaf80b5ef0 0x7ffaf81474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 -- 172.21.15.26:0/2722066772 >> v2:172.21.15.120:3300/0 conn(0x7ffaf80b5ef0 msgr2=0x7ffaf81474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/2722066772 >> v2:172.21.15.120:3300/0 conn(0x7ffaf80b5ef0 0x7ffaf81474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 -- 172.21.15.26:0/2722066772 >> v1:172.21.15.120:6789/0 conn(0x7ffaf80a8c40 legacy=0x7ffaf80aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 -- 172.21.15.26:0/2722066772 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf814ad60 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/2722066772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf81479c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7ffae8002f70 tx=0x7ffae8006eb0).ready entity=mon.0 client_cookie=fb34b0e193c29b24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf6ffd700 1 -- 172.21.15.26:0/2722066772 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae800d5c0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf6ffd700 1 -- 172.21.15.26:0/2722066772 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8000e90 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.698+0000 7ffaf6ffd700 1 -- 172.21.15.26:0/2722066772 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae800d950 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/2722066772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 msgr2=0x7ffaf81479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/2722066772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf81479c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7ffae8002f70 tx=0x7ffae8006eb0).stop 2022-07-20T02:32:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/2722066772 shutdown_connections 2022-07-20T02:32:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/2722066772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf81479c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/2722066772 >> v2:172.21.15.120:3300/0 conn(0x7ffaf80b5ef0 0x7ffaf81474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/2722066772 >> 172.21.15.26:0/2722066772 conn(0x7ffaf8060310 msgr2=0x7ffaf80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/2722066772 shutdown_connections 2022-07-20T02:32:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/2722066772 wait complete. 2022-07-20T02:32:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 Processor -- start 2022-07-20T02:32:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- start start 2022-07-20T02:32:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 0x7ffaf8147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf80b5ef0 0x7ffaf8147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8147d70 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8156560 con 0x7ffaf80b5ef0 2022-07-20T02:32:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80ab430 con 0x7ffaf80a8c40 2022-07-20T02:32:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39532/0 (socket says 172.21.15.26:39532) 2022-07-20T02:32:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 -- 172.21.15.26:0/1631705586 learned_addr learned my addr 172.21.15.26:0/1631705586 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafccb0700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf80b5ef0 0x7ffaf8147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 0x7ffaf8147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 -- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 msgr2=0x7ffaf8147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 0x7ffaf8147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 -- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf80b5ef0 msgr2=0x7ffaf8147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf80b5ef0 0x7ffaf8147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 -- 172.21.15.26:0/1631705586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae800b1e0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 0x7ffaf8147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf7fff700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7ffaec00c000 tx=0x7ffaec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffaec010780 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec0108e0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.702+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80aba40 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffaec010c40 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80b4bd0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf814b000 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffaec01c460 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf57fa700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 0x7ffad8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7ffaec06c710 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 0x7ffad8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.706+0000 7ffaf77fe700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 0x7ffad8055f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ffae801a000 tx=0x7ffae8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.710+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffaec0432c0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.846+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist ls", "format": "json"} v 0) v1 -- 0x7ffaf80574e0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.846+0000 7ffaf57fa700 1 -- 172.21.15.26:0/1631705586 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist ls", "format": "json"}]=0 listed 1 entries v77) v1 ==== 98+0+80 (secure 0 0 0) 0x7ffaec0468e0 con 0x7ffaf80a7fc0 2022-07-20T02:32:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:16.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 msgr2=0x7ffad8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 0x7ffad8055f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ffae801a000 tx=0x7ffae8015000).stop 2022-07-20T02:32:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 msgr2=0x7ffaf80b46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7ffaec00c000 tx=0x7ffaec006df0).stop 2022-07-20T02:32:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 shutdown_connections 2022-07-20T02:32:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad8053a70 0x7ffad8055f30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf80b5ef0 0x7ffaf8147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf80a8c40 0x7ffaf8147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 --2- 172.21.15.26:0/1631705586 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffaf80a7fc0 0x7ffaf80b46e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 >> 172.21.15.26:0/1631705586 conn(0x7ffaf8060310 msgr2=0x7ffaf805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 shutdown_connections 2022-07-20T02:32:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:16.850+0000 7ffafd4b1700 1 -- 172.21.15.26:0/1631705586 wait complete. 2022-07-20T02:32:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:listed 1 entries 2022-07-20T02:32:16.889 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr":"172.21.15.159:0/682748160","until":"2022-07-20T03:32:10.385916+0000"}] 2022-07-20T02:32:16.926 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T02:32:16.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:16.927 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:16.927 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:32:16.957 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T02:32:16.958 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:16.959 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:32:16.960 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:32:16.960 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:17.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:17.034 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:17.035 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:17.057 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:32:17.059 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:32:17.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:32:17.061 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:17.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 Processor -- start 2022-07-20T02:32:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 -- start start 2022-07-20T02:32:17.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f85740b5ef0 0x7f85741474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 0x7f85740af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85740b62d0 con 0x7f85740a7fc0 2022-07-20T02:32:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857414aac0 con 0x7f85740b5ef0 2022-07-20T02:32:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8579ed5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857414ad60 con 0x7f85740a8c40 2022-07-20T02:32:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f8578ed3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f85740b5ef0 0x7f85741474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8573fff700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f85740a7fc0 0x7f85740aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:53032/0 (socket says 172.21.15.26:53032) 2022-07-20T02:32:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8573fff700 1 -- 172.21.15.26:0/3381777400 learned_addr learned my addr 172.21.15.26:0/3381777400 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.234+0000 7f85796d4700 1 --2- 172.21.15.26:0/3381777400 >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 0x7f85740af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85737fe700 1 -- 172.21.15.26:0/3381777400 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470830444 0 0) 0x7f85740b62d0 con 0x7f85740a7fc0 2022-07-20T02:32:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85737fe700 1 -- 172.21.15.26:0/3381777400 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8560003260 con 0x7f85740a7fc0 2022-07-20T02:32:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8578ed3700 1 -- 172.21.15.26:0/3381777400 >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 msgr2=0x7f85740af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8578ed3700 1 --2- 172.21.15.26:0/3381777400 >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 0x7f85740af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8578ed3700 1 -- 172.21.15.26:0/3381777400 >> v1:172.21.15.120:6789/0 conn(0x7f85740a7fc0 legacy=0x7f85740aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8578ed3700 1 -- 172.21.15.26:0/3381777400 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857414b000 con 0x7f85740b5ef0 2022-07-20T02:32:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85796d4700 1 --2- 172.21.15.26:0/3381777400 >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 0x7f85740af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8578ed3700 1 --2- 172.21.15.26:0/3381777400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85741474d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568006df0).ready entity=mon.0 client_cookie=244b6b7595cad8fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85737fe700 1 -- 172.21.15.26:0/3381777400 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856800d4f0 con 0x7f85740b5ef0 2022-07-20T02:32:17.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85737fe700 1 -- 172.21.15.26:0/3381777400 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568000e90 con 0x7f85740b5ef0 2022-07-20T02:32:17.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f85737fe700 1 -- 172.21.15.26:0/3381777400 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856800d840 con 0x7f85740b5ef0 2022-07-20T02:32:17.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- 172.21.15.26:0/3381777400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 msgr2=0x7f85741474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3381777400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85741474d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568006df0).stop 2022-07-20T02:32:17.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- 172.21.15.26:0/3381777400 shutdown_connections 2022-07-20T02:32:17.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3381777400 >> v2:172.21.15.120:3300/0 conn(0x7f85740a8c40 0x7f85740af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3381777400 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85741474d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- 172.21.15.26:0/3381777400 >> 172.21.15.26:0/3381777400 conn(0x7f8574060310 msgr2=0x7f85740b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- 172.21.15.26:0/3381777400 shutdown_connections 2022-07-20T02:32:17.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- 172.21.15.26:0/3381777400 wait complete. 2022-07-20T02:32:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 Processor -- start 2022-07-20T02:32:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.238+0000 7f8579ed5700 1 -- start start 2022-07-20T02:32:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85740a7fc0 0x7f8574065950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85740a8c40 0x7f85740632c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574063ca0 con 0x7f85740b5ef0 2022-07-20T02:32:17.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574062ed0 con 0x7f85740a8c40 2022-07-20T02:32:17.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85740abb00 con 0x7f85740a7fc0 2022-07-20T02:32:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8578ed3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85740a7fc0 0x7f8574065950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8573fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85740a8c40 0x7f85740632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39550/0 (socket says 172.21.15.26:39550) 2022-07-20T02:32:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 -- 172.21.15.26:0/3499634927 learned_addr learned my addr 172.21.15.26:0/3499634927 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 -- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85740a7fc0 msgr2=0x7f8574065950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85740a7fc0 0x7f8574065950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 -- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85740a8c40 msgr2=0x7f85740632c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85740a8c40 0x7f85740632c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 -- 172.21.15.26:0/3499634927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c009470 con 0x7f85740b5ef0 2022-07-20T02:32:17.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f85796d4700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f856c00c000 tx=0x7f856c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856c00c850 con 0x7f85740b5ef0 2022-07-20T02:32:17.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856c00c9b0 con 0x7f85740b5ef0 2022-07-20T02:32:17.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856800b180 con 0x7f85740b5ef0 2022-07-20T02:32:17.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856c00cd10 con 0x7f85740b5ef0 2022-07-20T02:32:17.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.242+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85740ab610 con 0x7f85740b5ef0 2022-07-20T02:32:17.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85740630d0 con 0x7f85740b5ef0 2022-07-20T02:32:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f856c022070 con 0x7f85740b5ef0 2022-07-20T02:32:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8571ffb700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 0x7f8554055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f856c06c760 con 0x7f85740b5ef0 2022-07-20T02:32:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8578ed3700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 0x7f8554055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.246+0000 7f8578ed3700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 0x7f8554055ff0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f856801a000 tx=0x7f8568014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.250+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f856c043500 con 0x7f85740b5ef0 2022-07-20T02:32:17.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:17.312 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:17.316 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:17.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.378+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f857414b2a0 con 0x7f85740b5ef0 2022-07-20T02:32:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.378+0000 7f8571ffb700 1 -- 172.21.15.26:0/3499634927 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16383 (secure 0 0 0) 0x7f856c046b20 con 0x7f85740b5ef0 2022-07-20T02:32:17.387 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:17.388 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":77,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:10.565584+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:31:23.315398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T02:31:23.847170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.159:0/682748160":"2022-07-20T03:32:10.385916+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 msgr2=0x7f8554055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 0x7f8554055ff0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f856801a000 tx=0x7f8568014000).stop 2022-07-20T02:32:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 msgr2=0x7f85740637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f856c00c000 tx=0x7f856c0091d0).stop 2022-07-20T02:32:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 shutdown_connections 2022-07-20T02:32:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85740b5ef0 0x7f85740637b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8554053b30 0x7f8554055ff0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85740a8c40 0x7f85740632c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 --2- 172.21.15.26:0/3499634927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85740a7fc0 0x7f8574065950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 >> 172.21.15.26:0/3499634927 conn(0x7f8574060310 msgr2=0x7f8574148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 shutdown_connections 2022-07-20T02:32:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.382+0000 7f8579ed5700 1 -- 172.21.15.26:0/3499634927 wait complete. 2022-07-20T02:32:17.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:31:23.315398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:31:23.847170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:32:17.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:32:17.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.522+0000 7f8561985700 1 Processor -- start 2022-07-20T02:32:17.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.522+0000 7f8561985700 1 -- start start 2022-07-20T02:32:17.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f855c0b7d80 0x7f855c0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 0x7f855c146960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855c0617a0 con 0x7f855c0b7d80 2022-07-20T02:32:17.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855c0b2e40 con 0x7f855c0b3060 2022-07-20T02:32:17.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855c060b20 con 0x7f855c064220 2022-07-20T02:32:17.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f855c0b7d80 0x7f855c0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561184700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 0x7f855c146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59246/0 (socket says 172.21.15.26:59246) 2022-07-20T02:32:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f855c0b7d80 0x7f855c0b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39554/0 (socket says 172.21.15.26:39554) 2022-07-20T02:32:17.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855bfff700 1 -- 172.21.15.26:0/1010071629 learned_addr learned my addr 172.21.15.26:0/1010071629 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561184700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 0x7f855c146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:32842/0 (socket says 172.21.15.26:32842) 2022-07-20T02:32:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 -- 172.21.15.26:0/1010071629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 msgr2=0x7f855c146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 --2- 172.21.15.26:0/1010071629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 0x7f855c146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 -- 172.21.15.26:0/1010071629 >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 msgr2=0x7f855c0b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:17.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 --2- 172.21.15.26:0/1010071629 >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 -- 172.21.15.26:0/1010071629 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c060dc0 con 0x7f855c0b7d80 2022-07-20T02:32:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855bfff700 1 --2- 172.21.15.26:0/1010071629 >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8560983700 1 --2- 172.21.15.26:0/1010071629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c0b7d80 0x7f855c0b4020 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f854c002f70 tx=0x7f854c006df0).ready entity=mon.0 client_cookie=5b4dedf8454e5796 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855b7fe700 1 -- 172.21.15.26:0/1010071629 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f854c00d4a0 con 0x7f855c0b7d80 2022-07-20T02:32:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855b7fe700 1 -- 172.21.15.26:0/1010071629 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c000e90 con 0x7f855c0b7d80 2022-07-20T02:32:17.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f855b7fe700 1 -- 172.21.15.26:0/1010071629 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f854c00d7f0 con 0x7f855c0b7d80 2022-07-20T02:32:17.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- 172.21.15.26:0/1010071629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c0b7d80 msgr2=0x7f855c0b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- 172.21.15.26:0/1010071629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c0b7d80 0x7f855c0b4020 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f854c002f70 tx=0x7f854c006df0).stop 2022-07-20T02:32:17.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- 172.21.15.26:0/1010071629 shutdown_connections 2022-07-20T02:32:17.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- 172.21.15.26:0/1010071629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c064220 0x7f855c146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- 172.21.15.26:0/1010071629 >> v2:172.21.15.120:3300/0 conn(0x7f855c0b3060 0x7f855c0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 --2- 172.21.15.26:0/1010071629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c0b7d80 0x7f855c0b4020 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- 172.21.15.26:0/1010071629 >> 172.21.15.26:0/1010071629 conn(0x7f855c05f840 msgr2=0x7f855c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- 172.21.15.26:0/1010071629 shutdown_connections 2022-07-20T02:32:17.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.526+0000 7f8561985700 1 -- 172.21.15.26:0/1010071629 wait complete. 2022-07-20T02:32:17.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 Processor -- start 2022-07-20T02:32:17.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 -- start start 2022-07-20T02:32:17.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 0x7f855c0abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 0x7f855c143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0ab470 con 0x7f855c064220 2022-07-20T02:32:17.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0ac2d0 con 0x7f855c0b3060 2022-07-20T02:32:17.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561985700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0ad200 con 0x7f855c0b7d80 2022-07-20T02:32:17.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39560/0 (socket says 172.21.15.26:39560) 2022-07-20T02:32:17.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 -- 172.21.15.26:0/2393486160 learned_addr learned my addr 172.21.15.26:0/2393486160 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:17.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f855bfff700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 0x7f855c0abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561184700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 0x7f855c143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 -- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 msgr2=0x7f855c143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 0x7f855c143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 -- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 msgr2=0x7f855c0abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 0x7f855c0abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8560983700 1 -- 172.21.15.26:0/2393486160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854400b1e0 con 0x7f855c064220 2022-07-20T02:32:17.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f855bfff700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 0x7f855c0abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:17.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.530+0000 7f8561184700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 0x7f855c143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:17.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8560983700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f854c002f70 tx=0x7f854c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f854c013d50 con 0x7f855c064220 2022-07-20T02:32:17.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c01e400 con 0x7f855c064220 2022-07-20T02:32:17.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c00b180 con 0x7f855c064220 2022-07-20T02:32:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f854c01e760 con 0x7f855c064220 2022-07-20T02:32:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c0ab940 con 0x7f855c064220 2022-07-20T02:32:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855c0abb00 con 0x7f855c064220 2022-07-20T02:32:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f854c025030 con 0x7f855c064220 2022-07-20T02:32:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 0x7f8540055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f854c06ce60 con 0x7f855c064220 2022-07-20T02:32:17.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.534+0000 7f855bfff700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 0x7f8540055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.538+0000 7f855bfff700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 0x7f8540055e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f854400b000 tx=0x7f8544008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.542+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f854c043af0 con 0x7f855c064220 2022-07-20T02:32:17.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.674+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f855c061060 con 0x7f855c064220 2022-07-20T02:32:17.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.674+0000 7f8559ffb700 1 -- 172.21.15.26:0/2393486160 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v77) v1 ==== 110+0+10 (secure 0 0 0) 0x7f854c023070 con 0x7f855c064220 2022-07-20T02:32:17.679 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:32:17.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 msgr2=0x7f8540055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 0x7f8540055e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f854400b000 tx=0x7f8544008040).stop 2022-07-20T02:32:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 msgr2=0x7f855c0b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f854c002f70 tx=0x7f854c014000).stop 2022-07-20T02:32:17.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 shutdown_connections 2022-07-20T02:32:17.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8540053950 0x7f8540055e10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f855c0b7d80 0x7f855c143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c0b3060 0x7f855c0abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 --2- 172.21.15.26:0/2393486160 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f855c064220 0x7f855c0b5b80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 >> 172.21.15.26:0/2393486160 conn(0x7f855c05f840 msgr2=0x7f855c1491e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 shutdown_connections 2022-07-20T02:32:17.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.678+0000 7f8561985700 1 -- 172.21.15.26:0/2393486160 wait complete. 2022-07-20T02:32:17.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:32:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 Processor -- start 2022-07-20T02:32:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 -- start start 2022-07-20T02:32:17.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fed380a7620 0x7fed38066850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed380b2ea0 con 0x7fed380b2ac0 2022-07-20T02:32:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed38147900 con 0x7fed380a7620 2022-07-20T02:32:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed3ded0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed38147ba0 con 0x7fed38148580 2022-07-20T02:32:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed377fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fed380b2ac0 0x7fed380aff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38848/0 (socket says 172.21.15.26:38848) 2022-07-20T02:32:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed377fe700 1 -- 172.21.15.26:0/1953951180 learned_addr learned my addr 172.21.15.26:0/1953951180 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:17.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1953951180 >> v2:172.21.15.120:3300/0 conn(0x7fed380a7620 0x7fed38066850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed37fff700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888429318 0 0) 0x7fed380b2ea0 con 0x7fed380b2ac0 2022-07-20T02:32:17.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24003260 con 0x7fed380b2ac0 2022-07-20T02:32:17.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 -- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 msgr2=0x7fed38066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:17.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 -- 172.21.15.26:0/1953951180 >> v1:172.21.15.26:6789/0 conn(0x7fed380b2ac0 legacy=0x7fed380aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437192626 0 0) 0x7fed24003260 con 0x7fed380b2ac0 2022-07-20T02:32:17.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 -- 172.21.15.26:0/1953951180 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed38147e40 con 0x7fed380a7620 2022-07-20T02:32:17.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed38066850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fed28001800 tx=0x7fed2c006df0).ready entity=mon.1 client_cookie=e19418017d1b6e75 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed28001660 con 0x7fed380a7620 2022-07-20T02:32:17.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c000e90 con 0x7fed380a7620 2022-07-20T02:32:17.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.826+0000 7fed367fc700 1 -- 172.21.15.26:0/1953951180 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c012460 con 0x7fed380a7620 2022-07-20T02:32:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 msgr2=0x7fed38066850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed38066850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fed28001800 tx=0x7fed2c006df0).stop 2022-07-20T02:32:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- 172.21.15.26:0/1953951180 shutdown_connections 2022-07-20T02:32:17.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1953951180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed38066850 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- 172.21.15.26:0/1953951180 >> 172.21.15.26:0/1953951180 conn(0x7fed380b60a0 msgr2=0x7fed380b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- 172.21.15.26:0/1953951180 shutdown_connections 2022-07-20T02:32:17.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- 172.21.15.26:0/1953951180 wait complete. 2022-07-20T02:32:17.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 Processor -- start 2022-07-20T02:32:17.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- start start 2022-07-20T02:32:17.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed380ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed381566a0 con 0x7fed380b2ac0 2022-07-20T02:32:17.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed380658b0 con 0x7fed380a7620 2022-07-20T02:32:17.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed3ded0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380b46e0 con 0x7fed38148580 2022-07-20T02:32:17.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed36ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed36ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39576/0 (socket says 172.21.15.26:39576) 2022-07-20T02:32:17.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed36ffd700 1 -- 172.21.15.26:0/1396829446 learned_addr learned my addr 172.21.15.26:0/1396829446 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:17.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed37fff700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.830+0000 7fed377fe700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed380ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 -- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 msgr2=0x7fed38156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 -- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 msgr2=0x7fed380ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed380ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 -- 172.21.15.26:0/1396829446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c00b180 con 0x7fed380b2ac0 2022-07-20T02:32:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed36ffd700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7fed28001800 tx=0x7fed2c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c016070 con 0x7fed380b2ac0 2022-07-20T02:32:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c013e20 con 0x7fed380b2ac0 2022-07-20T02:32:17.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed380b4980 con 0x7fed380b2ac0 2022-07-20T02:32:17.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c002f50 con 0x7fed380b2ac0 2022-07-20T02:32:17.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed380b43d0 con 0x7fed380b2ac0 2022-07-20T02:32:17.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed381480e0 con 0x7fed380b2ac0 2022-07-20T02:32:17.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fed2c024030 con 0x7fed380b2ac0 2022-07-20T02:32:17.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 0x7fed18055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:17.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fed2c06cc80 con 0x7fed380b2ac0 2022-07-20T02:32:17.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.834+0000 7fed377fe700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 0x7fed18055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:17.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.838+0000 7fed377fe700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 0x7fed18055fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fed2800d000 tx=0x7fed280013c0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:17.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.846+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed2c043a20 con 0x7fed380b2ac0 2022-07-20T02:32:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.974+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fed380574e0 con 0x7fed380b2ac0 2022-07-20T02:32:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.974+0000 7fed34ff9700 1 -- 172.21.15.26:0/1396829446 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v77) v1 ==== 92+0+10 (secure 0 0 0) 0x7fed2c022070 con 0x7fed380b2ac0 2022-07-20T02:32:17.984 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 msgr2=0x7fed18055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 0x7fed18055fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fed2800d000 tx=0x7fed280013c0).stop 2022-07-20T02:32:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 msgr2=0x7fed380aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7fed28001800 tx=0x7fed2c014000).stop 2022-07-20T02:32:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 shutdown_connections 2022-07-20T02:32:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed380b2ac0 0x7fed380aede0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed18053b00 0x7fed18055fc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed38148580 0x7fed38156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 --2- 172.21.15.26:0/1396829446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed380a7620 0x7fed380ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 >> 172.21.15.26:0/1396829446 conn(0x7fed380b60a0 msgr2=0x7fed380afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 shutdown_connections 2022-07-20T02:32:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:17.978+0000 7fed3ded0700 1 -- 172.21.15.26:0/1396829446 wait complete. 2022-07-20T02:32:17.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:32:18.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.106+0000 7fe65952c700 1 Processor -- start 2022-07-20T02:32:18.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.106+0000 7fe65952c700 1 -- start start 2022-07-20T02:32:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.106+0000 7fe65952c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 0x7fe6540b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.106+0000 7fe65952c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.106+0000 7fe65952c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe6540b0d00 0x7fe6541450b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe654061010 con 0x7fe6540b0d00 2022-07-20T02:32:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe654144970 con 0x7fe6540b27f0 2022-07-20T02:32:18.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe654144c10 con 0x7fe654060a90 2022-07-20T02:32:18.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe6540b0d00 0x7fe6541450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe6540b0d00 0x7fe6541450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39586/0 (socket says 172.21.15.26:39586) 2022-07-20T02:32:18.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 -- 172.21.15.26:0/3079944604 learned_addr learned my addr 172.21.15.26:0/3079944604 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6527fc700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe652ffd700 1 --2- 172.21.15.26:0/3079944604 >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 0x7fe6540b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 -- 172.21.15.26:0/3079944604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 msgr2=0x7fe6540affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 -- 172.21.15.26:0/3079944604 >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 msgr2=0x7fe6540b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 --2- 172.21.15.26:0/3079944604 >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 0x7fe6540b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 -- 172.21.15.26:0/3079944604 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b1190 con 0x7fe6540b0d00 2022-07-20T02:32:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe652ffd700 1 --2- 172.21.15.26:0/3079944604 >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 0x7fe6540b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe6537fe700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe6541450b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fe64c0032f0 tx=0x7fe64c009290).ready entity=mon.0 client_cookie=e56fa877757f9686 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe651ffb700 1 -- 172.21.15.26:0/3079944604 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64c00b870 con 0x7fe6540b0d00 2022-07-20T02:32:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe651ffb700 1 -- 172.21.15.26:0/3079944604 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c00b9d0 con 0x7fe6540b0d00 2022-07-20T02:32:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe651ffb700 1 -- 172.21.15.26:0/3079944604 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64c00bb40 con 0x7fe6540b0d00 2022-07-20T02:32:18.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- 172.21.15.26:0/3079944604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 msgr2=0x7fe6541450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe6541450b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fe64c0032f0 tx=0x7fe64c009290).stop 2022-07-20T02:32:18.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- 172.21.15.26:0/3079944604 shutdown_connections 2022-07-20T02:32:18.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe6541450b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 --2- 172.21.15.26:0/3079944604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 --2- 172.21.15.26:0/3079944604 >> v2:172.21.15.120:3300/0 conn(0x7fe6540b27f0 0x7fe6540b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- 172.21.15.26:0/3079944604 >> 172.21.15.26:0/3079944604 conn(0x7fe6540b6110 msgr2=0x7fe6540b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- 172.21.15.26:0/3079944604 shutdown_connections 2022-07-20T02:32:18.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- 172.21.15.26:0/3079944604 wait complete. 2022-07-20T02:32:18.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 Processor -- start 2022-07-20T02:32:18.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.110+0000 7fe65952c700 1 -- start start 2022-07-20T02:32:18.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 0x7fe65413ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654062eb0 con 0x7fe6540b0d00 2022-07-20T02:32:18.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540644c0 con 0x7fe6540b27f0 2022-07-20T02:32:18.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe654147050 con 0x7fe654060a90 2022-07-20T02:32:18.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39590/0 (socket says 172.21.15.26:39590) 2022-07-20T02:32:18.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 -- 172.21.15.26:0/1773505248 learned_addr learned my addr 172.21.15.26:0/1773505248 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6537fe700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 0x7fe65413ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe652ffd700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 -- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 msgr2=0x7fe6540b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 -- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 msgr2=0x7fe65413ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 0x7fe65413ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 -- 172.21.15.26:0/1773505248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c009530 con 0x7fe6540b0d00 2022-07-20T02:32:18.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6537fe700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 0x7fe65413ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:18.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe652ffd700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:18.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe6527fc700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7fe64400a000 tx=0x7fe644008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe644011850 con 0x7fe6540b0d00 2022-07-20T02:32:18.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6440119b0 con 0x7fe6540b0d00 2022-07-20T02:32:18.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6541472f0 con 0x7fe6540b0d00 2022-07-20T02:32:18.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe644011d10 con 0x7fe6540b0d00 2022-07-20T02:32:18.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540a8870 con 0x7fe6540b0d00 2022-07-20T02:32:18.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe64401c460 con 0x7fe6540b0d00 2022-07-20T02:32:18.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.114+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6540af410 con 0x7fe6540b0d00 2022-07-20T02:32:18.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.118+0000 7fe643fff700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 0x7fe630055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.118+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fe64406c560 con 0x7fe6540b0d00 2022-07-20T02:32:18.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.118+0000 7fe652ffd700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 0x7fe630055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.118+0000 7fe652ffd700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 0x7fe630055e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe648008000 tx=0x7fe648008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.122+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6440432a0 con 0x7fe6540b0d00 2022-07-20T02:32:18.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6540ae5d0 con 0x7fe6540b0d00 2022-07-20T02:32:18.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe643fff700 1 -- 172.21.15.26:0/1773505248 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe6440468c0 con 0x7fe6540b0d00 2022-07-20T02:32:18.262 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:18.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 msgr2=0x7fe630055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 0x7fe630055e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe648008000 tx=0x7fe648008040).stop 2022-07-20T02:32:18.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 msgr2=0x7fe65413e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.250+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7fe64400a000 tx=0x7fe644008040).stop 2022-07-20T02:32:18.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 shutdown_connections 2022-07-20T02:32:18.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe6540b0d00 0x7fe65413e960 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe630053950 0x7fe630055e10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6540b27f0 0x7fe65413ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 --2- 172.21.15.26:0/1773505248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe654060a90 0x7fe6540b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 >> 172.21.15.26:0/1773505248 conn(0x7fe6540b6110 msgr2=0x7fe654066a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 shutdown_connections 2022-07-20T02:32:18.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.254+0000 7fe65952c700 1 -- 172.21.15.26:0/1773505248 wait complete. 2022-07-20T02:32:18.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:32:18.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 Processor -- start 2022-07-20T02:32:18.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 -- start start 2022-07-20T02:32:18.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02f00618a0 0x7f02f0147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f0143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f0061c80 con 0x7f02f0143d60 2022-07-20T02:32:18.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f01441f0 con 0x7f02f00618a0 2022-07-20T02:32:18.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02f5311700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00b2810 con 0x7f02f00b22d0 2022-07-20T02:32:18.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02f00618a0 0x7f02f0147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02f00618a0 0x7f02f0147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39596/0 (socket says 172.21.15.26:39596) 2022-07-20T02:32:18.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 -- 172.21.15.26:0/1976268930 learned_addr learned my addr 172.21.15.26:0/1976268930 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02ef7fe700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f0143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 -- 172.21.15.26:0/1976268930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 msgr2=0x7f02f0143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:18.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f0143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 -- 172.21.15.26:0/1976268930 >> v1:172.21.15.120:6789/0 conn(0x7f02f0143d60 legacy=0x7f02f00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 -- 172.21.15.26:0/1976268930 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00b2ab0 con 0x7f02f00618a0 2022-07-20T02:32:18.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02effff700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f00618a0 0x7f02f0147250 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f02e00032f0 tx=0x7f02e0009290).ready entity=mon.0 client_cookie=111a4745642f3e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02eeffd700 1 -- 172.21.15.26:0/1976268930 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02e000b8a0 con 0x7f02f00618a0 2022-07-20T02:32:18.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02eeffd700 1 -- 172.21.15.26:0/1976268930 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e000ba00 con 0x7f02f00618a0 2022-07-20T02:32:18.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.382+0000 7f02eeffd700 1 -- 172.21.15.26:0/1976268930 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02e000bb70 con 0x7f02f00618a0 2022-07-20T02:32:18.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- 172.21.15.26:0/1976268930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f00618a0 msgr2=0x7f02f0147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f00618a0 0x7f02f0147250 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f02e00032f0 tx=0x7f02e0009290).stop 2022-07-20T02:32:18.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- 172.21.15.26:0/1976268930 shutdown_connections 2022-07-20T02:32:18.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f0143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 --2- 172.21.15.26:0/1976268930 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f00618a0 0x7f02f0147250 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- 172.21.15.26:0/1976268930 >> 172.21.15.26:0/1976268930 conn(0x7f02f0066f70 msgr2=0x7f02f0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- 172.21.15.26:0/1976268930 shutdown_connections 2022-07-20T02:32:18.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- 172.21.15.26:0/1976268930 wait complete. 2022-07-20T02:32:18.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 Processor -- start 2022-07-20T02:32:18.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.386+0000 7f02f5311700 1 -- start start 2022-07-20T02:32:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02f00618a0 0x7f02f0146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f012b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01474a0 con 0x7f02f0143d60 2022-07-20T02:32:18.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f012b9f0 con 0x7f02f00618a0 2022-07-20T02:32:18.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f0140710 con 0x7f02f00b22d0 2022-07-20T02:32:18.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02effff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02f00618a0 0x7f02f0146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02ef7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f012b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39606/0 (socket says 172.21.15.26:39606) 2022-07-20T02:32:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 -- 172.21.15.26:0/135402354 learned_addr learned my addr 172.21.15.26:0/135402354 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 -- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 msgr2=0x7f02f012b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f012b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 -- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02f00618a0 msgr2=0x7f02f0146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02f00618a0 0x7f02f0146350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 -- 172.21.15.26:0/135402354 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0009530 con 0x7f02f0143d60 2022-07-20T02:32:18.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f4b10700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f02f00b6ab0 tx=0x7f02e8007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02e8012400 con 0x7f02f0143d60 2022-07-20T02:32:18.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e8012560 con 0x7f02f0143d60 2022-07-20T02:32:18.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f00b0590 con 0x7f02f0143d60 2022-07-20T02:32:18.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02e80128c0 con 0x7f02f0143d60 2022-07-20T02:32:18.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.390+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f013dc70 con 0x7f02f0143d60 2022-07-20T02:32:18.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f00574e0 con 0x7f02f0143d60 2022-07-20T02:32:18.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f02e8019030 con 0x7f02f0143d60 2022-07-20T02:32:18.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02ed7fa700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 0x7f02c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f02e806b4d0 con 0x7f02f0143d60 2022-07-20T02:32:18.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02effff700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 0x7f02c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.394+0000 7f02effff700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 0x7f02c8055f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f02e001b000 tx=0x7f02e0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.398+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02e8042100 con 0x7f02f0143d60 2022-07-20T02:32:18.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02f00b66d0 con 0x7f02f0143d60 2022-07-20T02:32:18.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02ed7fa700 1 -- 172.21.15.26:0/135402354 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+10 (secure 0 0 0) 0x7f02e8045720 con 0x7f02f0143d60 2022-07-20T02:32:18.532 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:18.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 msgr2=0x7f02c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 0x7f02c8055f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f02e001b000 tx=0x7f02e0016000).stop 2022-07-20T02:32:18.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 msgr2=0x7f02f012b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.526+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f02f00b6ab0 tx=0x7f02e8007bc0).stop 2022-07-20T02:32:18.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 shutdown_connections 2022-07-20T02:32:18.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02f0143d60 0x7f02f012b500 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02c8053a70 0x7f02c8055f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02f00b22d0 0x7f02f012b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 --2- 172.21.15.26:0/135402354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02f00618a0 0x7f02f0146350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 >> 172.21.15.26:0/135402354 conn(0x7f02f0066f70 msgr2=0x7f02f012c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 shutdown_connections 2022-07-20T02:32:18.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.530+0000 7f02f5311700 1 -- 172.21.15.26:0/135402354 wait complete. 2022-07-20T02:32:18.553 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:32:18.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:18.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 Processor -- start 2022-07-20T02:32:18.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 -- start start 2022-07-20T02:32:18.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc0b2920 0x7fd0fc147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0fc061050 0x7fd0fc148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd0fc066a80 0x7fd0fc066ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc14b040 con 0x7fd0fc066a80 2022-07-20T02:32:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc0a8860 con 0x7fd0fc061050 2022-07-20T02:32:18.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd100cab700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b1bc0 con 0x7fd0fc0b2920 2022-07-20T02:32:18.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd0fad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd0fc066a80 0x7fd0fc066ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd0fad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd0fc066a80 0x7fd0fc066ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39614/0 (socket says 172.21.15.26:39614) 2022-07-20T02:32:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd0fad9d700 1 -- 172.21.15.26:0/3546554699 learned_addr learned my addr 172.21.15.26:0/3546554699 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd0fa59c700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc0b2920 0x7fd0fc147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.662+0000 7fd0f9d9b700 1 --2- 172.21.15.26:0/3546554699 >> v2:172.21.15.120:3300/0 conn(0x7fd0fc061050 0x7fd0fc148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 -- 172.21.15.26:0/3546554699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc0b2920 msgr2=0x7fd0fc147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:18.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc0b2920 0x7fd0fc147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 -- 172.21.15.26:0/3546554699 >> v2:172.21.15.120:3300/0 conn(0x7fd0fc061050 msgr2=0x7fd0fc148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:18.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 --2- 172.21.15.26:0/3546554699 >> v2:172.21.15.120:3300/0 conn(0x7fd0fc061050 0x7fd0fc148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 -- 172.21.15.26:0/3546554699 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc147700 con 0x7fd0fc066a80 2022-07-20T02:32:18.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc066a80 0x7fd0fc066ec0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fd0f4002f70 tx=0x7fd0f4006eb0).ready entity=mon.0 client_cookie=5effa02b44dbf71e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0f959a700 1 -- 172.21.15.26:0/3546554699 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0f400d570 con 0x7fd0fc066a80 2022-07-20T02:32:18.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0f959a700 1 -- 172.21.15.26:0/3546554699 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f4000e90 con 0x7fd0fc066a80 2022-07-20T02:32:18.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0f959a700 1 -- 172.21.15.26:0/3546554699 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0f400d900 con 0x7fd0fc066a80 2022-07-20T02:32:18.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- 172.21.15.26:0/3546554699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc066a80 msgr2=0x7fd0fc066ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc066a80 0x7fd0fc066ec0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fd0f4002f70 tx=0x7fd0f4006eb0).stop 2022-07-20T02:32:18.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- 172.21.15.26:0/3546554699 shutdown_connections 2022-07-20T02:32:18.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc066a80 0x7fd0fc066ec0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- 172.21.15.26:0/3546554699 >> v2:172.21.15.120:3300/0 conn(0x7fd0fc061050 0x7fd0fc148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- 172.21.15.26:0/3546554699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc0b2920 0x7fd0fc147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- 172.21.15.26:0/3546554699 >> 172.21.15.26:0/3546554699 conn(0x7fd0fc0b60a0 msgr2=0x7fd0fc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- 172.21.15.26:0/3546554699 shutdown_connections 2022-07-20T02:32:18.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- 172.21.15.26:0/3546554699 wait complete. 2022-07-20T02:32:18.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 Processor -- start 2022-07-20T02:32:18.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- start start 2022-07-20T02:32:18.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 0x7fd0fc14e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0fc0b2920 0x7fd0fc0ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc064550 con 0x7fd0fc061050 2022-07-20T02:32:18.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc065bf0 con 0x7fd0fc0b2920 2022-07-20T02:32:18.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd100cab700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc064780 con 0x7fd0fc066a80 2022-07-20T02:32:18.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fa59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39616/0 (socket says 172.21.15.26:39616) 2022-07-20T02:32:18.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fa59c700 1 -- 172.21.15.26:0/3620924265 learned_addr learned my addr 172.21.15.26:0/3620924265 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:18.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0f9d9b700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 0x7fd0fc14e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.666+0000 7fd0fad9d700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0fc0b2920 0x7fd0fc0ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 -- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0fc0b2920 msgr2=0x7fd0fc0ab110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0fc0b2920 0x7fd0fc0ab110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 -- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 msgr2=0x7fd0fc065f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 -- 172.21.15.26:0/3620924265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f400b1e0 con 0x7fd0fc066a80 2022-07-20T02:32:18.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0fa59c700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:18.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0f9d9b700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 0x7fd0fc14e6a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd0fc0aa7a0 tx=0x7fd0ec0091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0ec012600 con 0x7fd0fc066a80 2022-07-20T02:32:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec012760 con 0x7fd0fc066a80 2022-07-20T02:32:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc064a20 con 0x7fd0fc066a80 2022-07-20T02:32:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0ec012ac0 con 0x7fd0fc066a80 2022-07-20T02:32:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc159fc0 con 0x7fd0fc066a80 2022-07-20T02:32:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc065490 con 0x7fd0fc066a80 2022-07-20T02:32:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd0ec022070 con 0x7fd0fc066a80 2022-07-20T02:32:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 0x7fd0d8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fd0ec06c770 con 0x7fd0fc066a80 2022-07-20T02:32:18.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.670+0000 7fd0fa59c700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 0x7fd0d8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:18.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.674+0000 7fd0fa59c700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 0x7fd0d8055e10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd0f000b000 tx=0x7fd0f0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:18.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.678+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd0ec0433a0 con 0x7fd0fc066a80 2022-07-20T02:32:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.838+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0fc064bb0 con 0x7fd0fc066a80 2022-07-20T02:32:18.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd0eb7fe700 1 -- 172.21.15.26:0/3620924265 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fd0ec0469c0 con 0x7fd0fc066a80 2022-07-20T02:32:18.845 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:18.846 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:32:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 msgr2=0x7fd0d8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 0x7fd0d8055e10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd0f000b000 tx=0x7fd0f0006df0).stop 2022-07-20T02:32:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 msgr2=0x7fd0fc14e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 0x7fd0fc14e6a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd0fc0aa7a0 tx=0x7fd0ec0091d0).stop 2022-07-20T02:32:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 shutdown_connections 2022-07-20T02:32:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd0d8053950 0x7fd0d8055e10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0fc0b2920 0x7fd0fc0ab110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0fc066a80 0x7fd0fc14e6a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 --2- 172.21.15.26:0/3620924265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0fc061050 0x7fd0fc065f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 >> 172.21.15.26:0/3620924265 conn(0x7fd0fc0b60a0 msgr2=0x7fd0fc14a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 shutdown_connections 2022-07-20T02:32:18.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:18.842+0000 7fd100cab700 1 -- 172.21.15.26:0/3620924265 wait complete. 2022-07-20T02:32:18.858 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:32:19.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:31:24.761580+0000', 'modified': '2022-07-20T02:31:25.765269+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6309': {'gid': 6309, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/489399789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 489399789}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 489399789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:32:19.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:32:19.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:32:19.313 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6309,"want_state":"up:active","state":"up:active","rank_uptime":54.528920874000001,"mdsmap_epoch":37,"osdmap_epoch":77,"osdmap_epoch_barrier":77,"uptime":58.912267010000001} 2022-07-20T02:32:19.321 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6309, "mdsmap_epoch": 37, "osdmap_epoch": 77, "osdmap_epoch_barrier": 77, "rank_uptime": 54.528920874, "state": "up:active", "uptime": 58.91226701, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:32:19.322 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:32:19.323 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:32:19.324 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:32:19.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.325 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:19.347 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:32:19.410 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:19.411 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:19.424 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:19.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.428 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:32:19.490 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:32:19.491 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:19.503 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:32:19.535 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:19.537 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:19.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:32:19.577+0000 7efd0fc0a080 -1 init, newargv = 0x560056529820 newargc=9ceph-fuse[22910]: starting ceph client 2022-07-20T02:32:19.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:32:19.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22910]: starting fuse 2022-07-20T02:32:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:19.621 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:19.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.623 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:32:19.636 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:32:19.637 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:32:19.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:32:19.660 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:32:19.671 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:32:19.672 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:32:19.672 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:32:19.673 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:19.674 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:32:19.675 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:32:19.676 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:32:19.676 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:19.677 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:32:19.677 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:32:19.677 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:32:19.678 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:32:19.679 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:19.679 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:32:19.680 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:32:19.680 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:32:19.681 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:32:19.681 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:32:19.681 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:19.682 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:32:19.682 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:32:19.738 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22910.asok 2022-07-20T02:32:19.739 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22910.asok 2022-07-20T02:32:19.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.741 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22910.asok status 2022-07-20T02:32:19.864 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:32:19.865 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:32:19.866 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:32:19.874 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:32:19.874 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:32:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:32:19.875 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:32:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22910", 2022-07-20T02:32:19.876 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:32:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:32:19.877 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:32:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 6448, 2022-07-20T02:32:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:32:19.887 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:32:19.887 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:32:19.888 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 6448 2022-07-20T02:32:19.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:19.889 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:32:19.889 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:32:19.890 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:32:19.890 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 682992868 2022-07-20T02:32:19.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:32:19.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:19.891 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:32:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:32:19.892 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:32:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 682992868 2022-07-20T02:32:19.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:19.894 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.6448 172.21.15.159:0/682992868", 2022-07-20T02:32:19.895 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/682992868", 2022-07-20T02:32:19.899 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:32:19.899 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 37, 2022-07-20T02:32:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 77, 2022-07-20T02:32:19.900 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:32:19.901 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:32:19.901 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:32:19.902 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:32:19.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.905 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:19.906 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:19.916 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:32:19.918 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:19.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:19.919 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:19.986 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:32:19.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:20.016 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:20.039 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:20.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:20.110 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:32:20.111 INFO:tasks.cephfs.mount:Terminating background process 2022-07-20T02:32:20.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:20.113 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:20.114 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:20.136 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:32:20.136 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:20.137 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:32:20.138 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:32:20.139 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:20.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22910]: fuse finished with error 0 and tester_r 0 2022-07-20T02:32:20.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:20.216 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:20.217 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:20.276 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:32:20.277 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:32:20.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:32:20.378 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:20.388 INFO:tasks.cephfs.mount:Terminating background process 2022-07-20T02:32:20.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:20.390 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:20.391 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:20.413 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:32:20.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:32:20.414 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:32:20.415 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:32:20.425 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:20.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22515]: fuse finished with error 0 and tester_r 0 2022-07-20T02:32:20.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:20.462 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:20.463 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:20.512 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:32:20.514 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:32:20.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:32:20.634 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:20.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:32:20.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 Processor -- start 2022-07-20T02:32:20.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- start start 2022-07-20T02:32:20.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 0x7fa480067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 0x7fa4800979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 0x7fa480098320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48005ff70 con 0x7fa480097eb0 2022-07-20T02:32:20.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4800606e0 con 0x7fa480061860 2022-07-20T02:32:20.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800b3640 con 0x7fa4800b3260 2022-07-20T02:32:20.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48681b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 0x7fa480098320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa485819700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 0x7fa4800979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 0x7fa480067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48681b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 0x7fa480098320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39628/0 (socket says 172.21.15.26:39628) 2022-07-20T02:32:20.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48681b700 1 -- 172.21.15.26:0/3927947002 learned_addr learned my addr 172.21.15.26:0/3927947002 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:20.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 -- 172.21.15.26:0/3927947002 >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 msgr2=0x7fa4800979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:20.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 --2- 172.21.15.26:0/3927947002 >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 0x7fa4800979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 -- 172.21.15.26:0/3927947002 >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 msgr2=0x7fa480098320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:20.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 --2- 172.21.15.26:0/3927947002 >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 0x7fa480098320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 -- 172.21.15.26:0/3927947002 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4800601a0 con 0x7fa4800b3260 2022-07-20T02:32:20.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa485819700 1 --2- 172.21.15.26:0/3927947002 >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 0x7fa4800979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48601a700 1 --2- 172.21.15.26:0/3927947002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 0x7fa480067900 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa48006af80 tx=0x7fa478006e80).ready entity=mon.2 client_cookie=46e09b2eec5737e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa485018700 1 -- 172.21.15.26:0/3927947002 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa47800b900 con 0x7fa4800b3260 2022-07-20T02:32:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa485018700 1 -- 172.21.15.26:0/3927947002 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478000e90 con 0x7fa4800b3260 2022-07-20T02:32:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa485018700 1 -- 172.21.15.26:0/3927947002 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa47800bc50 con 0x7fa4800b3260 2022-07-20T02:32:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- 172.21.15.26:0/3927947002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 msgr2=0x7fa480067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:20.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- 172.21.15.26:0/3927947002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 0x7fa480067900 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa48006af80 tx=0x7fa478006e80).stop 2022-07-20T02:32:20.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- 172.21.15.26:0/3927947002 shutdown_connections 2022-07-20T02:32:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- 172.21.15.26:0/3927947002 >> v2:172.21.15.26:3300/0 conn(0x7fa480097eb0 0x7fa480098320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- 172.21.15.26:0/3927947002 >> v2:172.21.15.120:3300/0 conn(0x7fa480061860 0x7fa4800979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 --2- 172.21.15.26:0/3927947002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4800b3260 0x7fa480067900 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.766+0000 7fa48701c700 1 -- 172.21.15.26:0/3927947002 >> 172.21.15.26:0/3927947002 conn(0x7fa4800b67f0 msgr2=0x7fa4800b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:20.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- 172.21.15.26:0/3927947002 shutdown_connections 2022-07-20T02:32:20.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- 172.21.15.26:0/3927947002 wait complete. 2022-07-20T02:32:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 Processor -- start 2022-07-20T02:32:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- start start 2022-07-20T02:32:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa480061860 0x7fa48013be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 0x7fa480139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48009a7e0 con 0x7fa480097eb0 2022-07-20T02:32:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa480099b60 con 0x7fa4800b3260 2022-07-20T02:32:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48701c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa480099e00 con 0x7fa480061860 2022-07-20T02:32:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39632/0 (socket says 172.21.15.26:39632) 2022-07-20T02:32:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 -- 172.21.15.26:0/488727920 learned_addr learned my addr 172.21.15.26:0/488727920 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48601a700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa480061860 0x7fa48013be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48681b700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 0x7fa480139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 -- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa480061860 msgr2=0x7fa48013be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa480061860 0x7fa48013be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 -- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 msgr2=0x7fa480139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 0x7fa480139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 -- 172.21.15.26:0/488727920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474009470 con 0x7fa480097eb0 2022-07-20T02:32:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa48681b700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 0x7fa480139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.770+0000 7fa485819700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fa47400c000 tx=0x7fa4740091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa47400ee00 con 0x7fa480097eb0 2022-07-20T02:32:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47400cbe0 con 0x7fa480097eb0 2022-07-20T02:32:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47800d410 con 0x7fa480097eb0 2022-07-20T02:32:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4740123f0 con 0x7fa480097eb0 2022-07-20T02:32:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48009a340 con 0x7fa480097eb0 2022-07-20T02:32:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48009a500 con 0x7fa480097eb0 2022-07-20T02:32:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa474012550 con 0x7fa480097eb0 2022-07-20T02:32:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 0x7fa460055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fa47406cba0 con 0x7fa480097eb0 2022-07-20T02:32:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.774+0000 7fa48601a700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 0x7fa460055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.778+0000 7fa48601a700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 0x7fa460055ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa47801c000 tx=0x7fa4800976b0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.782+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4740438e0 con 0x7fa480097eb0 2022-07-20T02:32:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.910+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa480067d10 con 0x7fa480097eb0 2022-07-20T02:32:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.910+0000 7fa4737fe700 1 -- 172.21.15.26:0/488727920 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v77) v1 ==== 81+0+34164 (secure 0 0 0) 0x7fa474046f00 con 0x7fa480097eb0 2022-07-20T02:32:20.916 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:20.916 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:20.917 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 77, 2022-07-20T02:32:20.917 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:32:20.918 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:32:20.918 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:32:10.565584+0000", 2022-07-20T02:32:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:32:20.931 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:32:20.932 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:32:20.933 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:32:20.934 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:32:20.934 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:32:20.935 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:32:20.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:20.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:32:20.936 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:32:20.937 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:32:20.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:32:20.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:32:20.938 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 14, 2022-07-20T02:32:20.939 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:32:20.939 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:32:20.940 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:32:20.940 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:32:20.940 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:32:20.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:20.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:32:20.948 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:32:20.949 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:32:20.949 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:20.950 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:20.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:20.951 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:20.951 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:20.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:20.952 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:20.953 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:20.953 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:32:20.970 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:32:20.970 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:32:20.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:32:20.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:32:20.972 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:20.972 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:20.973 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:20.977 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:20.977 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:20.977 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:20.977 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:20.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:20.978 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:32:20.979 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:20.979 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:20.979 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:20.980 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:20.980 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:20.981 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:20.981 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:20.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:20.985 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:20.995 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:20.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:20.997 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:20.998 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:20.999 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:21.000 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:21.007 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:21.008 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:21.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:21.012 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:21.013 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:21.014 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:21.015 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:21.016 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:21.017 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:32:21.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:32:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:32:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:32:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:32:21.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:21.021 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:21.022 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:21.023 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.025 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:32:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:21.026 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:21.027 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:21.028 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:21.029 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:21.030 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:21.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:21.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:21.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.034 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:21.050 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:21.052 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:21.053 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:21.055 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:21.062 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:21.063 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:21.064 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:21.065 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:32:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 13, 2022-07-20T02:32:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:32:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:31:23.315398+0000", 2022-07-20T02:32:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "75", 2022-07-20T02:32:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:21.091 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:21.092 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:21.095 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:21.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:21.097 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:21.098 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:21.099 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:21.100 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:32:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:32:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:32:21.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.104 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:21.104 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:32:21.104 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:32:21.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.105 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 14, 2022-07-20T02:32:21.106 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:32:21.106 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:31:23.847170+0000", 2022-07-20T02:32:21.106 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:21.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:21.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:21.107 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:21.107 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:21.108 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:21.108 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:21.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:21.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:21.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:21.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:21.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:21.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:21.109 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:21.110 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:21.110 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:21.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:21.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:21.111 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:21.111 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:21.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.111 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "76", 2022-07-20T02:32:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:21.113 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:21.113 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:21.113 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:21.113 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:21.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:21.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:21.116 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:21.116 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:21.116 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:21.116 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:21.117 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:21.117 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:21.117 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:21.117 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:21.118 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:21.118 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:21.118 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:21.119 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:21.119 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:21.119 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:21.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:21.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.120 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:21.120 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:21.120 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:21.121 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:21.121 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:21.121 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:21.121 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:21.122 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:21.122 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:21.122 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:21.123 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:21.123 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:21.123 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:21.123 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:32:21.124 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:32:21.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:21.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:32:21.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:21.125 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:32:21.128 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.128 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.129 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.129 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.129 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.129 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.129 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.130 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:21.130 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.130 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:32:21.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:32:21.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:32:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:32:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.158 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:32:21.160 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:32:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:32:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:32:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:32:21.181 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:32:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:32:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:32:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.182 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:32:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.184 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.185 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.186 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:32:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:32:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:32:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:32:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:32:21.198 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.199 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:32:21.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:32:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:32:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:32:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:21.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:32:21.208 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:32:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:32:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:32:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.209 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:32:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.211 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.213 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:32:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:32:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.218 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:32:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:32:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.222 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:32:21.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:32:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:32:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:32:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:21.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:32:21.232 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:32:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:32:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:32:21.233 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:21.235 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:32:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.236 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:21.237 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:32:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:32:21.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.241 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:32:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:32:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:32:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:32:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:32:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:32:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:32:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:32:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:32:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:32:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:32:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.257 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:32:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:32:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:32:21.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:32:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.265 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:32:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:32:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.268 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:32:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:32:21.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:32:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:32:21.273 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:32:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:32:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.274 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.275 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:32:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.276 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.277 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.278 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:32:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.280 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:32:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:32:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:32:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:32:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:32:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.290 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:32:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:32:21.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:32:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:32:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:32:21.294 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:32:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:32:21.296 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.297 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.298 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:32:21.300 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:32:21.301 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.302 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:32:21.303 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:32:21.304 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:32:21.315 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:32:21.316 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.317 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.317 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:32:21.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:32:21.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:21.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:32:21.335 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:32:21.335 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:32:21.336 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:32:21.336 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.336 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.336 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.337 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:21.338 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:32:21.339 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:21.339 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:21.339 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:21.339 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:21.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:21.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:21.340 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:21.342 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:21.343 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:21.343 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:21.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:21.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:32:21.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:32:21.346 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.347 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:21.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:32:21.348 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:32:21.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.350 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:21.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:32:21.351 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:32:21.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:21.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:21.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:32:21.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.356 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:21.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:32:21.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:21.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:32:21.358 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:32:21.358 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:32:21.358 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:32:21.358 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:21.359 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:21.359 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:21.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:21.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.360 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:21.360 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:32:21.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.360 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:21.361 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.361 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.361 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.361 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.362 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.362 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:32:21.362 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.363 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:21.363 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.364 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.364 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.364 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.365 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.365 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:32:21.365 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.366 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:21.366 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.366 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.366 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.367 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.367 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.367 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:32:21.367 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.368 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:21.368 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.369 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.369 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.369 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.369 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.370 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:32:21.370 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.372 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.372 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.372 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.372 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.372 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:32:21.373 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.374 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:21.375 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.375 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.376 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.376 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.376 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.376 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:32:21.377 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.377 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:21.378 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.378 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.378 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.378 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.379 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.379 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:32:21.379 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:21.381 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:21.381 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:21.381 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:32:21.382 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:21.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.382 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:21.382 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:32:21.383 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:32:21.383 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:32:21.384 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:32:21.385 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:32:21.385 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.159:0/682748160": "2022-07-20T03:32:10.385916+0000" 2022-07-20T02:32:21.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.385 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:32:21.386 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:32:21.386 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:32:21.386 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:21.386 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:32:21.387 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:32:21.387 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:21.387 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:32:21.387 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:21.388 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:21.388 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:32:21.388 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:32:21.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.389 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:32:21.389 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:21.389 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:21.389 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:32:21.390 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:21.390 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:21.390 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:32:21.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:21.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:21.391 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:32:21.391 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:32:21.391 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:32:21.392 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:32:21.392 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:32:21.392 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:21.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 msgr2=0x7fa460055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 0x7fa460055ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa47801c000 tx=0x7fa4800976b0).stop 2022-07-20T02:32:21.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 msgr2=0x7fa4801398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fa47400c000 tx=0x7fa4740091d0).stop 2022-07-20T02:32:21.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 shutdown_connections 2022-07-20T02:32:21.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa480097eb0 0x7fa4801398f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa460053b30 0x7fa460055ff0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4800b3260 0x7fa480139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 --2- 172.21.15.26:0/488727920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa480061860 0x7fa48013be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 >> 172.21.15.26:0/488727920 conn(0x7fa4800b67f0 msgr2=0x7fa4800b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 shutdown_connections 2022-07-20T02:32:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:20.914+0000 7fa48701c700 1 -- 172.21.15.26:0/488727920 wait complete. 2022-07-20T02:32:21.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:21.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 Processor -- start 2022-07-20T02:32:21.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 -- start start 2022-07-20T02:32:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f200621f0 0x7f1f20067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20060250 0x7f1f20060690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b0f50 con 0x7f1f200aff40 2022-07-20T02:32:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2013e7f0 con 0x7f1f200621f0 2022-07-20T02:32:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f2672b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200b0320 con 0x7f1f20060250 2022-07-20T02:32:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f24f28700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f200621f0 0x7f1f20067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f25f2a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20060250 0x7f1f20060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f25729700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1f200aff40 0x7f1f20066b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:53130/0 (socket says 172.21.15.26:53130) 2022-07-20T02:32:21.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f24f28700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f200621f0 0x7f1f20067240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59332/0 (socket says 172.21.15.26:59332) 2022-07-20T02:32:21.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f25729700 1 -- 172.21.15.26:0/2123403346 learned_addr learned my addr 172.21.15.26:0/2123403346 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:21.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f24f28700 1 -- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20060250 msgr2=0x7f1f20060690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:21.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f24f28700 1 --2- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20060250 0x7f1f20060690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.514+0000 7f1f24f28700 1 -- 172.21.15.26:0/2123403346 >> v1:172.21.15.120:6789/0 conn(0x7f1f200aff40 legacy=0x7f1f20066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2123403346 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540121977 0 0) 0x7f1f200b0f50 con 0x7f1f200aff40 2022-07-20T02:32:21.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f24f28700 1 -- 172.21.15.26:0/2123403346 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2005f8e0 con 0x7f1f200621f0 2022-07-20T02:32:21.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f24f28700 1 --2- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20067240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1f10002f70 tx=0x7f1f10006df0).ready entity=mon.1 client_cookie=be2db5675d0d36af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2123403346 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1000d570 con 0x7f1f200621f0 2022-07-20T02:32:21.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2123403346 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10000e90 con 0x7f1f200621f0 2022-07-20T02:32:21.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f0ffff700 1 -- 172.21.15.26:0/2123403346 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1000d8c0 con 0x7f1f200621f0 2022-07-20T02:32:21.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 msgr2=0x7f1f20067240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20067240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1f10002f70 tx=0x7f1f10006df0).stop 2022-07-20T02:32:21.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- 172.21.15.26:0/2123403346 shutdown_connections 2022-07-20T02:32:21.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f20060250 0x7f1f20060690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- 172.21.15.26:0/2123403346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20067240 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- 172.21.15.26:0/2123403346 >> 172.21.15.26:0/2123403346 conn(0x7f1f200b3050 msgr2=0x7f1f200b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- 172.21.15.26:0/2123403346 shutdown_connections 2022-07-20T02:32:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- 172.21.15.26:0/2123403346 wait complete. 2022-07-20T02:32:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 Processor -- start 2022-07-20T02:32:21.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 -- start start 2022-07-20T02:32:21.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.518+0000 7f1f2672b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 0x7f1f20150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200abee0 con 0x7f1f20060250 2022-07-20T02:32:21.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2014ffb0 con 0x7f1f200621f0 2022-07-20T02:32:21.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f201396d0 con 0x7f1f200aff40 2022-07-20T02:32:21.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39646/0 (socket says 172.21.15.26:39646) 2022-07-20T02:32:21.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 -- 172.21.15.26:0/199035389 learned_addr learned my addr 172.21.15.26:0/199035389 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:21.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25f2a700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 0x7f1f20150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f24f28700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 -- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 msgr2=0x7f1f20150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 0x7f1f20150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 -- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 msgr2=0x7f1f20150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 -- 172.21.15.26:0/199035389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1000b180 con 0x7f1f20060250 2022-07-20T02:32:21.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25f2a700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 0x7f1f20150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:21.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f25729700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f1f200aad90 tx=0x7f1f1c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1c0070d0 con 0x7f1f20060250 2022-07-20T02:32:21.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c007230 con 0x7f1f20060250 2022-07-20T02:32:21.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1c005610 con 0x7f1f20060250 2022-07-20T02:32:21.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f201501a0 con 0x7f1f20060250 2022-07-20T02:32:21.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f20139430 con 0x7f1f20060250 2022-07-20T02:32:21.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.522+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2005fb10 con 0x7f1f20060250 2022-07-20T02:32:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.526+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f1c010070 con 0x7f1f20060250 2022-07-20T02:32:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.526+0000 7f1f0e7fc700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 0x7f1f00055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.526+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f1f1c06b690 con 0x7f1f20060250 2022-07-20T02:32:21.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.526+0000 7f1f24f28700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 0x7f1f00055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.526+0000 7f1f24f28700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 0x7f1f00055fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1f1001a000 tx=0x7f1f10014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.530+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f1c042430 con 0x7f1f20060250 2022-07-20T02:32:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.694+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f200574e0 con 0x7f1f20060250 2022-07-20T02:32:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.698+0000 7f1f0e7fc700 1 -- 172.21.15.26:0/199035389 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f1f1c045a50 con 0x7f1f20060250 2022-07-20T02:32:21.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:21.704 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:31:25.765269+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/489399789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":489399789},{"type":"v1","addr":"172.21.15.26:6837","nonce":489399789}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:32:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 msgr2=0x7f1f00055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 0x7f1f00055fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1f1001a000 tx=0x7f1f10014000).stop 2022-07-20T02:32:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 msgr2=0x7f1f200b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f1f200aad90 tx=0x7f1f1c007c40).stop 2022-07-20T02:32:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 shutdown_connections 2022-07-20T02:32:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053b00 0x7f1f00055fc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200aff40 0x7f1f20150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f200621f0 0x7f1f20150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 --2- 172.21.15.26:0/199035389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f20060250 0x7f1f200b25c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 >> 172.21.15.26:0/199035389 conn(0x7f1f200b3050 msgr2=0x7f1f20140130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 shutdown_connections 2022-07-20T02:32:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.702+0000 7f1f2672b700 1 -- 172.21.15.26:0/199035389 wait complete. 2022-07-20T02:32:21.718 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2022-07-20T02:32:21.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:32:21.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 Processor -- start 2022-07-20T02:32:21.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 -- start start 2022-07-20T02:32:21.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f59201474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59200a7fc0 0x7f59201479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59200a8c40 0x7f59200aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5920147eb0 con 0x7f59200a7fc0 2022-07-20T02:32:21.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200b62d0 con 0x7f59200a8c40 2022-07-20T02:32:21.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5926617700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592014aac0 con 0x7f59200b5ef0 2022-07-20T02:32:21.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59200a7fc0 0x7f59201479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59200a7fc0 0x7f59201479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39654/0 (socket says 172.21.15.26:39654) 2022-07-20T02:32:21.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 -- 172.21.15.26:0/3063487246 learned_addr learned my addr 172.21.15.26:0/3063487246 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:21.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5925615700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f59201474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5925e16700 1 --2- 172.21.15.26:0/3063487246 >> v2:172.21.15.120:3300/0 conn(0x7f59200a8c40 0x7f59200aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 -- 172.21.15.26:0/3063487246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 msgr2=0x7f59201474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:21.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f59201474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 -- 172.21.15.26:0/3063487246 >> v2:172.21.15.120:3300/0 conn(0x7f59200a8c40 msgr2=0x7f59200aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:21.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 --2- 172.21.15.26:0/3063487246 >> v2:172.21.15.120:3300/0 conn(0x7f59200a8c40 0x7f59200aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 -- 172.21.15.26:0/3063487246 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592014ad60 con 0x7f59200a7fc0 2022-07-20T02:32:21.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5925615700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f59201474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:21.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f5924e14700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59201479c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f59100032f0 tx=0x7f5910009290).ready entity=mon.0 client_cookie=65e6d21f92408c60 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f591ffff700 1 -- 172.21.15.26:0/3063487246 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591000b680 con 0x7f59200a7fc0 2022-07-20T02:32:21.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f591ffff700 1 -- 172.21.15.26:0/3063487246 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591000b7e0 con 0x7f59200a7fc0 2022-07-20T02:32:21.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.830+0000 7f591ffff700 1 -- 172.21.15.26:0/3063487246 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591000b950 con 0x7f59200a7fc0 2022-07-20T02:32:21.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- 172.21.15.26:0/3063487246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 msgr2=0x7f59201479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59201479c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f59100032f0 tx=0x7f5910009290).stop 2022-07-20T02:32:21.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- 172.21.15.26:0/3063487246 shutdown_connections 2022-07-20T02:32:21.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 --2- 172.21.15.26:0/3063487246 >> v2:172.21.15.120:3300/0 conn(0x7f59200a8c40 0x7f59200aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59201479c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 --2- 172.21.15.26:0/3063487246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f59201474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- 172.21.15.26:0/3063487246 >> 172.21.15.26:0/3063487246 conn(0x7f5920060310 msgr2=0x7f59200b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:21.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- 172.21.15.26:0/3063487246 shutdown_connections 2022-07-20T02:32:21.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- 172.21.15.26:0/3063487246 wait complete. 2022-07-20T02:32:21.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 Processor -- start 2022-07-20T02:32:21.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.834+0000 7f5926617700 1 -- start start 2022-07-20T02:32:21.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 0x7f5920159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f592015a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200ac1e0 con 0x7f59200a7fc0 2022-07-20T02:32:21.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920147600 con 0x7f59200a8c40 2022-07-20T02:32:21.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200abf40 con 0x7f59200b5ef0 2022-07-20T02:32:21.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39658/0 (socket says 172.21.15.26:39658) 2022-07-20T02:32:21.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 -- 172.21.15.26:0/520864592 learned_addr learned my addr 172.21.15.26:0/520864592 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925e16700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f592015a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5924e14700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 0x7f5920159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 -- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 msgr2=0x7f592015a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f592015a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 -- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 msgr2=0x7f5920159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:21.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 0x7f5920159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:21.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 -- 172.21.15.26:0/520864592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c00b180 con 0x7f59200a7fc0 2022-07-20T02:32:21.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5924e14700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 0x7f5920159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:21.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925e16700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f592015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5925615700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f590c00c000 tx=0x7f590c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f590c010780 con 0x7f59200a7fc0 2022-07-20T02:32:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c0108e0 con 0x7f59200a7fc0 2022-07-20T02:32:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f590c010c40 con 0x7f59200a7fc0 2022-07-20T02:32:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5910009530 con 0x7f59200a7fc0 2022-07-20T02:32:21.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.838+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200aaa10 con 0x7f59200a7fc0 2022-07-20T02:32:21.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592014b000 con 0x7f59200a7fc0 2022-07-20T02:32:21.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f590c01c460 con 0x7f59200a7fc0 2022-07-20T02:32:21.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f591e7fc700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 0x7f58fc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:21.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f590c06c670 con 0x7f59200a7fc0 2022-07-20T02:32:21.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f5924e14700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 0x7f58fc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:21.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.842+0000 7f5924e14700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 0x7f58fc055ea0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f591001a000 tx=0x7f5910017000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:21.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:21.850+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f590c043310 con 0x7f59200a7fc0 2022-07-20T02:32:22.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.014+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f59200574e0 con 0x7f59200a7fc0 2022-07-20T02:32:22.630 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:32:22.622+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:32:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.630+0000 7f591e7fc700 1 -- 172.21.15.26:0/520864592 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) v1 ==== 161+0+0 (secure 0 0 0) 0x7f590c046930 con 0x7f59200a7fc0 2022-07-20T02:32:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 msgr2=0x7f58fc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 0x7f58fc055ea0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f591001a000 tx=0x7f5910017000).stop 2022-07-20T02:32:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 msgr2=0x7f59200ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f590c00c000 tx=0x7f590c006df0).stop 2022-07-20T02:32:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 shutdown_connections 2022-07-20T02:32:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58fc0539e0 0x7f58fc055ea0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59200b5ef0 0x7f592015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59200a8c40 0x7f5920159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 --2- 172.21.15.26:0/520864592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59200a7fc0 0x7f59200ab660 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 >> 172.21.15.26:0/520864592 conn(0x7f5920060310 msgr2=0x7f592005cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 shutdown_connections 2022-07-20T02:32:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.634+0000 7f5926617700 1 -- 172.21.15.26:0/520864592 wait complete. 2022-07-20T02:32:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:32:22.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:22.682 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:32:22.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.754+0000 7f1a89cad700 1 Processor -- start 2022-07-20T02:32:22.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.754+0000 7f1a89cad700 1 -- start start 2022-07-20T02:32:22.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1a84143090 0x7f1a84098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a840b0810 0x7f1a840950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a84143470 con 0x7f1a840b0c50 2022-07-20T02:32:22.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8405fd90 con 0x7f1a84143090 2022-07-20T02:32:22.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a84097d50 con 0x7f1a840b0810 2022-07-20T02:32:22.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1a84143090 0x7f1a84098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a83fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1a840b0c50 0x7f1a840955d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38962/0 (socket says 172.21.15.26:38962) 2022-07-20T02:32:22.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a83fff700 1 -- 172.21.15.26:0/290221663 learned_addr learned my addr 172.21.15.26:0/290221663 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:22.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a7bfff700 1 --2- 172.21.15.26:0/290221663 >> v2:172.21.15.120:3300/0 conn(0x7f1a840b0810 0x7f1a840950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a82ffd700 1 -- 172.21.15.26:0/290221663 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686915146 0 0) 0x7f1a84143470 con 0x7f1a840b0c50 2022-07-20T02:32:22.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a82ffd700 1 -- 172.21.15.26:0/290221663 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a64003260 con 0x7f1a840b0c50 2022-07-20T02:32:22.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 -- 172.21.15.26:0/290221663 >> v2:172.21.15.120:3300/0 conn(0x7f1a840b0810 msgr2=0x7f1a840950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:22.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 --2- 172.21.15.26:0/290221663 >> v2:172.21.15.120:3300/0 conn(0x7f1a840b0810 0x7f1a840950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 -- 172.21.15.26:0/290221663 >> v1:172.21.15.26:6789/0 conn(0x7f1a840b0c50 legacy=0x7f1a840955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 -- 172.21.15.26:0/290221663 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a84097ff0 con 0x7f1a84143090 2022-07-20T02:32:22.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a837fe700 1 --2- 172.21.15.26:0/290221663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a84143090 0x7f1a84098490 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f1a74002f70 tx=0x7f1a74006df0).ready entity=mon.0 client_cookie=3c7a6b05a027c312 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:22.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a82ffd700 1 -- 172.21.15.26:0/290221663 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7400d4f0 con 0x7f1a84143090 2022-07-20T02:32:22.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a82ffd700 1 -- 172.21.15.26:0/290221663 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74000e90 con 0x7f1a84143090 2022-07-20T02:32:22.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a82ffd700 1 -- 172.21.15.26:0/290221663 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7400d840 con 0x7f1a84143090 2022-07-20T02:32:22.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- 172.21.15.26:0/290221663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a84143090 msgr2=0x7f1a84098490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 --2- 172.21.15.26:0/290221663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a84143090 0x7f1a84098490 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f1a74002f70 tx=0x7f1a74006df0).stop 2022-07-20T02:32:22.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- 172.21.15.26:0/290221663 shutdown_connections 2022-07-20T02:32:22.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 --2- 172.21.15.26:0/290221663 >> v2:172.21.15.120:3300/0 conn(0x7f1a840b0810 0x7f1a840950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 --2- 172.21.15.26:0/290221663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a84143090 0x7f1a84098490 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- 172.21.15.26:0/290221663 >> 172.21.15.26:0/290221663 conn(0x7f1a840b5e70 msgr2=0x7f1a840b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:22.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- 172.21.15.26:0/290221663 shutdown_connections 2022-07-20T02:32:22.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.758+0000 7f1a89cad700 1 -- 172.21.15.26:0/290221663 wait complete. 2022-07-20T02:32:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.762+0000 7f1a89cad700 1 Processor -- start 2022-07-20T02:32:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.762+0000 7f1a89cad700 1 -- start start 2022-07-20T02:32:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.762+0000 7f1a89cad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.762+0000 7f1a89cad700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a840b0c50 0x7f1a8413cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.762+0000 7f1a89cad700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 0x7f1a8413d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a840aed80 con 0x7f1a840b0810 2022-07-20T02:32:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a840af920 con 0x7f1a840b0c50 2022-07-20T02:32:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a840afbc0 con 0x7f1a84143090 2022-07-20T02:32:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39684/0 (socket says 172.21.15.26:39684) 2022-07-20T02:32:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 -- 172.21.15.26:0/3437302804 learned_addr learned my addr 172.21.15.26:0/3437302804 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a7bfff700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a840b0c50 0x7f1a8413cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a83fff700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 0x7f1a8413d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 -- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 msgr2=0x7f1a8413d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 0x7f1a8413d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 -- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a840b0c50 msgr2=0x7f1a8413cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a840b0c50 0x7f1a8413cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 -- 172.21.15.26:0/3437302804 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7400b180 con 0x7f1a840b0810 2022-07-20T02:32:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a83fff700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 0x7f1a8413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a837fe700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f1a74002f70 tx=0x7f1a74014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a74016070 con 0x7f1a840b0810 2022-07-20T02:32:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74013a00 con 0x7f1a840b0810 2022-07-20T02:32:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a840aa730 con 0x7f1a840b0810 2022-07-20T02:32:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a74013d60 con 0x7f1a840b0810 2022-07-20T02:32:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8413d700 con 0x7f1a840b0810 2022-07-20T02:32:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a84095ce0 con 0x7f1a840b0810 2022-07-20T02:32:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1a74025030 con 0x7f1a840b0810 2022-07-20T02:32:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 0x7f1a68055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f1a7406cfd0 con 0x7f1a840b0810 2022-07-20T02:32:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.766+0000 7f1a7bfff700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 0x7f1a68055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:22.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.770+0000 7f1a7bfff700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 0x7f1a68055e10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1a6c00b000 tx=0x7f1a6c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.774+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a74043d10 con 0x7f1a840b0810 2022-07-20T02:32:22.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.938+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a840574e0 con 0x7f1a840b0810 2022-07-20T02:32:22.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.938+0000 7f1a817fa700 1 -- 172.21.15.26:0/3437302804 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2132 (secure 0 0 0) 0x7f1a74023090 con 0x7f1a840b0810 2022-07-20T02:32:22.945 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:22.945 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:31:24.761580+0000","modified":"2022-07-20T02:32:22.625368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T02:32:22.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 msgr2=0x7f1a68055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 0x7f1a68055e10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1a6c00b000 tx=0x7f1a6c006df0).stop 2022-07-20T02:32:22.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 msgr2=0x7f1a84060f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:22.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f1a74002f70 tx=0x7f1a74014000).stop 2022-07-20T02:32:22.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 shutdown_connections 2022-07-20T02:32:22.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a68053950 0x7f1a68055e10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a84143090 0x7f1a8413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a840b0c50 0x7f1a8413cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 --2- 172.21.15.26:0/3437302804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a840b0810 0x7f1a84060f70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:22.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 >> 172.21.15.26:0/3437302804 conn(0x7f1a840b5e70 msgr2=0x7f1a84142540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:22.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 shutdown_connections 2022-07-20T02:32:22.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:22.942+0000 7f1a89cad700 1 -- 172.21.15.26:0/3437302804 wait complete. 2022-07-20T02:32:22.957 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 38 2022-07-20T02:32:22.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:32:23.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 Processor -- start 2022-07-20T02:32:23.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 -- start start 2022-07-20T02:32:23.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa7dc098320 0x7fa7dc0a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc098800 con 0x7fa7dc0b1c60 2022-07-20T02:32:23.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc098aa0 con 0x7fa7dc060730 2022-07-20T02:32:23.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e3bd1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc05f8e0 con 0x7fa7dc098320 2022-07-20T02:32:23.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e116c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa7dc098320 0x7fa7dc0a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e196d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa7dc0b1c60 0x7fa7dc060020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38972/0 (socket says 172.21.15.26:38972) 2022-07-20T02:32:23.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.070+0000 7fa7e196d700 1 -- 172.21.15.26:0/89042259 learned_addr learned my addr 172.21.15.26:0/89042259 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:23.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507859483 0 0) 0x7fa7dc098aa0 con 0x7fa7dc060730 2022-07-20T02:32:23.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c8003260 con 0x7fa7dc060730 2022-07-20T02:32:23.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174775929 0 0) 0x7fa7dc098800 con 0x7fa7dc0b1c60 2022-07-20T02:32:23.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e116c700 1 -- 172.21.15.26:0/89042259 >> v1:172.21.15.120:6789/0 conn(0x7fa7dc060730 legacy=0x7fa7dc142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e116c700 1 -- 172.21.15.26:0/89042259 >> v1:172.21.15.26:6789/0 conn(0x7fa7dc0b1c60 legacy=0x7fa7dc060020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e116c700 1 -- 172.21.15.26:0/89042259 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc05fb80 con 0x7fa7dc098320 2022-07-20T02:32:23.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e116c700 1 --2- 172.21.15.26:0/89042259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9710 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fa7d00032f0 tx=0x7fa7d00091d0).ready entity=mon.0 client_cookie=47841bb1c65d25c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7d000b6c0 con 0x7fa7dc098320 2022-07-20T02:32:23.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d000b820 con 0x7fa7dc098320 2022-07-20T02:32:23.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e096b700 1 -- 172.21.15.26:0/89042259 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7d000b990 con 0x7fa7dc098320 2022-07-20T02:32:23.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/89042259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 msgr2=0x7fa7dc0a9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/89042259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9710 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fa7d00032f0 tx=0x7fa7d00091d0).stop 2022-07-20T02:32:23.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/89042259 shutdown_connections 2022-07-20T02:32:23.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/89042259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9710 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/89042259 >> 172.21.15.26:0/89042259 conn(0x7fa7dc0b5e70 msgr2=0x7fa7dc0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/89042259 shutdown_connections 2022-07-20T02:32:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/89042259 wait complete. 2022-07-20T02:32:23.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 Processor -- start 2022-07-20T02:32:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.074+0000 7fa7e3bd1700 1 -- start start 2022-07-20T02:32:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7dc060730 0x7fa7dc0b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7dc0b1c60 0x7fa7dc143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0aefc0 con 0x7fa7dc098320 2022-07-20T02:32:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0ae910 con 0x7fa7dc0b1c60 2022-07-20T02:32:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0aebb0 con 0x7fa7dc060730 2022-07-20T02:32:23.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39698/0 (socket says 172.21.15.26:39698) 2022-07-20T02:32:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 -- 172.21.15.26:0/2769089871 learned_addr learned my addr 172.21.15.26:0/2769089871 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e216e700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7dc0b1c60 0x7fa7dc143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e196d700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7dc060730 0x7fa7dc0b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 -- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7dc060730 msgr2=0x7fa7dc0b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7dc060730 0x7fa7dc0b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 -- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7dc0b1c60 msgr2=0x7fa7dc143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7dc0b1c60 0x7fa7dc143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 -- 172.21.15.26:0/2769089871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d0009470 con 0x7fa7dc098320 2022-07-20T02:32:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e116c700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7fa7d00032f0 tx=0x7fa7d0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7d000b6c0 con 0x7fa7dc098320 2022-07-20T02:32:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d0011e20 con 0x7fa7dc098320 2022-07-20T02:32:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc0af1f0 con 0x7fa7dc098320 2022-07-20T02:32:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7d001f5e0 con 0x7fa7dc098320 2022-07-20T02:32:23.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.078+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc0ae600 con 0x7fa7dc098320 2022-07-20T02:32:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc0af380 con 0x7fa7dc098320 2022-07-20T02:32:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa7d0026030 con 0x7fa7dc098320 2022-07-20T02:32:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7daffd700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 0x7fa7bc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6549+0+0 (secure 0 0 0) 0x7fa7d006cb80 con 0x7fa7dc098320 2022-07-20T02:32:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7e196d700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 0x7fa7bc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.082+0000 7fa7e196d700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 0x7fa7bc055e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fa7cc00d000 tx=0x7fa7cc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.090+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa7d00437b0 con 0x7fa7dc098320 2022-07-20T02:32:23.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.254+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa7dc0574e0 con 0x7fa7dc098320 2022-07-20T02:32:23.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.650+0000 7fa7daffd700 1 -- 172.21.15.26:0/2769089871 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa7d0046dd0 con 0x7fa7dc098320 2022-07-20T02:32:23.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 msgr2=0x7fa7bc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 0x7fa7bc055e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fa7cc00d000 tx=0x7fa7cc009040).stop 2022-07-20T02:32:23.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 msgr2=0x7fa7dc0a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7fa7d00032f0 tx=0x7fa7d0016000).stop 2022-07-20T02:32:23.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 shutdown_connections 2022-07-20T02:32:23.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7dc098320 0x7fa7dc0a9c80 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7bc053980 0x7fa7bc055e40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7dc0b1c60 0x7fa7dc143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 --2- 172.21.15.26:0/2769089871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7dc060730 0x7fa7dc0b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 >> 172.21.15.26:0/2769089871 conn(0x7fa7dc0b5e70 msgr2=0x7fa7dc060e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:23.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 shutdown_connections 2022-07-20T02:32:23.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.654+0000 7fa7e3bd1700 1 -- 172.21.15.26:0/2769089871 wait complete. 2022-07-20T02:32:23.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:32:23.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 Processor -- start 2022-07-20T02:32:23.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 -- start start 2022-07-20T02:32:23.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff55c0b2850 0x7ff55c0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c065810 con 0x7ff55c0b3290 2022-07-20T02:32:23.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c0ac3f0 con 0x7ff55c0b7780 2022-07-20T02:32:23.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff56109f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c066db0 con 0x7ff55c0b2850 2022-07-20T02:32:23.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff55b59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff55c0b3290 0x7ff55c1406d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38990/0 (socket says 172.21.15.26:38990) 2022-07-20T02:32:23.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff55b59e700 1 -- 172.21.15.26:0/1402309245 learned_addr learned my addr 172.21.15.26:0/1402309245 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:23.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.786+0000 7ff55a59c700 1 --2- 172.21.15.26:0/1402309245 >> v2:172.21.15.120:3300/0 conn(0x7ff55c0b2850 0x7ff55c0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff559d9b700 1 -- 172.21.15.26:0/1402309245 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124157159 0 0) 0x7ff55c065810 con 0x7ff55c0b3290 2022-07-20T02:32:23.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff559d9b700 1 -- 172.21.15.26:0/1402309245 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540003260 con 0x7ff55c0b3290 2022-07-20T02:32:23.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff55a59c700 1 -- 172.21.15.26:0/1402309245 >> v1:172.21.15.120:6789/0 conn(0x7ff55c0b7780 legacy=0x7ff55c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff55a59c700 1 -- 172.21.15.26:0/1402309245 >> v1:172.21.15.26:6789/0 conn(0x7ff55c0b3290 legacy=0x7ff55c1406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff55a59c700 1 -- 172.21.15.26:0/1402309245 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c0b2dd0 con 0x7ff55c0b2850 2022-07-20T02:32:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff55a59c700 1 --2- 172.21.15.26:0/1402309245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b7b60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff550002f70 tx=0x7ff550006df0).ready entity=mon.1 client_cookie=a63c72267be56a56 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff559d9b700 1 -- 172.21.15.26:0/1402309245 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55000d610 con 0x7ff55c0b2850 2022-07-20T02:32:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff559d9b700 1 -- 172.21.15.26:0/1402309245 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550000e90 con 0x7ff55c0b2850 2022-07-20T02:32:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff559d9b700 1 -- 172.21.15.26:0/1402309245 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55000d960 con 0x7ff55c0b2850 2022-07-20T02:32:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 -- 172.21.15.26:0/1402309245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 msgr2=0x7ff55c0b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 --2- 172.21.15.26:0/1402309245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b7b60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff550002f70 tx=0x7ff550006df0).stop 2022-07-20T02:32:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 -- 172.21.15.26:0/1402309245 shutdown_connections 2022-07-20T02:32:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 --2- 172.21.15.26:0/1402309245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b7b60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 -- 172.21.15.26:0/1402309245 >> 172.21.15.26:0/1402309245 conn(0x7ff55c064b10 msgr2=0x7ff55c064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 -- 172.21.15.26:0/1402309245 shutdown_connections 2022-07-20T02:32:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.790+0000 7ff56109f700 1 -- 172.21.15.26:0/1402309245 wait complete. 2022-07-20T02:32:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 Processor -- start 2022-07-20T02:32:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 -- start start 2022-07-20T02:32:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff55c0b7780 0x7ff55c152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c142ca0 con 0x7ff55c0b3290 2022-07-20T02:32:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c14a410 con 0x7ff55c0b2850 2022-07-20T02:32:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff56109f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c14a6b0 con 0x7ff55c0b7780 2022-07-20T02:32:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39714/0 (socket says 172.21.15.26:39714) 2022-07-20T02:32:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 -- 172.21.15.26:0/803348177 learned_addr learned my addr 172.21.15.26:0/803348177 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55ad9d700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55b59e700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff55c0b7780 0x7ff55c152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 -- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff55c0b7780 msgr2=0x7ff55c152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff55c0b7780 0x7ff55c152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 -- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 msgr2=0x7ff55c0b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 -- 172.21.15.26:0/803348177 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55000b180 con 0x7ff55c0b3290 2022-07-20T02:32:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.794+0000 7ff55a59c700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7ff550002f70 tx=0x7ff550014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550016070 con 0x7ff55c0b3290 2022-07-20T02:32:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55000de90 con 0x7ff55c0b3290 2022-07-20T02:32:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c152470 con 0x7ff55c0b3290 2022-07-20T02:32:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550013da0 con 0x7ff55c0b3290 2022-07-20T02:32:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c0b0c40 con 0x7ff55c0b3290 2022-07-20T02:32:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c14aa30 con 0x7ff55c0b3290 2022-07-20T02:32:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff550024030 con 0x7ff55c0b3290 2022-07-20T02:32:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 0x7ff538055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6549+0+0 (secure 0 0 0) 0x7ff55006cf20 con 0x7ff55c0b3290 2022-07-20T02:32:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.798+0000 7ff55ad9d700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 0x7ff538055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:23.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.802+0000 7ff55ad9d700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 0x7ff538055fc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff54c00d000 tx=0x7ff54c007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:23.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.806+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff550043ad0 con 0x7ff55c0b3290 2022-07-20T02:32:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:23.934+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff55c065a40 con 0x7ff55c0b3290 2022-07-20T02:32:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.650+0000 7ff54bfff700 1 -- 172.21.15.26:0/803348177 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 185+0+0 (secure 0 0 0) 0x7ff550022020 con 0x7ff55c0b3290 2022-07-20T02:32:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 msgr2=0x7ff538055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 0x7ff538055fc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff54c00d000 tx=0x7ff54c007b90).stop 2022-07-20T02:32:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 msgr2=0x7ff55c142440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7ff550002f70 tx=0x7ff550014000).stop 2022-07-20T02:32:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 shutdown_connections 2022-07-20T02:32:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff55c0b3290 0x7ff55c142440 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff538053b00 0x7ff538055fc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff55c0b7780 0x7ff55c152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 --2- 172.21.15.26:0/803348177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff55c0b2850 0x7ff55c0b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 >> 172.21.15.26:0/803348177 conn(0x7ff55c064b10 msgr2=0x7ff55c144f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 shutdown_connections 2022-07-20T02:32:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.654+0000 7ff56109f700 1 -- 172.21.15.26:0/803348177 wait complete. 2022-07-20T02:32:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:32:24.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:32:24.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.798+0000 7f8fed9a0700 1 Processor -- start 2022-07-20T02:32:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.798+0000 7f8fed9a0700 1 -- start start 2022-07-20T02:32:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed9a0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8fe80add30 0x7f8fe80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed9a0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 0x7f8fe80afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed9a0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80aa1e0 con 0x7f8fe8060950 2022-07-20T02:32:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed9a0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80ae110 con 0x7f8fe80add30 2022-07-20T02:32:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed9a0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80a9f40 con 0x7f8fe80af6d0 2022-07-20T02:32:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8fe80add30 0x7f8fe80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8fe8060950 0x7f8fe8060dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39002/0 (socket says 172.21.15.26:39002) 2022-07-20T02:32:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe7fff700 1 -- 172.21.15.26:0/827558843 learned_addr learned my addr 172.21.15.26:0/827558843 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed19f700 1 --2- 172.21.15.26:0/827558843 >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 0x7f8fe80afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354573311 0 0) 0x7f8fe80aa1e0 con 0x7f8fe8060950 2022-07-20T02:32:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fcc003260 con 0x7f8fe8060950 2022-07-20T02:32:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164938947 0 0) 0x7f8fcc003260 con 0x7f8fe8060950 2022-07-20T02:32:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 -- 172.21.15.26:0/827558843 >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 msgr2=0x7f8fe80afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 --2- 172.21.15.26:0/827558843 >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 0x7f8fe80afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 -- 172.21.15.26:0/827558843 >> v1:172.21.15.26:6789/0 conn(0x7f8fe8060950 legacy=0x7f8fe8060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 -- 172.21.15.26:0/827558843 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80b1110 con 0x7f8fe80add30 2022-07-20T02:32:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fed19f700 1 --2- 172.21.15.26:0/827558843 >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 0x7f8fe80afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fec99e700 1 --2- 172.21.15.26:0/827558843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe80add30 0x7f8fe80b2ab0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f8fd4001800 tx=0x7f8fdc006df0).ready entity=mon.0 client_cookie=800923232fa15ffe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fdc014070 con 0x7f8fe80add30 2022-07-20T02:32:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc000e90 con 0x7f8fe80add30 2022-07-20T02:32:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.802+0000 7f8fe77fe700 1 -- 172.21.15.26:0/827558843 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fd4001660 con 0x7f8fe80add30 2022-07-20T02:32:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- 172.21.15.26:0/827558843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe80add30 msgr2=0x7f8fe80b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/827558843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe80add30 0x7f8fe80b2ab0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f8fd4001800 tx=0x7f8fdc006df0).stop 2022-07-20T02:32:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- 172.21.15.26:0/827558843 shutdown_connections 2022-07-20T02:32:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/827558843 >> v2:172.21.15.120:3300/0 conn(0x7f8fe80af6d0 0x7f8fe80afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/827558843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe80add30 0x7f8fe80b2ab0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- 172.21.15.26:0/827558843 >> 172.21.15.26:0/827558843 conn(0x7f8fe80b3050 msgr2=0x7f8fe80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:24.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- 172.21.15.26:0/827558843 shutdown_connections 2022-07-20T02:32:24.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- 172.21.15.26:0/827558843 wait complete. 2022-07-20T02:32:24.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 Processor -- start 2022-07-20T02:32:24.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.806+0000 7f8fed9a0700 1 -- start start 2022-07-20T02:32:24.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 0x7f8fe80ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80af6d0 0x7f8fe8148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8147e50 con 0x7f8fe8060950 2022-07-20T02:32:24.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80ac750 con 0x7f8fe80af6d0 2022-07-20T02:32:24.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8149070 con 0x7f8fe80add30 2022-07-20T02:32:24.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39726/0 (socket says 172.21.15.26:39726) 2022-07-20T02:32:24.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 -- 172.21.15.26:0/3102595470 learned_addr learned my addr 172.21.15.26:0/3102595470 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:24.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed19f700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80af6d0 0x7f8fe8148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fe7fff700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 0x7f8fe80ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 -- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 msgr2=0x7f8fe80ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 0x7f8fe80ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 -- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80af6d0 msgr2=0x7f8fe8148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:24.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80af6d0 0x7f8fe8148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:24.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 -- 172.21.15.26:0/3102595470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0009470 con 0x7f8fe8060950 2022-07-20T02:32:24.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fe7fff700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 0x7f8fe80ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:24.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fec99e700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f8fd4001800 tx=0x7f8fdc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:24.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fdc00f5e0 con 0x7f8fe8060950 2022-07-20T02:32:24.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc010d50 con 0x7f8fe8060950 2022-07-20T02:32:24.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc00b180 con 0x7f8fe8060950 2022-07-20T02:32:24.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fdc01c630 con 0x7f8fe8060950 2022-07-20T02:32:24.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe8148770 con 0x7f8fe8060950 2022-07-20T02:32:24.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.810+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe80b13b0 con 0x7f8fe8060950 2022-07-20T02:32:24.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.814+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8fdc022020 con 0x7f8fe8060950 2022-07-20T02:32:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.814+0000 7f8fe5ffb700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 0x7f8fc8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.814+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f8fdc06d960 con 0x7f8fe8060950 2022-07-20T02:32:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.814+0000 7f8fe7fff700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 0x7f8fc8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.814+0000 7f8fe7fff700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 0x7f8fc8055f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8fd4008000 tx=0x7f8fd4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:24.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.822+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fdc044860 con 0x7f8fe8060950 2022-07-20T02:32:24.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:24.950+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8fe80574e0 con 0x7f8fe8060950 2022-07-20T02:32:25.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.666+0000 7f8fe5ffb700 1 -- 172.21.15.26:0/3102595470 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8fdc047e80 con 0x7f8fe8060950 2022-07-20T02:32:25.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 msgr2=0x7f8fc8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 0x7f8fc8055f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8fd4008000 tx=0x7f8fd4009040).stop 2022-07-20T02:32:25.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 msgr2=0x7f8fe8065b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f8fd4001800 tx=0x7f8fdc016000).stop 2022-07-20T02:32:25.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 shutdown_connections 2022-07-20T02:32:25.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fc8053aa0 0x7f8fc8055f60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80af6d0 0x7f8fe8148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80add30 0x7f8fe80ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 --2- 172.21.15.26:0/3102595470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fe8060950 0x7f8fe8065b50 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 >> 172.21.15.26:0/3102595470 conn(0x7f8fe80b3050 msgr2=0x7f8fe80b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 shutdown_connections 2022-07-20T02:32:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.674+0000 7f8fed9a0700 1 -- 172.21.15.26:0/3102595470 wait complete. 2022-07-20T02:32:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:32:25.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:32:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 Processor -- start 2022-07-20T02:32:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 -- start start 2022-07-20T02:32:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c5c060f00 0x7f5c5c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c1474e0 con 0x7f5c5c147000 2022-07-20T02:32:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c0b4e40 con 0x7f5c5c05cfd0 2022-07-20T02:32:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62b49700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c05d3b0 con 0x7f5c5c060f00 2022-07-20T02:32:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c61b47700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c61b47700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39736/0 (socket says 172.21.15.26:39736) 2022-07-20T02:32:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c61b47700 1 -- 172.21.15.26:0/1553637759 learned_addr learned my addr 172.21.15.26:0/1553637759 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c62348700 1 --2- 172.21.15.26:0/1553637759 >> v2:172.21.15.120:3300/0 conn(0x7f5c5c060f00 0x7f5c5c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.794+0000 7f5c61b47700 1 -- 172.21.15.26:0/1553637759 >> v2:172.21.15.120:3300/0 conn(0x7f5c5c060f00 msgr2=0x7f5c5c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c61b47700 1 --2- 172.21.15.26:0/1553637759 >> v2:172.21.15.120:3300/0 conn(0x7f5c5c060f00 0x7f5c5c13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c61b47700 1 -- 172.21.15.26:0/1553637759 >> v1:172.21.15.120:6789/0 conn(0x7f5c5c147000 legacy=0x7f5c5c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c60b45700 1 -- 172.21.15.26:0/1553637759 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696231662 0 0) 0x7f5c5c1474e0 con 0x7f5c5c147000 2022-07-20T02:32:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c61b47700 1 -- 172.21.15.26:0/1553637759 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c1431e0 con 0x7f5c5c05cfd0 2022-07-20T02:32:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c61b47700 1 --2- 172.21.15.26:0/1553637759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f5c4c002f70 tx=0x7f5c4c006eb0).ready entity=mon.0 client_cookie=9285fb6cea6594c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c60b45700 1 -- 172.21.15.26:0/1553637759 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c4c00d5c0 con 0x7f5c5c05cfd0 2022-07-20T02:32:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c60b45700 1 -- 172.21.15.26:0/1553637759 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c000e90 con 0x7f5c5c05cfd0 2022-07-20T02:32:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c60b45700 1 -- 172.21.15.26:0/1553637759 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c4c00d910 con 0x7f5c5c05cfd0 2022-07-20T02:32:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 -- 172.21.15.26:0/1553637759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c05cfd0 msgr2=0x7f5c5c143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 --2- 172.21.15.26:0/1553637759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f5c4c002f70 tx=0x7f5c4c006eb0).stop 2022-07-20T02:32:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 -- 172.21.15.26:0/1553637759 shutdown_connections 2022-07-20T02:32:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 --2- 172.21.15.26:0/1553637759 >> v2:172.21.15.120:3300/0 conn(0x7f5c5c060f00 0x7f5c5c13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 --2- 172.21.15.26:0/1553637759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c05cfd0 0x7f5c5c143bc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 -- 172.21.15.26:0/1553637759 >> 172.21.15.26:0/1553637759 conn(0x7f5c5c0620c0 msgr2=0x7f5c5c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 -- 172.21.15.26:0/1553637759 shutdown_connections 2022-07-20T02:32:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.798+0000 7f5c62b49700 1 -- 172.21.15.26:0/1553637759 wait complete. 2022-07-20T02:32:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 Processor -- start 2022-07-20T02:32:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 -- start start 2022-07-20T02:32:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c5c060f00 0x7f5c5c1562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c5c147000 0x7f5c5c144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0b8600 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0ad810 con 0x7f5c5c060f00 2022-07-20T02:32:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62b49700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c144a60 con 0x7f5c5c147000 2022-07-20T02:32:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c61b47700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c5c060f00 0x7f5c5c1562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39744/0 (socket says 172.21.15.26:39744) 2022-07-20T02:32:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 -- 172.21.15.26:0/4096349035 learned_addr learned my addr 172.21.15.26:0/4096349035 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c61346700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c5c147000 0x7f5c5c144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 -- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c5c147000 msgr2=0x7f5c5c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c5c147000 0x7f5c5c144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 -- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c5c060f00 msgr2=0x7f5c5c1562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c5c060f00 0x7f5c5c1562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.802+0000 7f5c62348700 1 -- 172.21.15.26:0/4096349035 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c00b1e0 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c62348700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f5c54006e00 tx=0x7f5c540091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:25.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c54012600 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54012760 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c54012ac0 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c146980 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c0b7cf0 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5c54022070 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c144de0 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 0x7f5c3c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f5c5406c460 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.806+0000 7f5c61b47700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 0x7f5c3c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.810+0000 7f5c61b47700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 0x7f5c3c055f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f5c4c01c000 tx=0x7f5c4c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.810+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c54043430 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.942+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5c5c0ad980 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c4affd700 1 -- 172.21.15.26:0/4096349035 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c540160a0 con 0x7f5c5c0b92f0 2022-07-20T02:32:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 msgr2=0x7f5c3c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 0x7f5c3c055f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f5c4c01c000 tx=0x7f5c4c015000).stop 2022-07-20T02:32:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 msgr2=0x7f5c5c146490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f5c54006e00 tx=0x7f5c540091d0).stop 2022-07-20T02:32:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 shutdown_connections 2022-07-20T02:32:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5c5c0b92f0 0x7f5c5c146490 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5c3c053a70 0x7f5c3c055f30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c5c147000 0x7f5c5c144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 --2- 172.21.15.26:0/4096349035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c5c060f00 0x7f5c5c1562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 >> 172.21.15.26:0/4096349035 conn(0x7f5c5c05cfd0 msgr2=0x7f5c5c0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 shutdown_connections 2022-07-20T02:32:25.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:25.950+0000 7f5c62b49700 1 -- 172.21.15.26:0/4096349035 wait complete. 2022-07-20T02:32:25.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-20T02:32:26.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 Processor -- start 2022-07-20T02:32:26.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 -- start start 2022-07-20T02:32:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbd805cfd0 0x7fcbd8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcbd8147000 0x7fcbd80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd81474e0 con 0x7fcbd8060f00 2022-07-20T02:32:26.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.090+0000 7fcbdedce700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd805d3b0 con 0x7fcbd805cfd0 2022-07-20T02:32:26.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd81431e0 con 0x7fcbd8147000 2022-07-20T02:32:26.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbd805cfd0 0x7fcbd8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbd805cfd0 0x7fcbd8143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39748/0 (socket says 172.21.15.26:39748) 2022-07-20T02:32:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 -- 172.21.15.26:0/3301576977 learned_addr learned my addr 172.21.15.26:0/3301576977 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdd5cb700 1 --2- 172.21.15.26:0/3301576977 >> v2:172.21.15.120:3300/0 conn(0x7fcbd8147000 0x7fcbd80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 -- 172.21.15.26:0/3301576977 >> v2:172.21.15.120:3300/0 conn(0x7fcbd8147000 msgr2=0x7fcbd80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 --2- 172.21.15.26:0/3301576977 >> v2:172.21.15.120:3300/0 conn(0x7fcbd8147000 0x7fcbd80b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 -- 172.21.15.26:0/3301576977 >> v1:172.21.15.120:6789/0 conn(0x7fcbd8060f00 legacy=0x7fcbd813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 -- 172.21.15.26:0/3301576977 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8143480 con 0x7fcbd805cfd0 2022-07-20T02:32:26.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdddcc700 1 --2- 172.21.15.26:0/3301576977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd805cfd0 0x7fcbd8143bc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fcbd4002f70 tx=0x7fcbd4006eb0).ready entity=mon.0 client_cookie=5f48450a0639bcdf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdcdca700 1 -- 172.21.15.26:0/3301576977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd400d5c0 con 0x7fcbd805cfd0 2022-07-20T02:32:26.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdcdca700 1 -- 172.21.15.26:0/3301576977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4000e90 con 0x7fcbd805cfd0 2022-07-20T02:32:26.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdcdca700 1 -- 172.21.15.26:0/3301576977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd400d950 con 0x7fcbd805cfd0 2022-07-20T02:32:26.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- 172.21.15.26:0/3301576977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd805cfd0 msgr2=0x7fcbd8143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 --2- 172.21.15.26:0/3301576977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd805cfd0 0x7fcbd8143bc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fcbd4002f70 tx=0x7fcbd4006eb0).stop 2022-07-20T02:32:26.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- 172.21.15.26:0/3301576977 shutdown_connections 2022-07-20T02:32:26.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 --2- 172.21.15.26:0/3301576977 >> v2:172.21.15.120:3300/0 conn(0x7fcbd8147000 0x7fcbd80b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 --2- 172.21.15.26:0/3301576977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd805cfd0 0x7fcbd8143bc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- 172.21.15.26:0/3301576977 >> 172.21.15.26:0/3301576977 conn(0x7fcbd80620c0 msgr2=0x7fcbd805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- 172.21.15.26:0/3301576977 shutdown_connections 2022-07-20T02:32:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.094+0000 7fcbdedce700 1 -- 172.21.15.26:0/3301576977 wait complete. 2022-07-20T02:32:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 Processor -- start 2022-07-20T02:32:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- start start 2022-07-20T02:32:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbd8060f00 0x7fcbd80b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 0x7fcbd81380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81561d0 con 0x7fcbd8144910 2022-07-20T02:32:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8156440 con 0x7fcbd8147000 2022-07-20T02:32:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81566e0 con 0x7fcbd8060f00 2022-07-20T02:32:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 0x7fcbd81380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdd5cb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdd5cb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59446/0 (socket says 172.21.15.26:59446) 2022-07-20T02:32:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdd5cb700 1 -- 172.21.15.26:0/361961706 learned_addr learned my addr 172.21.15.26:0/361961706 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdddcc700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbd8060f00 0x7fcbd80b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 -- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbd8060f00 msgr2=0x7fcbd80b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbd8060f00 0x7fcbd80b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 -- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 msgr2=0x7fcbd8144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 -- 172.21.15.26:0/361961706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd400b1e0 con 0x7fcbd8144910 2022-07-20T02:32:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdd5cb700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbde5cd700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 0x7fcbd81380b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fcbd8068660 tx=0x7fcbd0007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd000f470 con 0x7fcbd8144910 2022-07-20T02:32:26.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd000f5d0 con 0x7fcbd8144910 2022-07-20T02:32:26.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd81468d0 con 0x7fcbd8144910 2022-07-20T02:32:26.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd000f930 con 0x7fcbd8144910 2022-07-20T02:32:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.098+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd8146d70 con 0x7fcbd8144910 2022-07-20T02:32:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd8146a60 con 0x7fcbd8144910 2022-07-20T02:32:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbd0010070 con 0x7fcbd8144910 2022-07-20T02:32:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbcaffd700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 0x7fcbb8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbdddcc700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 0x7fcbb8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fcbd006b280 con 0x7fcbd8144910 2022-07-20T02:32:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.102+0000 7fcbdddcc700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 0x7fcbb8055f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fcbd401c000 tx=0x7fcbd4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.110+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbd00422f0 con 0x7fcbd8144910 2022-07-20T02:32:26.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.274+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7fcbd8143720 con 0x7fcbd8144910 2022-07-20T02:32:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.670+0000 7fcbcaffd700 1 -- 172.21.15.26:0/361961706 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v302) v1 ==== 139+0+0 (secure 0 0 0) 0x7fcbd0045910 con 0x7fcbd8144910 2022-07-20T02:32:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 msgr2=0x7fcbb8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 0x7fcbb8055f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fcbd401c000 tx=0x7fcbd4015000).stop 2022-07-20T02:32:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 msgr2=0x7fcbd81380b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 0x7fcbd81380b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fcbd8068660 tx=0x7fcbd0007eb0).stop 2022-07-20T02:32:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 shutdown_connections 2022-07-20T02:32:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbd8144910 0x7fcbd81380b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcbb8053a70 0x7fcbb8055f30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbd8147000 0x7fcbd8144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 --2- 172.21.15.26:0/361961706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbd8060f00 0x7fcbd80b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 >> 172.21.15.26:0/361961706 conn(0x7fcbd805cfd0 msgr2=0x7fcbd80b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 shutdown_connections 2022-07-20T02:32:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.674+0000 7fcbdedce700 1 -- 172.21.15.26:0/361961706 wait complete. 2022-07-20T02:32:26.690 INFO:tasks.cephfs_test_runner:test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:32:26.691 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:32:26.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-20T02:32:26.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 Processor -- start 2022-07-20T02:32:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 -- start start 2022-07-20T02:32:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b51f0 0x7ff384098750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3840a95b0 0x7ff3840b0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840b55d0 con 0x7ff3840b2430 2022-07-20T02:32:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3840b1f10 con 0x7ff3840a95b0 2022-07-20T02:32:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38bb3e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840996b0 con 0x7ff3840b51f0 2022-07-20T02:32:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3840a95b0 0x7ff3840b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3840a95b0 0x7ff3840b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39766/0 (socket says 172.21.15.26:39766) 2022-07-20T02:32:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff3890d9700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff3840b2430 0x7ff384098fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39044/0 (socket says 172.21.15.26:39044) 2022-07-20T02:32:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 -- 172.21.15.26:0/508715908 learned_addr learned my addr 172.21.15.26:0/508715908 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff3898da700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b51f0 0x7ff384098750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff3888d8700 1 -- 172.21.15.26:0/508715908 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400277102 0 0) 0x7ff3840b55d0 con 0x7ff3840b2430 2022-07-20T02:32:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff3888d8700 1 -- 172.21.15.26:0/508715908 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff370003260 con 0x7ff3840b2430 2022-07-20T02:32:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 -- 172.21.15.26:0/508715908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b51f0 msgr2=0x7ff384098750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b51f0 0x7ff384098750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 -- 172.21.15.26:0/508715908 >> v1:172.21.15.26:6789/0 conn(0x7ff3840b2430 legacy=0x7ff384098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.810+0000 7ff38a0db700 1 -- 172.21.15.26:0/508715908 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff384097300 con 0x7ff3840a95b0 2022-07-20T02:32:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38a0db700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840a95b0 0x7ff3840b0840 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7ff37c0032f0 tx=0x7ff37c009290).ready entity=mon.0 client_cookie=f0982b9dea523e99 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff3888d8700 1 -- 172.21.15.26:0/508715908 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c00b6c0 con 0x7ff3840a95b0 2022-07-20T02:32:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff3888d8700 1 -- 172.21.15.26:0/508715908 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c00b820 con 0x7ff3840a95b0 2022-07-20T02:32:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff3888d8700 1 -- 172.21.15.26:0/508715908 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c00b990 con 0x7ff3840a95b0 2022-07-20T02:32:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- 172.21.15.26:0/508715908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840a95b0 msgr2=0x7ff3840b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840a95b0 0x7ff3840b0840 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7ff37c0032f0 tx=0x7ff37c009290).stop 2022-07-20T02:32:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- 172.21.15.26:0/508715908 shutdown_connections 2022-07-20T02:32:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840a95b0 0x7ff3840b0840 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/508715908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b51f0 0x7ff384098750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- 172.21.15.26:0/508715908 >> 172.21.15.26:0/508715908 conn(0x7ff3840b5ea0 msgr2=0x7ff3840b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- 172.21.15.26:0/508715908 shutdown_connections 2022-07-20T02:32:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- 172.21.15.26:0/508715908 wait complete. 2022-07-20T02:32:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 Processor -- start 2022-07-20T02:32:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- start start 2022-07-20T02:32:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3840a95b0 0x7ff38414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 0x7ff384063340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff384066d00 con 0x7ff3840b51f0 2022-07-20T02:32:26.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff384090590 con 0x7ff3840a95b0 2022-07-20T02:32:26.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38bb3e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff384090090 con 0x7ff3840b2430 2022-07-20T02:32:26.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38a0db700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38a0db700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39772/0 (socket says 172.21.15.26:39772) 2022-07-20T02:32:26.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff38a0db700 1 -- 172.21.15.26:0/30119884 learned_addr learned my addr 172.21.15.26:0/30119884 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:26.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff3890d9700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 0x7ff384063340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.814+0000 7ff3898da700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3840a95b0 0x7ff38414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 -- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 msgr2=0x7ff384063340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 0x7ff384063340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 -- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3840a95b0 msgr2=0x7ff38414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:26.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3840a95b0 0x7ff38414a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:26.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 -- 172.21.15.26:0/30119884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c009530 con 0x7ff3840b51f0 2022-07-20T02:32:26.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff3890d9700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 0x7ff384063340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:26.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38a0db700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7ff37c0032f0 tx=0x7ff37c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c021480 con 0x7ff3840b51f0 2022-07-20T02:32:26.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c0215e0 con 0x7ff3840b51f0 2022-07-20T02:32:26.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff384063830 con 0x7ff3840b51f0 2022-07-20T02:32:26.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37c021940 con 0x7ff3840b51f0 2022-07-20T02:32:26.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3840902f0 con 0x7ff3840b51f0 2022-07-20T02:32:26.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3840574e0 con 0x7ff3840b51f0 2022-07-20T02:32:26.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff37c00f070 con 0x7ff3840b51f0 2022-07-20T02:32:26.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 0x7ff364055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:26.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7ff37c06d040 con 0x7ff3840b51f0 2022-07-20T02:32:26.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.818+0000 7ff3898da700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 0x7ff364055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:26.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.822+0000 7ff3898da700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 0x7ff364055ea0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff37400b000 tx=0x7ff374006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:26.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.826+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff37c0440f0 con 0x7ff3840b51f0 2022-07-20T02:32:26.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:26.994+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7ff38409b280 con 0x7ff3840b51f0 2022-07-20T02:32:27.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.674+0000 7ff382ffd700 1 -- 172.21.15.26:0/30119884 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v303) v1 ==== 129+0+0 (secure 0 0 0) 0x7ff37c015030 con 0x7ff3840b51f0 2022-07-20T02:32:27.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 msgr2=0x7ff364055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 0x7ff364055ea0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff37400b000 tx=0x7ff374006df0).stop 2022-07-20T02:32:27.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 msgr2=0x7ff384064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7ff37c0032f0 tx=0x7ff37c016000).stop 2022-07-20T02:32:27.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 shutdown_connections 2022-07-20T02:32:27.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3840b51f0 0x7ff384064b10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3640539e0 0x7ff364055ea0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3840b2430 0x7ff384063340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 --2- 172.21.15.26:0/30119884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3840a95b0 0x7ff38414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 >> 172.21.15.26:0/30119884 conn(0x7ff3840b5ea0 msgr2=0x7ff38405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:27.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 shutdown_connections 2022-07-20T02:32:27.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.678+0000 7ff38bb3e700 1 -- 172.21.15.26:0/30119884 wait complete. 2022-07-20T02:32:27.705 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:32:27.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 Processor -- start 2022-07-20T02:32:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 -- start start 2022-07-20T02:32:27.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 0x7f78cc060a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f78cc0b0680 0x7f78cc141460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc0ac430 con 0x7f78cc061010 2022-07-20T02:32:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc0b1d00 con 0x7f78cc0b0680 2022-07-20T02:32:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d3c0b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc0b2ea0 con 0x7f78cc0b2ac0 2022-07-20T02:32:27.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d11a6700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f78cc061010 0x7f78cc0aff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39058/0 (socket says 172.21.15.26:39058) 2022-07-20T02:32:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d11a6700 1 -- 172.21.15.26:0/3704244953 learned_addr learned my addr 172.21.15.26:0/3704244953 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:27.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d21a8700 1 --2- 172.21.15.26:0/3704244953 >> v2:172.21.15.26:3300/0 conn(0x7f78cc0b0680 0x7f78cc141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d19a7700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 0x7f78cc060a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d09a5700 1 -- 172.21.15.26:0/3704244953 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597955259 0 0) 0x7f78cc0ac430 con 0x7f78cc061010 2022-07-20T02:32:27.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d09a5700 1 -- 172.21.15.26:0/3704244953 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b4003260 con 0x7f78cc061010 2022-07-20T02:32:27.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d21a8700 1 -- 172.21.15.26:0/3704244953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 msgr2=0x7f78cc060a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:27.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d21a8700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 0x7f78cc060a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.814+0000 7f78d21a8700 1 -- 172.21.15.26:0/3704244953 >> v1:172.21.15.26:6789/0 conn(0x7f78cc061010 legacy=0x7f78cc0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d21a8700 1 -- 172.21.15.26:0/3704244953 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc0ac190 con 0x7f78cc0b0680 2022-07-20T02:32:27.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d19a7700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 0x7f78cc060a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:27.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d21a8700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc0b0680 0x7f78cc141460 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f78c4002f70 tx=0x7f78c4006df0).ready entity=mon.0 client_cookie=c56f7e94a003ef6f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:27.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d09a5700 1 -- 172.21.15.26:0/3704244953 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78c400d4a0 con 0x7f78cc0b0680 2022-07-20T02:32:27.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d09a5700 1 -- 172.21.15.26:0/3704244953 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4000e90 con 0x7f78cc0b0680 2022-07-20T02:32:27.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d09a5700 1 -- 172.21.15.26:0/3704244953 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78c400d7f0 con 0x7f78cc0b0680 2022-07-20T02:32:27.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 -- 172.21.15.26:0/3704244953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc0b0680 msgr2=0x7f78cc141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc0b0680 0x7f78cc141460 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f78c4002f70 tx=0x7f78c4006df0).stop 2022-07-20T02:32:27.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 -- 172.21.15.26:0/3704244953 shutdown_connections 2022-07-20T02:32:27.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc0b0680 0x7f78cc141460 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/3704244953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b2ac0 0x7f78cc060a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 -- 172.21.15.26:0/3704244953 >> 172.21.15.26:0/3704244953 conn(0x7f78cc0b60a0 msgr2=0x7f78cc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:27.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 -- 172.21.15.26:0/3704244953 shutdown_connections 2022-07-20T02:32:27.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.818+0000 7f78d3c0b700 1 -- 172.21.15.26:0/3704244953 wait complete. 2022-07-20T02:32:27.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 Processor -- start 2022-07-20T02:32:27.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 -- start start 2022-07-20T02:32:27.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 0x7f78cc066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78cc0b2ac0 0x7f78cc13b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc065e30 con 0x7f78cc061010 2022-07-20T02:32:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc063030 con 0x7f78cc0b2ac0 2022-07-20T02:32:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d3c0b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc150c50 con 0x7f78cc0b0680 2022-07-20T02:32:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39784/0 (socket says 172.21.15.26:39784) 2022-07-20T02:32:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 -- 172.21.15.26:0/4057095193 learned_addr learned my addr 172.21.15.26:0/4057095193 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:27.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d21a8700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78cc0b2ac0 0x7f78cc13b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d11a6700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 0x7f78cc066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 -- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 msgr2=0x7f78cc066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 0x7f78cc066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 -- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78cc0b2ac0 msgr2=0x7f78cc13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78cc0b2ac0 0x7f78cc13b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 -- 172.21.15.26:0/4057095193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc00b180 con 0x7f78cc061010 2022-07-20T02:32:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d11a6700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 0x7f78cc066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.822+0000 7f78d19a7700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f78bc00c000 tx=0x7f78bc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78bc010780 con 0x7f78cc061010 2022-07-20T02:32:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc0108e0 con 0x7f78cc061010 2022-07-20T02:32:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c400b180 con 0x7f78cc061010 2022-07-20T02:32:27.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78bc010c40 con 0x7f78cc061010 2022-07-20T02:32:27.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc13e520 con 0x7f78cc061010 2022-07-20T02:32:27.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc0b13a0 con 0x7f78cc061010 2022-07-20T02:32:27.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f78bc01c460 con 0x7f78cc061010 2022-07-20T02:32:27.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 0x7f78ac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:27.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f78bc06c3e0 con 0x7f78cc061010 2022-07-20T02:32:27.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.826+0000 7f78d11a6700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 0x7f78ac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:27.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.830+0000 7f78d11a6700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 0x7f78ac055f30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f78c800b000 tx=0x7f78c8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:27.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.830+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f78bc0432f0 con 0x7f78cc061010 2022-07-20T02:32:27.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.962+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f78cc0574e0 con 0x7f78cc061010 2022-07-20T02:32:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78baffd700 1 -- 172.21.15.26:0/4057095193 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f78bc046910 con 0x7f78cc061010 2022-07-20T02:32:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 msgr2=0x7f78ac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 0x7f78ac055f30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f78c800b000 tx=0x7f78c8009040).stop 2022-07-20T02:32:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 msgr2=0x7f78cc13d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:28.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f78bc00c000 tx=0x7f78bc006df0).stop 2022-07-20T02:32:28.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 shutdown_connections 2022-07-20T02:32:28.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f78ac053a70 0x7f78ac055f30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78cc0b2ac0 0x7f78cc13b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78cc0b0680 0x7f78cc066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 --2- 172.21.15.26:0/4057095193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78cc061010 0x7f78cc13d800 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 >> 172.21.15.26:0/4057095193 conn(0x7f78cc0b60a0 msgr2=0x7f78cc0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:28.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 shutdown_connections 2022-07-20T02:32:28.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:27.970+0000 7f78d3c0b700 1 -- 172.21.15.26:0/4057095193 wait complete. 2022-07-20T02:32:28.005 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:32:28.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:28.036 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:28.061 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:28.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:28.129 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:32:28.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:28.131 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:28.131 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:28.162 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:32:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:32:28.165 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:28.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:28.177 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:28.178 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:32:28.206 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:32:28.207 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:32:28.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:32:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 Processor -- start 2022-07-20T02:32:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 -- start start 2022-07-20T02:32:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bfc0b0f50 0x7f5bfc1473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0609c0 con 0x7f5bfc05f9e0 2022-07-20T02:32:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b13f0 con 0x7f5bfc0b0f50 2022-07-20T02:32:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.326+0000 7f5c014f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b1690 con 0x7f5bfc060550 2022-07-20T02:32:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5bfb7fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bfc0b0f50 0x7f5bfc1473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:33080/0 (socket says 172.21.15.26:33080) 2022-07-20T02:32:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 -- 172.21.15.26:0/2273362420 learned_addr learned my addr 172.21.15.26:0/2273362420 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 -- 172.21.15.26:0/2273362420 >> v2:172.21.15.120:3300/0 conn(0x7f5bfc0b0f50 msgr2=0x7f5bfc1473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 --2- 172.21.15.26:0/2273362420 >> v2:172.21.15.120:3300/0 conn(0x7f5bfc0b0f50 0x7f5bfc1473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 -- 172.21.15.26:0/2273362420 >> v1:172.21.15.120:6789/0 conn(0x7f5bfc05f9e0 legacy=0x7f5bfc0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:28.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 -- 172.21.15.26:0/2273362420 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0b1930 con 0x7f5bfc060550 2022-07-20T02:32:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c00cf1700 1 --2- 172.21.15.26:0/2273362420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5bf40032f0 tx=0x7f5bf4009290).ready entity=mon.2 client_cookie=c819bbc5028973b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5bfaffd700 1 -- 172.21.15.26:0/2273362420 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bf400b8a0 con 0x7f5bfc060550 2022-07-20T02:32:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5bfaffd700 1 -- 172.21.15.26:0/2273362420 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf400ba00 con 0x7f5bfc060550 2022-07-20T02:32:28.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5bfaffd700 1 -- 172.21.15.26:0/2273362420 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bf400bb70 con 0x7f5bfc060550 2022-07-20T02:32:28.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 -- 172.21.15.26:0/2273362420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 msgr2=0x7f5bfc066c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:28.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 --2- 172.21.15.26:0/2273362420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5bf40032f0 tx=0x7f5bf4009290).stop 2022-07-20T02:32:28.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 -- 172.21.15.26:0/2273362420 shutdown_connections 2022-07-20T02:32:28.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 --2- 172.21.15.26:0/2273362420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc066c40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 --2- 172.21.15.26:0/2273362420 >> v2:172.21.15.120:3300/0 conn(0x7f5bfc0b0f50 0x7f5bfc1473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 -- 172.21.15.26:0/2273362420 >> 172.21.15.26:0/2273362420 conn(0x7f5bfc0b3050 msgr2=0x7f5bfc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:28.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 -- 172.21.15.26:0/2273362420 shutdown_connections 2022-07-20T02:32:28.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.330+0000 7f5c014f2700 1 -- 172.21.15.26:0/2273362420 wait complete. 2022-07-20T02:32:28.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 Processor -- start 2022-07-20T02:32:28.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- start start 2022-07-20T02:32:28.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 0x7f5bfc0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1577b0 con 0x7f5bfc05f9e0 2022-07-20T02:32:28.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc063870 con 0x7f5bfc0b0f50 2022-07-20T02:32:28.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc063b10 con 0x7f5bfc060550 2022-07-20T02:32:28.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c00cf1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 0x7f5bfc0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:33082/0 (socket says 172.21.15.26:33082) 2022-07-20T02:32:28.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c00cf1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 0x7f5bfc0ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59494/0 (socket says 172.21.15.26:59494) 2022-07-20T02:32:28.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 -- 172.21.15.26:0/540534392 learned_addr learned my addr 172.21.15.26:0/540534392 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:28.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39800/0 (socket says 172.21.15.26:39800) 2022-07-20T02:32:28.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 -- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 msgr2=0x7f5bfc0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 0x7f5bfc0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 -- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 msgr2=0x7f5bfc0aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 -- 172.21.15.26:0/540534392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf4009530 con 0x7f5bfc060550 2022-07-20T02:32:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfbfff700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bfb7fe700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f5be400c000 tx=0x7f5be4006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:28.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be4011750 con 0x7f5bfc060550 2022-07-20T02:32:28.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be40118b0 con 0x7f5bfc060550 2022-07-20T02:32:28.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc063db0 con 0x7f5bfc060550 2022-07-20T02:32:28.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be4011c10 con 0x7f5bfc060550 2022-07-20T02:32:28.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.334+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc064250 con 0x7f5bfc060550 2022-07-20T02:32:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc063f40 con 0x7f5bfc060550 2022-07-20T02:32:28.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5be4011d70 con 0x7f5bfc060550 2022-07-20T02:32:28.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5bf97fa700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 0x7f5be0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:28.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f5be406c1d0 con 0x7f5bfc060550 2022-07-20T02:32:28.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5bfbfff700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 0x7f5be0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:28.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.338+0000 7f5bfbfff700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 0x7f5be0055e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f5bec00d000 tx=0x7f5bec009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:28.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.346+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5be4043240 con 0x7f5bfc060550 2022-07-20T02:32:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.482+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5bfc0574e0 con 0x7f5bfc060550 2022-07-20T02:32:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5bf97fa700 1 -- 172.21.15.26:0/540534392 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v80) v1 ==== 81+0+29338 (secure 0 0 0) 0x7f5be4046860 con 0x7f5bfc060550 2022-07-20T02:32:28.501 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:28.502 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:28.503 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 80, 2022-07-20T02:32:28.503 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:32:28.503 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:32:28.504 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:32:25.653858+0000", 2022-07-20T02:32:28.504 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:28.505 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:28.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:32:28.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:32:28.517 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:32:28.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:32:28.518 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:32:28.519 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:32:28.520 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:32:28.520 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:28.521 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:32:28.521 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:32:28.522 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:32:28.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:32:28.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:32:28.524 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 14, 2022-07-20T02:32:28.524 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:32:28.525 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:32:28.525 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:32:28.528 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:32:28.529 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:32:28.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.530 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:32:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:32:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:32:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:32:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:32:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:32:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:32:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:32:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:28.545 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:28.546 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:28.546 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:28.547 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:28.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.549 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:32:28.549 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:28.551 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:28.551 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:28.552 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:28.552 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:28.553 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:28.553 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:28.556 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:28.556 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:28.557 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:28.557 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:28.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:28.570 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:28.576 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:28.576 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:28.576 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:28.576 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:28.577 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:28.577 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:28.577 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:28.577 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:28.578 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:28.578 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:28.578 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:28.578 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:28.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:28.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.581 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:28.581 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:28.582 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:28.582 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:28.583 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:28.583 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:28.584 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:28.584 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:28.585 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:28.585 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:28.586 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:28.586 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:28.587 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:32:28.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.588 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:28.588 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:32:28.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.591 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:32:28.591 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:32:28.592 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:32:28.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:28.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:28.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:28.595 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:28.614 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:28.615 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:28.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:28.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:32:28.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:28.619 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:28.620 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:28.621 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:28.623 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:28.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:28.625 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:28.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:28.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:28.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:28.630 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:28.631 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:28.632 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:28.633 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:32:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:32:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:32:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:28.636 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:28.637 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:28.643 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:28.646 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:28.647 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:28.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:32:28.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:32:28.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:32:28.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:32:28.750 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.757 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:28.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:32:28.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:32:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:32:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:32:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:32:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:32:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:32:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:32:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:32:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 64, 2022-07-20T02:32:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:32:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:32:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:32:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:32:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:32:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:32:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:32:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:32:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:32:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:32:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:32:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:32:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:32:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:32:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:32:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:32:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:32:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:32:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:32:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:32:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:32:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:32:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:32:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:32:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:32:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:32:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:32:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:32:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:32:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:32:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.900 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:28.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:32:28.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:32:28.911 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.912 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:28.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:32:28.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:32:28.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:28.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:32:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:32:28.925 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:32:28.925 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:32:28.926 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:28.928 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:28.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:28.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:28.931 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:32:28.931 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:28.931 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:28.932 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:28.933 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:28.933 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:28.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:28.934 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:28.935 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:28.935 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:28.935 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:28.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:28.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:32:28.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:32:28.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.947 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:28.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:32:28.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:32:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:32:28.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.959 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:28.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:32:28.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:28.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:28.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.963 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:28.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:28.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:28.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:32:28.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:28.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:28.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:28.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:32:29.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:29.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:32:29.020 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:32:29.021 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:32:29.021 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:32:29.022 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:29.022 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:29.023 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:29.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.024 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:29.025 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:32:29.025 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:29.026 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:29.026 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:29.027 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:29.027 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:29.028 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:29.028 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:29.035 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:29.035 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:29.036 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:29.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:29.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:32:29.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:32:29.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.083 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:29.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:32:29.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:32:29.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.112 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:29.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:32:29.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:32:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:32:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:32:29.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:32:29.125 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:32:29.126 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:32:29.132 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:32:29.133 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:29.136 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:29.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.144 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:29.145 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:32:29.145 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:29.145 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:29.145 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:29.146 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:29.146 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:29.147 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:29.147 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:29.147 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 74, 2022-07-20T02:32:29.147 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:29.148 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:29.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:29.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:32:29.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:32:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.151 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:32:29.153 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:32:29.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.157 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:29.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:32:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:32:29.166 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.183 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:29.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:32:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:32:29.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:29.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.196 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:32:29.197 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:32:29.198 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:32:29.198 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:32:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:29.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.201 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:29.201 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:32:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:29.203 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:29.203 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:29.203 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:29.203 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:29.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:29.204 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:29.204 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:32:29.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:32:29.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.208 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:29.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:32:29.223 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:32:29.235 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.238 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:29.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:32:29.241 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:32:29.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.248 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:29.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:29.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:32:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:32:29.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:32:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:32:29.272 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:32:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:32:29.273 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:29.282 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:29.293 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:29.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:29.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:32:29.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:29.316 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.317 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:32:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:32:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:32:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:32:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:32:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:32:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:32:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:32:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:32:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:32:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:32:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:32:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:32:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/489399789": "2022-07-21T02:32:22.021092+0000", 2022-07-20T02:32:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/489399789": "2022-07-21T02:32:22.021092+0000", 2022-07-20T02:32:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.159:0/682748160": "2022-07-20T03:32:10.385916+0000" 2022-07-20T02:32:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:32:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:32:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:32:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:32:29.370 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:32:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:32:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:29.373 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:32:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:32:29.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:32:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:29.375 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:32:29.376 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:29.377 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:32:29.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:29.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:29.379 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:32:29.379 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:32:29.379 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:32:29.380 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:32:29.380 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:32:29.381 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:29.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 msgr2=0x7f5be0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 0x7f5be0055e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f5bec00d000 tx=0x7f5bec009040).stop 2022-07-20T02:32:29.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 msgr2=0x7f5bfc0ab180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f5be400c000 tx=0x7f5be4006df0).stop 2022-07-20T02:32:29.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 shutdown_connections 2022-07-20T02:32:29.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5be0053950 0x7f5be0055e10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bfc0b0f50 0x7f5bfc0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bfc060550 0x7f5bfc0ab180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 --2- 172.21.15.26:0/540534392 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5bfc05f9e0 0x7f5bfc0aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 >> 172.21.15.26:0/540534392 conn(0x7f5bfc0b3050 msgr2=0x7f5bfc143440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:29.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 shutdown_connections 2022-07-20T02:32:29.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:28.486+0000 7f5c014f2700 1 -- 172.21.15.26:0/540534392 wait complete. 2022-07-20T02:32:29.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:29.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 Processor -- start 2022-07-20T02:32:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 -- start start 2022-07-20T02:32:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1170061a20 0x7f11701432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f117012c750 0x7f11700b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1170061e00 con 0x7f11701468c0 2022-07-20T02:32:29.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1170147040 con 0x7f117012c750 2022-07-20T02:32:29.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1175cff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11701472e0 con 0x7f1170061a20 2022-07-20T02:32:29.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f117012c750 0x7f11700b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1167fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f11701468c0 0x7f117012c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39086/0 (socket says 172.21.15.26:39086) 2022-07-20T02:32:29.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f1167fff700 1 -- 172.21.15.26:0/3540004862 learned_addr learned my addr 172.21.15.26:0/3540004862 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:29.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116f7fe700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1170061a20 0x7f11701432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116effd700 1 -- 172.21.15.26:0/3540004862 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492821370 0 0) 0x7f1170061e00 con 0x7f11701468c0 2022-07-20T02:32:29.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116effd700 1 -- 172.21.15.26:0/3540004862 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003260 con 0x7f11701468c0 2022-07-20T02:32:29.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 -- 172.21.15.26:0/3540004862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1170061a20 msgr2=0x7f11701432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:29.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1170061a20 0x7f11701432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 -- 172.21.15.26:0/3540004862 >> v1:172.21.15.26:6789/0 conn(0x7f11701468c0 legacy=0x7f117012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 -- 172.21.15.26:0/3540004862 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170147580 con 0x7f117012c750 2022-07-20T02:32:29.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116ffff700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f117012c750 0x7f11700b2720 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f11680032f0 tx=0x7f11680091d0).ready entity=mon.0 client_cookie=e198f8cffd8e15ab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116effd700 1 -- 172.21.15.26:0/3540004862 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116800b600 con 0x7f117012c750 2022-07-20T02:32:29.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116effd700 1 -- 172.21.15.26:0/3540004862 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116800b760 con 0x7f117012c750 2022-07-20T02:32:29.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.502+0000 7f116effd700 1 -- 172.21.15.26:0/3540004862 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116800b8d0 con 0x7f117012c750 2022-07-20T02:32:29.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- 172.21.15.26:0/3540004862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f117012c750 msgr2=0x7f11700b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f117012c750 0x7f11700b2720 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f11680032f0 tx=0x7f11680091d0).stop 2022-07-20T02:32:29.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- 172.21.15.26:0/3540004862 shutdown_connections 2022-07-20T02:32:29.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f117012c750 0x7f11700b2720 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- 172.21.15.26:0/3540004862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1170061a20 0x7f11701432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- 172.21.15.26:0/3540004862 >> 172.21.15.26:0/3540004862 conn(0x7f117005faf0 msgr2=0x7f1170066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:29.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- 172.21.15.26:0/3540004862 shutdown_connections 2022-07-20T02:32:29.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- 172.21.15.26:0/3540004862 wait complete. 2022-07-20T02:32:29.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 Processor -- start 2022-07-20T02:32:29.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- start start 2022-07-20T02:32:29.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117012c750 0x7f11701527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 0x7f1170152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170143580 con 0x7f1170061a20 2022-07-20T02:32:29.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701531d0 con 0x7f117012c750 2022-07-20T02:32:29.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1175cff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700ba4a0 con 0x7f11701468c0 2022-07-20T02:32:29.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39812/0 (socket says 172.21.15.26:39812) 2022-07-20T02:32:29.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 -- 172.21.15.26:0/1477152783 learned_addr learned my addr 172.21.15.26:0/1477152783 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:29.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116ffff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 0x7f1170152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f1167fff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117012c750 0x7f11701527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 -- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 msgr2=0x7f1170152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 0x7f1170152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 -- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117012c750 msgr2=0x7f11701527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117012c750 0x7f11701527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116f7fe700 1 -- 172.21.15.26:0/1477152783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168009470 con 0x7f1170061a20 2022-07-20T02:32:29.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.506+0000 7f116ffff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 0x7f1170152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:29.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116f7fe700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f1170129830 tx=0x7f1160006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1160010780 con 0x7f1170061a20 2022-07-20T02:32:29.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11600108e0 con 0x7f1170061a20 2022-07-20T02:32:29.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1170152350 con 0x7f1170061a20 2022-07-20T02:32:29.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1160010c40 con 0x7f1170061a20 2022-07-20T02:32:29.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11701557d0 con 0x7f1170061a20 2022-07-20T02:32:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f116001c460 con 0x7f1170061a20 2022-07-20T02:32:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11700ada40 con 0x7f1170061a20 2022-07-20T02:32:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 0x7f1154055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f1167fff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 0x7f1154055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.510+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f116006c0c0 con 0x7f1170061a20 2022-07-20T02:32:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.514+0000 7f1167fff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 0x7f1154055ff0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f115800d000 tx=0x7f1158009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.518+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11600430d0 con 0x7f1170061a20 2022-07-20T02:32:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.682+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f117012cbc0 con 0x7f1170061a20 2022-07-20T02:32:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.682+0000 7f116d7fa700 1 -- 172.21.15.26:0/1477152783 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f11600466f0 con 0x7f1170061a20 2022-07-20T02:32:29.692 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:29.693 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35},{"gid":6600,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/3415620727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3415620727},{"type":"v1","addr":"172.21.15.26:6837","nonce":3415620727}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":40}],"filesystems":[]} 2022-07-20T02:32:29.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 msgr2=0x7f1154055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 0x7f1154055ff0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f115800d000 tx=0x7f1158009040).stop 2022-07-20T02:32:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 msgr2=0x7f11700b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f1170129830 tx=0x7f1160006df0).stop 2022-07-20T02:32:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 shutdown_connections 2022-07-20T02:32:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1154053b30 0x7f1154055ff0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11701468c0 0x7f1170152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f117012c750 0x7f11701527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 --2- 172.21.15.26:0/1477152783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1170061a20 0x7f11700b5db0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 >> 172.21.15.26:0/1477152783 conn(0x7f117005faf0 msgr2=0x7f11700b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:29.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 shutdown_connections 2022-07-20T02:32:29.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.686+0000 7f1175cff700 1 -- 172.21.15.26:0/1477152783 wait complete. 2022-07-20T02:32:29.713 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 40 2022-07-20T02:32:29.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.818+0000 7f5625b89700 1 Processor -- start 2022-07-20T02:32:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.818+0000 7f5625b89700 1 -- start start 2022-07-20T02:32:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.818+0000 7f5625b89700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f56201465b0 0x7f5620066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200b1ae0 con 0x7f56200b24c0 2022-07-20T02:32:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200b1d80 con 0x7f56200672a0 2022-07-20T02:32:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56200b2e20 con 0x7f56201465b0 2022-07-20T02:32:29.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f56200b24c0 0x7f56200604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39102/0 (socket says 172.21.15.26:39102) 2022-07-20T02:32:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561ffff700 1 -- 172.21.15.26:0/1002447332 learned_addr learned my addr 172.21.15.26:0/1002447332 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5624b87700 1 --2- 172.21.15.26:0/1002447332 >> v2:172.21.15.120:3300/0 conn(0x7f56201465b0 0x7f5620066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181004346 0 0) 0x7f56200b1ae0 con 0x7f56200b24c0 2022-07-20T02:32:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c003260 con 0x7f56200b24c0 2022-07-20T02:32:29.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5624b87700 1 -- 172.21.15.26:0/1002447332 >> v1:172.21.15.120:6789/0 conn(0x7f56200672a0 legacy=0x7f5620146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312247428 0 0) 0x7f560c003260 con 0x7f56200b24c0 2022-07-20T02:32:29.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5624b87700 1 -- 172.21.15.26:0/1002447332 >> v1:172.21.15.26:6789/0 conn(0x7f56200b24c0 legacy=0x7f56200604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5624b87700 1 -- 172.21.15.26:0/1002447332 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200b2020 con 0x7f56201465b0 2022-07-20T02:32:29.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5624b87700 1 --2- 172.21.15.26:0/1002447332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56201465b0 0x7f5620066db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5614001800 tx=0x7f5610006df0).ready entity=mon.1 client_cookie=641a7272f8379d2b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5614001660 con 0x7f56201465b0 2022-07-20T02:32:29.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5610000e90 con 0x7f56201465b0 2022-07-20T02:32:29.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f561f7fe700 1 -- 172.21.15.26:0/1002447332 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561000d4f0 con 0x7f56201465b0 2022-07-20T02:32:29.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- 172.21.15.26:0/1002447332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56201465b0 msgr2=0x7f5620066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 --2- 172.21.15.26:0/1002447332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56201465b0 0x7f5620066db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5614001800 tx=0x7f5610006df0).stop 2022-07-20T02:32:29.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- 172.21.15.26:0/1002447332 shutdown_connections 2022-07-20T02:32:29.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 --2- 172.21.15.26:0/1002447332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56201465b0 0x7f5620066db0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- 172.21.15.26:0/1002447332 >> 172.21.15.26:0/1002447332 conn(0x7f56200b3050 msgr2=0x7f56200b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- 172.21.15.26:0/1002447332 shutdown_connections 2022-07-20T02:32:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.822+0000 7f5625b89700 1 -- 172.21.15.26:0/1002447332 wait complete. 2022-07-20T02:32:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 Processor -- start 2022-07-20T02:32:29.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- start start 2022-07-20T02:32:29.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56200672a0 0x7f5620143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5620066540 0x7f56201526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200ad530 con 0x7f56201465b0 2022-07-20T02:32:29.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200aa340 con 0x7f5620066540 2022-07-20T02:32:29.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5620152fe0 con 0x7f56200672a0 2022-07-20T02:32:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39828/0 (socket says 172.21.15.26:39828) 2022-07-20T02:32:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 -- 172.21.15.26:0/2078233534 learned_addr learned my addr 172.21.15.26:0/2078233534 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625388700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5620066540 0x7f56201526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5624b87700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56200672a0 0x7f5620143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 -- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56200672a0 msgr2=0x7f5620143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56200672a0 0x7f5620143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 -- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5620066540 msgr2=0x7f56201526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5620066540 0x7f56201526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 -- 172.21.15.26:0/2078233534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561000b180 con 0x7f56201465b0 2022-07-20T02:32:29.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561ffff700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f56140018c0 tx=0x7f56140012c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5614007170 con 0x7f56201465b0 2022-07-20T02:32:29.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56200aa570 con 0x7f56201465b0 2022-07-20T02:32:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56140072d0 con 0x7f56201465b0 2022-07-20T02:32:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5620143670 con 0x7f56201465b0 2022-07-20T02:32:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5614004710 con 0x7f56201465b0 2022-07-20T02:32:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.826+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5620060bf0 con 0x7f56201465b0 2022-07-20T02:32:29.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.830+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f561401b070 con 0x7f56201465b0 2022-07-20T02:32:29.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.830+0000 7f561dffb700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 0x7f5600055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.830+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f561406ba70 con 0x7f56201465b0 2022-07-20T02:32:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.830+0000 7f5624b87700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 0x7f5600055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.830+0000 7f5624b87700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 0x7f5600055e10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f561001a000 tx=0x7f5610014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:29.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.834+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5614042a80 con 0x7f56201465b0 2022-07-20T02:32:30.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.998+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56200574e0 con 0x7f56201465b0 2022-07-20T02:32:30.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:29.998+0000 7f561dffb700 1 -- 172.21.15.26:0/2078233534 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f5614022020 con 0x7f56201465b0 2022-07-20T02:32:30.004 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:30.004 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/220988495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":220988495},{"type":"v1","addr":"172.21.15.120:6833","nonce":220988495}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1140616761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1140616761},{"type":"v1","addr":"172.21.15.26:6835","nonce":1140616761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35},{"gid":6600,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/3415620727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3415620727},{"type":"v1","addr":"172.21.15.26:6837","nonce":3415620727}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":40}],"filesystems":[]} 2022-07-20T02:32:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 msgr2=0x7f5600055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 0x7f5600055e10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f561001a000 tx=0x7f5610014000).stop 2022-07-20T02:32:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 msgr2=0x7f56201521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f56140018c0 tx=0x7f56140012c0).stop 2022-07-20T02:32:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 shutdown_connections 2022-07-20T02:32:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56201465b0 0x7f56201521e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5600053950 0x7f5600055e10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5620066540 0x7f56201526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 --2- 172.21.15.26:0/2078233534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56200672a0 0x7f5620143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 >> 172.21.15.26:0/2078233534 conn(0x7f56200b3050 msgr2=0x7f562013ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 shutdown_connections 2022-07-20T02:32:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.002+0000 7f5625b89700 1 -- 172.21.15.26:0/2078233534 wait complete. 2022-07-20T02:32:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 40 2022-07-20T02:32:30.062 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:32:30.063 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:32:30.064 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:32:30.064 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:30.076 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:32:30.076 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:32:30.077 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:32:30.078 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:30.078 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:32:30.079 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:32:30.079 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:32:30.080 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:30.083 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:32:30.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:32:30.112 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:32:30.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:32:30.114 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:32:30.115 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:32:30.156 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:32:30.170 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:30.173 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:32:30.173 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:30.176 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:32:30.189 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:30.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:32:30.234 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:32:30.242 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:32:30.257 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:32:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 Processor -- start 2022-07-20T02:32:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- start start 2022-07-20T02:32:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f91800a95a0 0x7f9180060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9180067dc0 con 0x7f91800b3260 2022-07-20T02:32:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91800b3640 con 0x7f9180095b10 2022-07-20T02:32:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91800615c0 con 0x7f91800a95a0 2022-07-20T02:32:30.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91848cf700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f91800b3260 0x7f91800676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39138/0 (socket says 172.21.15.26:39138) 2022-07-20T02:32:30.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91848cf700 1 -- 172.21.15.26:0/2996495770 learned_addr learned my addr 172.21.15.26:0/2996495770 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:30.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917ffff700 1 --2- 172.21.15.26:0/2996495770 >> v2:172.21.15.26:3300/0 conn(0x7f91800a95a0 0x7f9180060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:30.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917f7fe700 1 -- 172.21.15.26:0/2996495770 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769235332 0 0) 0x7f9180067dc0 con 0x7f91800b3260 2022-07-20T02:32:30.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917f7fe700 1 -- 172.21.15.26:0/2996495770 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c003260 con 0x7f91800b3260 2022-07-20T02:32:30.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917ffff700 1 -- 172.21.15.26:0/2996495770 >> v1:172.21.15.120:6789/0 conn(0x7f9180095b10 legacy=0x7f9180095fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917ffff700 1 -- 172.21.15.26:0/2996495770 >> v1:172.21.15.26:6789/0 conn(0x7f91800b3260 legacy=0x7f91800676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917ffff700 1 -- 172.21.15.26:0/2996495770 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9180061860 con 0x7f91800a95a0 2022-07-20T02:32:30.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917ffff700 1 --2- 172.21.15.26:0/2996495770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180060950 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f9174002f70 tx=0x7f9174006df0).ready entity=mon.0 client_cookie=11b6f2b0bd130dfd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:30.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917f7fe700 1 -- 172.21.15.26:0/2996495770 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f917400d610 con 0x7f91800a95a0 2022-07-20T02:32:30.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917f7fe700 1 -- 172.21.15.26:0/2996495770 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9174000e90 con 0x7f91800a95a0 2022-07-20T02:32:30.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f917f7fe700 1 -- 172.21.15.26:0/2996495770 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f917400d960 con 0x7f91800a95a0 2022-07-20T02:32:30.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- 172.21.15.26:0/2996495770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 msgr2=0x7f9180060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 --2- 172.21.15.26:0/2996495770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180060950 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f9174002f70 tx=0x7f9174006df0).stop 2022-07-20T02:32:30.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- 172.21.15.26:0/2996495770 shutdown_connections 2022-07-20T02:32:30.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 --2- 172.21.15.26:0/2996495770 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180060950 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- 172.21.15.26:0/2996495770 >> 172.21.15.26:0/2996495770 conn(0x7f91800b67f0 msgr2=0x7f91800b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:30.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- 172.21.15.26:0/2996495770 shutdown_connections 2022-07-20T02:32:30.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.306+0000 7f91858d1700 1 -- 172.21.15.26:0/2996495770 wait complete. 2022-07-20T02:32:30.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 Processor -- start 2022-07-20T02:32:30.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- start start 2022-07-20T02:32:30.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 0x7f91800aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:30.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:30.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91800b3260 0x7f9180063440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:30.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91800af240 con 0x7f91800a95a0 2022-07-20T02:32:30.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91800ab8c0 con 0x7f91800b3260 2022-07-20T02:32:30.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91800abb60 con 0x7f9180095b10 2022-07-20T02:32:30.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:30.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 0x7f91800aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:30.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39868/0 (socket says 172.21.15.26:39868) 2022-07-20T02:32:30.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917ffff700 1 -- 172.21.15.26:0/3373224697 learned_addr learned my addr 172.21.15.26:0/3373224697 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91850d0700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91800b3260 0x7f9180063440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 -- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91800b3260 msgr2=0x7f9180063440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91800b3260 0x7f9180063440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 -- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 msgr2=0x7f9180062f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:30.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:30.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 -- 172.21.15.26:0/3373224697 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917400b180 con 0x7f9180095b10 2022-07-20T02:32:30.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917ffff700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91848cf700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 0x7f91800aeb30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9170001490 tx=0x7f9170007e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:30.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9170005ba0 con 0x7f9180095b10 2022-07-20T02:32:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9170005d00 con 0x7f9180095b10 2022-07-20T02:32:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91800afd30 con 0x7f9180095b10 2022-07-20T02:32:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91700072c0 con 0x7f9180095b10 2022-07-20T02:32:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.310+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9180065850 con 0x7f9180095b10 2022-07-20T02:32:30.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9170010070 con 0x7f9180095b10 2022-07-20T02:32:30.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91800b01e0 con 0x7f9180095b10 2022-07-20T02:32:30.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f917dffb700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 0x7f9160055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:30.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f917006b7d0 con 0x7f9180095b10 2022-07-20T02:32:30.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f917ffff700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 0x7f9160055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:30.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.314+0000 7f917ffff700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 0x7f9160055f60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f917401a000 tx=0x7f9174014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:30.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.322+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91700426b0 con 0x7f9180095b10 2022-07-20T02:32:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:30.462+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f91800af410 con 0x7f9180095b10 2022-07-20T02:32:31.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.270+0000 7f917dffb700 1 -- 172.21.15.26:0/3373224697 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v83) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9170045cd0 con 0x7f9180095b10 2022-07-20T02:32:31.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 msgr2=0x7f9160055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 0x7f9160055f60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f917401a000 tx=0x7f9174014000).stop 2022-07-20T02:32:31.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 msgr2=0x7f91800aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 0x7f91800aeb30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9170001490 tx=0x7f9170007e40).stop 2022-07-20T02:32:31.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 shutdown_connections 2022-07-20T02:32:31.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f91800a95a0 0x7f9180062f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9160053aa0 0x7f9160055f60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91800b3260 0x7f9180063440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 --2- 172.21.15.26:0/3373224697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9180095b10 0x7f91800aeb30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 >> 172.21.15.26:0/3373224697 conn(0x7f91800b67f0 msgr2=0x7f91800b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:31.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 shutdown_connections 2022-07-20T02:32:31.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.274+0000 7f91858d1700 1 -- 172.21.15.26:0/3373224697 wait complete. 2022-07-20T02:32:31.280 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:32:31.294 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:32:31.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:32:31.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 Processor -- start 2022-07-20T02:32:31.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 -- start start 2022-07-20T02:32:31.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa668060f00 0x7fa66813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6681474e0 con 0x7fa668147000 2022-07-20T02:32:31.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6680b4e40 con 0x7fa66805cfd0 2022-07-20T02:32:31.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66de83700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66805d3b0 con 0x7fa668060f00 2022-07-20T02:32:31.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66ce81700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66ce81700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39876/0 (socket says 172.21.15.26:39876) 2022-07-20T02:32:31.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66d682700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa668060f00 0x7fa66813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66ce81700 1 -- 172.21.15.26:0/166006561 learned_addr learned my addr 172.21.15.26:0/166006561 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:31.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66d682700 1 -- 172.21.15.26:0/166006561 >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 msgr2=0x7fa668143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66d682700 1 --2- 172.21.15.26:0/166006561 >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66d682700 1 -- 172.21.15.26:0/166006561 >> v1:172.21.15.120:6789/0 conn(0x7fa668147000 legacy=0x7fa6680b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66d682700 1 -- 172.21.15.26:0/166006561 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6681431e0 con 0x7fa668060f00 2022-07-20T02:32:31.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.478+0000 7fa66ce81700 1 --2- 172.21.15.26:0/166006561 >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:31.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66d682700 1 --2- 172.21.15.26:0/166006561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668060f00 0x7fa66813c9b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa6600032f0 tx=0x7fa6600091d0).ready entity=mon.1 client_cookie=11b394d683131d50 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:31.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa6677fe700 1 -- 172.21.15.26:0/166006561 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa66000b680 con 0x7fa668060f00 2022-07-20T02:32:31.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa6677fe700 1 -- 172.21.15.26:0/166006561 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66000b7e0 con 0x7fa668060f00 2022-07-20T02:32:31.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa6677fe700 1 -- 172.21.15.26:0/166006561 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa66000b950 con 0x7fa668060f00 2022-07-20T02:32:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- 172.21.15.26:0/166006561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668060f00 msgr2=0x7fa66813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 --2- 172.21.15.26:0/166006561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668060f00 0x7fa66813c9b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa6600032f0 tx=0x7fa6600091d0).stop 2022-07-20T02:32:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- 172.21.15.26:0/166006561 shutdown_connections 2022-07-20T02:32:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 --2- 172.21.15.26:0/166006561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668060f00 0x7fa66813c9b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 --2- 172.21.15.26:0/166006561 >> v2:172.21.15.26:3300/0 conn(0x7fa66805cfd0 0x7fa668143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- 172.21.15.26:0/166006561 >> 172.21.15.26:0/166006561 conn(0x7fa6680620c0 msgr2=0x7fa66805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- 172.21.15.26:0/166006561 shutdown_connections 2022-07-20T02:32:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- 172.21.15.26:0/166006561 wait complete. 2022-07-20T02:32:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 Processor -- start 2022-07-20T02:32:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.482+0000 7fa66de83700 1 -- start start 2022-07-20T02:32:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668060f00 0x7fa668137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668147000 0x7fa668155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668137bc0 con 0x7fa668142d40 2022-07-20T02:32:31.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668152400 con 0x7fa668147000 2022-07-20T02:32:31.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa668141d30 con 0x7fa668060f00 2022-07-20T02:32:31.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39884/0 (socket says 172.21.15.26:39884) 2022-07-20T02:32:31.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 -- 172.21.15.26:0/451930503 learned_addr learned my addr 172.21.15.26:0/451930503 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:31.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66ce81700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668060f00 0x7fa668137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa667fff700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668147000 0x7fa668155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 -- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668060f00 msgr2=0x7fa668137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668060f00 0x7fa668137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 -- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668147000 msgr2=0x7fa668155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668147000 0x7fa668155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 -- 172.21.15.26:0/451930503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65000b1e0 con 0x7fa668142d40 2022-07-20T02:32:31.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66d682700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fa6600032f0 tx=0x7fa660016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:31.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa660011e20 con 0x7fa668142d40 2022-07-20T02:32:31.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660021440 con 0x7fa668142d40 2022-07-20T02:32:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa660009470 con 0x7fa668142d40 2022-07-20T02:32:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6600217a0 con 0x7fa668142d40 2022-07-20T02:32:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6680ae260 con 0x7fa668142d40 2022-07-20T02:32:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668142520 con 0x7fa668142d40 2022-07-20T02:32:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa660027020 con 0x7fa668142d40 2022-07-20T02:32:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.486+0000 7fa665ffb700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 0x7fa64c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.490+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa66006d2d0 con 0x7fa668142d40 2022-07-20T02:32:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.490+0000 7fa66ce81700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 0x7fa64c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.490+0000 7fa66ce81700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 0x7fa64c055fc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fa65000b000 tx=0x7fa650006eb0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.494+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa660044330 con 0x7fa668142d40 2022-07-20T02:32:31.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.662+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa6681568d0 con 0x7fa668142d40 2022-07-20T02:32:31.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.662+0000 7fa665ffb700 1 -- 172.21.15.26:0/451930503 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fa660015020 con 0x7fa668142d40 2022-07-20T02:32:31.683 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:31.686 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:31.687 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:32:31.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.687 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:32:31.688 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:32:31.688 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.689 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:31.689 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.690 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.690 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.692 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:32:31.692 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:32:31.693 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.693 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:31.695 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.697 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.697 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.699 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:32:31.707 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:32:31.707 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.708 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:31.708 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.708 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.709 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:32:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:32:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:32:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:32:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:32:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:32:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.743 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.748 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:32:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:32:31.749 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.762 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.763 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:32:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:32:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:32:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:32:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:32:31.822 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:32:31.823 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:32:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:32:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:32:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:32:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:32:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:32:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:32:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:32:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:32:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:32:31.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.891 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:32:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:32:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:31.893 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:32:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:32:31.914 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:32:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:32:31.916 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:32:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:32:31.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:31.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:31.924 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 msgr2=0x7fa64c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 0x7fa64c055fc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fa65000b000 tx=0x7fa650006eb0).stop 2022-07-20T02:32:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 msgr2=0x7fa668156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fa6600032f0 tx=0x7fa660016000).stop 2022-07-20T02:32:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 shutdown_connections 2022-07-20T02:32:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa668142d40 0x7fa668156240 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053b00 0x7fa64c055fc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa668147000 0x7fa668155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 --2- 172.21.15.26:0/451930503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668060f00 0x7fa668137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:31.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 >> 172.21.15.26:0/451930503 conn(0x7fa66805cfd0 msgr2=0x7fa668139c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:31.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 shutdown_connections 2022-07-20T02:32:31.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:31.666+0000 7fa66de83700 1 -- 172.21.15.26:0/451930503 wait complete. 2022-07-20T02:32:31.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 Processor -- start 2022-07-20T02:32:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 -- start start 2022-07-20T02:32:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 0x7fe938066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9380b1bc0 con 0x7fe9380b2920 2022-07-20T02:32:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe938147700 con 0x7fe938061050 2022-07-20T02:32:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93814b040 con 0x7fe9381484a0 2022-07-20T02:32:32.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9377fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe9380b2920 0x7fe938147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39170/0 (socket says 172.21.15.26:39170) 2022-07-20T02:32:32.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9377fe700 1 -- 172.21.15.26:0/1026358620 learned_addr learned my addr 172.21.15.26:0/1026358620 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:32.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe936ffd700 1 --2- 172.21.15.26:0/1026358620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 0x7fe938066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394412354 0 0) 0x7fe9380b1bc0 con 0x7fe9380b2920 2022-07-20T02:32:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924003260 con 0x7fe9380b2920 2022-07-20T02:32:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436128312 0 0) 0x7fe938147700 con 0x7fe938061050 2022-07-20T02:32:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9380b1bc0 con 0x7fe938061050 2022-07-20T02:32:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe936ffd700 1 -- 172.21.15.26:0/1026358620 >> v1:172.21.15.120:6789/0 conn(0x7fe938061050 legacy=0x7fe938066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe936ffd700 1 -- 172.21.15.26:0/1026358620 >> v1:172.21.15.26:6789/0 conn(0x7fe9380b2920 legacy=0x7fe938147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe936ffd700 1 -- 172.21.15.26:0/1026358620 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe938147930 con 0x7fe9381484a0 2022-07-20T02:32:32.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe936ffd700 1 --2- 172.21.15.26:0/1026358620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 0x7fe938066a80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe928002f70 tx=0x7fe928006df0).ready entity=mon.2 client_cookie=77f82155a6ca682a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe92800d5c0 con 0x7fe9381484a0 2022-07-20T02:32:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe928000e90 con 0x7fe9381484a0 2022-07-20T02:32:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe9367fc700 1 -- 172.21.15.26:0/1026358620 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe92800d910 con 0x7fe9381484a0 2022-07-20T02:32:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 -- 172.21.15.26:0/1026358620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 msgr2=0x7fe938066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.069+0000 7fe93db6c700 1 --2- 172.21.15.26:0/1026358620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 0x7fe938066a80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe928002f70 tx=0x7fe928006df0).stop 2022-07-20T02:32:32.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- 172.21.15.26:0/1026358620 shutdown_connections 2022-07-20T02:32:32.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 --2- 172.21.15.26:0/1026358620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9381484a0 0x7fe938066a80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- 172.21.15.26:0/1026358620 >> 172.21.15.26:0/1026358620 conn(0x7fe9380b60a0 msgr2=0x7fe9380b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- 172.21.15.26:0/1026358620 shutdown_connections 2022-07-20T02:32:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- 172.21.15.26:0/1026358620 wait complete. 2022-07-20T02:32:32.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 Processor -- start 2022-07-20T02:32:32.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- start start 2022-07-20T02:32:32.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe938061050 0x7fe938148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9381484a0 0x7fe9380ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380adab0 con 0x7fe9380b2920 2022-07-20T02:32:32.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380a9700 con 0x7fe9381484a0 2022-07-20T02:32:32.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe93db6c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380a9930 con 0x7fe938061050 2022-07-20T02:32:32.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe936ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe936ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39896/0 (socket says 172.21.15.26:39896) 2022-07-20T02:32:32.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe936ffd700 1 -- 172.21.15.26:0/3234178888 learned_addr learned my addr 172.21.15.26:0/3234178888 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:32.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe9377fe700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe938061050 0x7fe938148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.073+0000 7fe937fff700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9381484a0 0x7fe9380ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 -- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe938061050 msgr2=0x7fe938148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe938061050 0x7fe938148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 -- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9381484a0 msgr2=0x7fe9380ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9381484a0 0x7fe9380ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 -- 172.21.15.26:0/3234178888 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92800b180 con 0x7fe9380b2920 2022-07-20T02:32:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe936ffd700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fe928002f70 tx=0x7fe928014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe928013de0 con 0x7fe9380b2920 2022-07-20T02:32:32.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92801e400 con 0x7fe9380b2920 2022-07-20T02:32:32.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9380aa900 con 0x7fe9380b2920 2022-07-20T02:32:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe92801e760 con 0x7fe9380b2920 2022-07-20T02:32:32.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe938066c70 con 0x7fe9380b2920 2022-07-20T02:32:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93814ada0 con 0x7fe9380b2920 2022-07-20T02:32:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe92801c070 con 0x7fe9380b2920 2022-07-20T02:32:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 0x7fe918055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe92806c730 con 0x7fe9380b2920 2022-07-20T02:32:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.077+0000 7fe9377fe700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 0x7fe918055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.081+0000 7fe9377fe700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 0x7fe918055fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe92c00d000 tx=0x7fe92c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.089+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe928043790 con 0x7fe9380b2920 2022-07-20T02:32:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.217+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9380574e0 con 0x7fe9380b2920 2022-07-20T02:32:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe934ff9700 1 -- 172.21.15.26:0/3234178888 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13463 (secure 0 0 0) 0x7fe928023070 con 0x7fe9380b2920 2022-07-20T02:32:32.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:32.237 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":83,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:31.264553+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:32.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 msgr2=0x7fe918055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 0x7fe918055fc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe92c00d000 tx=0x7fe92c009040).stop 2022-07-20T02:32:32.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 msgr2=0x7fe9380ab240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fe928002f70 tx=0x7fe928014000).stop 2022-07-20T02:32:32.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 shutdown_connections 2022-07-20T02:32:32.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9380b2920 0x7fe9380ab240 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe918053b00 0x7fe918055fc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9381484a0 0x7fe9380ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 --2- 172.21.15.26:0/3234178888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe938061050 0x7fe938148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 >> 172.21.15.26:0/3234178888 conn(0x7fe9380b60a0 msgr2=0x7fe9381498f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 shutdown_connections 2022-07-20T02:32:32.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.221+0000 7fe93db6c700 1 -- 172.21.15.26:0/3234178888 wait complete. 2022-07-20T02:32:32.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:32:32.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:32:32.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.393+0000 7f003ae65700 1 Processor -- start 2022-07-20T02:32:32.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.393+0000 7f003ae65700 1 -- start start 2022-07-20T02:32:32.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003ae65700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f00340b3260 0x7f0034060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003ae65700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0034095b10 0x7f0034095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003ae65700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0034067dc0 con 0x7f00340a95a0 2022-07-20T02:32:32.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003ae65700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340b3640 con 0x7f0034095b10 2022-07-20T02:32:32.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003ae65700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340615c0 con 0x7f00340b3260 2022-07-20T02:32:32.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0034095b10 0x7f0034095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0034095b10 0x7f0034095f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39906/0 (socket says 172.21.15.26:39906) 2022-07-20T02:32:32.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 -- 172.21.15.26:0/4248046951 learned_addr learned my addr 172.21.15.26:0/4248046951 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:32.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f0039e63700 1 --2- 172.21.15.26:0/4248046951 >> v2:172.21.15.120:3300/0 conn(0x7f00340b3260 0x7f0034060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 -- 172.21.15.26:0/4248046951 >> v2:172.21.15.120:3300/0 conn(0x7f00340b3260 msgr2=0x7f0034060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:32.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 --2- 172.21.15.26:0/4248046951 >> v2:172.21.15.120:3300/0 conn(0x7f00340b3260 0x7f0034060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 -- 172.21.15.26:0/4248046951 >> v1:172.21.15.120:6789/0 conn(0x7f00340a95a0 legacy=0x7f00340676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 -- 172.21.15.26:0/4248046951 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034061860 con 0x7f0034095b10 2022-07-20T02:32:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f003a664700 1 --2- 172.21.15.26:0/4248046951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0034095b10 0x7f0034095f50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c009290).ready entity=mon.0 client_cookie=c49dfbbaad7e8d94 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f0038e61700 1 -- 172.21.15.26:0/4248046951 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c00b870 con 0x7f0034095b10 2022-07-20T02:32:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f0038e61700 1 -- 172.21.15.26:0/4248046951 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c00b9d0 con 0x7f0034095b10 2022-07-20T02:32:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.397+0000 7f0038e61700 1 -- 172.21.15.26:0/4248046951 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c00bb40 con 0x7f0034095b10 2022-07-20T02:32:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- 172.21.15.26:0/4248046951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0034095b10 msgr2=0x7f0034095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- 172.21.15.26:0/4248046951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0034095b10 0x7f0034095f50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c009290).stop 2022-07-20T02:32:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- 172.21.15.26:0/4248046951 shutdown_connections 2022-07-20T02:32:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- 172.21.15.26:0/4248046951 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0034095b10 0x7f0034095f50 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- 172.21.15.26:0/4248046951 >> v2:172.21.15.120:3300/0 conn(0x7f00340b3260 0x7f0034060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- 172.21.15.26:0/4248046951 >> 172.21.15.26:0/4248046951 conn(0x7f00340b67f0 msgr2=0x7f00340b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- 172.21.15.26:0/4248046951 shutdown_connections 2022-07-20T02:32:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- 172.21.15.26:0/4248046951 wait complete. 2022-07-20T02:32:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 Processor -- start 2022-07-20T02:32:32.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- start start 2022-07-20T02:32:32.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0034095b10 0x7f00340920a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00340b3260 0x7f00340915e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003405f8e0 con 0x7f00340a95a0 2022-07-20T02:32:32.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003405fb80 con 0x7f00340b3260 2022-07-20T02:32:32.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.401+0000 7f003ae65700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003405fe20 con 0x7f0034095b10 2022-07-20T02:32:32.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039e63700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0034095b10 0x7f00340920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39910/0 (socket says 172.21.15.26:39910) 2022-07-20T02:32:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 -- 172.21.15.26:0/752456020 learned_addr learned my addr 172.21.15.26:0/752456020 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f003a664700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00340b3260 0x7f00340915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 -- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0034095b10 msgr2=0x7f00340920a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0034095b10 0x7f00340920a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 -- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00340b3260 msgr2=0x7f00340915e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00340b3260 0x7f00340915e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 -- 172.21.15.26:0/752456020 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c009530 con 0x7f00340a95a0 2022-07-20T02:32:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0039662700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f0034069b90 tx=0x7f0024007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0024007030 con 0x7f00340a95a0 2022-07-20T02:32:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024007190 con 0x7f00340a95a0 2022-07-20T02:32:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340600c0 con 0x7f00340a95a0 2022-07-20T02:32:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0024006510 con 0x7f00340a95a0 2022-07-20T02:32:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340b5750 con 0x7f00340a95a0 2022-07-20T02:32:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0034067150 con 0x7f00340a95a0 2022-07-20T02:32:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0024010030 con 0x7f00340a95a0 2022-07-20T02:32:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.405+0000 7f0022ffd700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 0x7f0014055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.409+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f002406b8c0 con 0x7f00340a95a0 2022-07-20T02:32:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.409+0000 7f0039e63700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 0x7f0014055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.409+0000 7f0039e63700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 0x7f0014055ea0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f003000c000 tx=0x7f0030008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.409+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0024042940 con 0x7f00340a95a0 2022-07-20T02:32:32.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.537+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0034061b00 con 0x7f00340a95a0 2022-07-20T02:32:32.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.537+0000 7f0022ffd700 1 -- 172.21.15.26:0/752456020 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v83) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0024045f60 con 0x7f00340a95a0 2022-07-20T02:32:32.543 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:32:32.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 msgr2=0x7f0014055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 0x7f0014055ea0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f003000c000 tx=0x7f0030008040).stop 2022-07-20T02:32:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 msgr2=0x7f00340afa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f0034069b90 tx=0x7f0024007c10).stop 2022-07-20T02:32:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 shutdown_connections 2022-07-20T02:32:32.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00340a95a0 0x7f00340afa00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f00140539e0 0x7f0014055ea0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00340b3260 0x7f00340915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 --2- 172.21.15.26:0/752456020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0034095b10 0x7f00340920a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 >> 172.21.15.26:0/752456020 conn(0x7f00340b67f0 msgr2=0x7f00340b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 shutdown_connections 2022-07-20T02:32:32.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.541+0000 7f003ae65700 1 -- 172.21.15.26:0/752456020 wait complete. 2022-07-20T02:32:32.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:32:32.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.677+0000 7f4de63ec700 1 Processor -- start 2022-07-20T02:32:32.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.677+0000 7f4de63ec700 1 -- start start 2022-07-20T02:32:32.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 0x7f4de0067070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4de009b9f0 0x7f4de0067560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de00613a0 con 0x7f4de00b31a0 2022-07-20T02:32:32.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de00615f0 con 0x7f4de009b9f0 2022-07-20T02:32:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de00b1980 con 0x7f4de00a9690 2022-07-20T02:32:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4de00b31a0 0x7f4de00b1e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39196/0 (socket says 172.21.15.26:39196) 2022-07-20T02:32:32.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddffff700 1 -- 172.21.15.26:0/1451878173 learned_addr learned my addr 172.21.15.26:0/1451878173 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:32.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 --2- 172.21.15.26:0/1451878173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 0x7f4de0067070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de4989700 1 --2- 172.21.15.26:0/1451878173 >> v2:172.21.15.120:3300/0 conn(0x7f4de009b9f0 0x7f4de0067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddf7fe700 1 -- 172.21.15.26:0/1451878173 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027362136 0 0) 0x7f4de00613a0 con 0x7f4de00b31a0 2022-07-20T02:32:32.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddf7fe700 1 -- 172.21.15.26:0/1451878173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc0003260 con 0x7f4de00b31a0 2022-07-20T02:32:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 -- 172.21.15.26:0/1451878173 >> v2:172.21.15.120:3300/0 conn(0x7f4de009b9f0 msgr2=0x7f4de0067560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 --2- 172.21.15.26:0/1451878173 >> v2:172.21.15.120:3300/0 conn(0x7f4de009b9f0 0x7f4de0067560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 -- 172.21.15.26:0/1451878173 >> v1:172.21.15.26:6789/0 conn(0x7f4de00b31a0 legacy=0x7f4de00b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 -- 172.21.15.26:0/1451878173 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de00b3580 con 0x7f4de00a9690 2022-07-20T02:32:32.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4dd7fff700 1 --2- 172.21.15.26:0/1451878173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 0x7f4de0067070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4dc8002f70 tx=0x7f4dc8006df0).ready entity=mon.2 client_cookie=2efec65b7526dc74 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddf7fe700 1 -- 172.21.15.26:0/1451878173 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dc800d4a0 con 0x7f4de00a9690 2022-07-20T02:32:32.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddf7fe700 1 -- 172.21.15.26:0/1451878173 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc8000e90 con 0x7f4de00a9690 2022-07-20T02:32:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4ddf7fe700 1 -- 172.21.15.26:0/1451878173 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dc800d7f0 con 0x7f4de00a9690 2022-07-20T02:32:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- 172.21.15.26:0/1451878173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 msgr2=0x7f4de0067070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 --2- 172.21.15.26:0/1451878173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 0x7f4de0067070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4dc8002f70 tx=0x7f4dc8006df0).stop 2022-07-20T02:32:32.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- 172.21.15.26:0/1451878173 shutdown_connections 2022-07-20T02:32:32.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 --2- 172.21.15.26:0/1451878173 >> v2:172.21.15.120:3300/0 conn(0x7f4de009b9f0 0x7f4de0067560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 --2- 172.21.15.26:0/1451878173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00a9690 0x7f4de0067070 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- 172.21.15.26:0/1451878173 >> 172.21.15.26:0/1451878173 conn(0x7f4de00b6920 msgr2=0x7f4de00b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- 172.21.15.26:0/1451878173 shutdown_connections 2022-07-20T02:32:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.681+0000 7f4de63ec700 1 -- 172.21.15.26:0/1451878173 wait complete. 2022-07-20T02:32:32.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 Processor -- start 2022-07-20T02:32:32.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- start start 2022-07-20T02:32:32.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de00a9690 0x7f4de00aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 0x7f4de01429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0156c00 con 0x7f4de009b9f0 2022-07-20T02:32:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de01566c0 con 0x7f4de00a9690 2022-07-20T02:32:32.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0156960 con 0x7f4de00b31a0 2022-07-20T02:32:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4ddffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4ddffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39924/0 (socket says 172.21.15.26:39924) 2022-07-20T02:32:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4ddffff700 1 -- 172.21.15.26:0/352121314 learned_addr learned my addr 172.21.15.26:0/352121314 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:32.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 0x7f4de01429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4dd7fff700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de00a9690 0x7f4de00aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 -- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de00a9690 msgr2=0x7f4de00aa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de00a9690 0x7f4de00aa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 -- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 msgr2=0x7f4de0156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 -- 172.21.15.26:0/352121314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc800b180 con 0x7f4de00b31a0 2022-07-20T02:32:32.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4ddffff700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de4989700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 0x7f4de01429d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4dd800c000 tx=0x7f4dd80091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dd8012600 con 0x7f4de00b31a0 2022-07-20T02:32:32.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8012760 con 0x7f4de00b31a0 2022-07-20T02:32:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de0093e80 con 0x7f4de00b31a0 2022-07-20T02:32:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dd8012ac0 con 0x7f4de00b31a0 2022-07-20T02:32:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.685+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de00921a0 con 0x7f4de00b31a0 2022-07-20T02:32:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de0094010 con 0x7f4de00b31a0 2022-07-20T02:32:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4dd8012c20 con 0x7f4de00b31a0 2022-07-20T02:32:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4dddffb700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 0x7f4dc4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4ddffff700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 0x7f4dc4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f4dd806c0c0 con 0x7f4de00b31a0 2022-07-20T02:32:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.689+0000 7f4ddffff700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 0x7f4dc4055e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f4dd000d000 tx=0x7f4dd0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.697+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4dd80430b0 con 0x7f4de00b31a0 2022-07-20T02:32:32.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.821+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4de00574e0 con 0x7f4de00b31a0 2022-07-20T02:32:32.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4dddffb700 1 -- 172.21.15.26:0/352121314 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v83) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4dd8016070 con 0x7f4de00b31a0 2022-07-20T02:32:32.832 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:32.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 msgr2=0x7f4dc4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 0x7f4dc4055e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f4dd000d000 tx=0x7f4dd0009040).stop 2022-07-20T02:32:32.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 msgr2=0x7f4de01429d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:32.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 0x7f4de01429d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4dd800c000 tx=0x7f4dd80091d0).stop 2022-07-20T02:32:32.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 shutdown_connections 2022-07-20T02:32:32.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4dc4053950 0x7f4dc4055e10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de00b31a0 0x7f4de01429d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de00a9690 0x7f4de00aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 --2- 172.21.15.26:0/352121314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4de009b9f0 0x7f4de0156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:32.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 >> 172.21.15.26:0/352121314 conn(0x7f4de00b6920 msgr2=0x7f4de00b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:32.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 shutdown_connections 2022-07-20T02:32:32.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:32.825+0000 7f4de63ec700 1 -- 172.21.15.26:0/352121314 wait complete. 2022-07-20T02:32:32.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:32:33.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 Processor -- start 2022-07-20T02:32:33.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 -- start start 2022-07-20T02:32:33.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f42e80b5ef0 0x7f42e81474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f42e80a8c40 0x7f42e80af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e80b62d0 con 0x7f42e80a7fc0 2022-07-20T02:32:33.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e814aac0 con 0x7f42e80a8c40 2022-07-20T02:32:33.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e814ad60 con 0x7f42e80b5ef0 2022-07-20T02:32:33.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f42e80a8c40 0x7f42e80af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f42e80a8c40 0x7f42e80af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39934/0 (socket says 172.21.15.26:39934) 2022-07-20T02:32:33.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 -- 172.21.15.26:0/1989852015 learned_addr learned my addr 172.21.15.26:0/1989852015 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:33.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e659c700 1 --2- 172.21.15.26:0/1989852015 >> v2:172.21.15.120:3300/0 conn(0x7f42e80b5ef0 0x7f42e81474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 -- 172.21.15.26:0/1989852015 >> v2:172.21.15.120:3300/0 conn(0x7f42e80b5ef0 msgr2=0x7f42e81474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:33.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 --2- 172.21.15.26:0/1989852015 >> v2:172.21.15.120:3300/0 conn(0x7f42e80b5ef0 0x7f42e81474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 -- 172.21.15.26:0/1989852015 >> v1:172.21.15.120:6789/0 conn(0x7f42e80a7fc0 legacy=0x7f42e80aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.025+0000 7f42e6d9d700 1 -- 172.21.15.26:0/1989852015 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e814b000 con 0x7f42e80a8c40 2022-07-20T02:32:33.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e6d9d700 1 --2- 172.21.15.26:0/1989852015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80a8c40 0x7f42e80af550 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f42e00032f0 tx=0x7f42e0009290).ready entity=mon.0 client_cookie=318ed25a68f022b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e559a700 1 -- 172.21.15.26:0/1989852015 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42e000b870 con 0x7f42e80a8c40 2022-07-20T02:32:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e559a700 1 -- 172.21.15.26:0/1989852015 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e000b9d0 con 0x7f42e80a8c40 2022-07-20T02:32:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e559a700 1 -- 172.21.15.26:0/1989852015 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42e000bb40 con 0x7f42e80a8c40 2022-07-20T02:32:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- 172.21.15.26:0/1989852015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80a8c40 msgr2=0x7f42e80af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- 172.21.15.26:0/1989852015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80a8c40 0x7f42e80af550 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f42e00032f0 tx=0x7f42e0009290).stop 2022-07-20T02:32:33.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- 172.21.15.26:0/1989852015 shutdown_connections 2022-07-20T02:32:33.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- 172.21.15.26:0/1989852015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80a8c40 0x7f42e80af550 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- 172.21.15.26:0/1989852015 >> v2:172.21.15.120:3300/0 conn(0x7f42e80b5ef0 0x7f42e81474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- 172.21.15.26:0/1989852015 >> 172.21.15.26:0/1989852015 conn(0x7f42e8060310 msgr2=0x7f42e80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:33.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- 172.21.15.26:0/1989852015 shutdown_connections 2022-07-20T02:32:33.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- 172.21.15.26:0/1989852015 wait complete. 2022-07-20T02:32:33.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 Processor -- start 2022-07-20T02:32:33.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- start start 2022-07-20T02:32:33.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 0x7f42e80b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 0x7f42e81437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80aacb0 con 0x7f42e80b5ef0 2022-07-20T02:32:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80a9e40 con 0x7f42e80a8c40 2022-07-20T02:32:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.029+0000 7f42e759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80aa0e0 con 0x7f42e80a7fc0 2022-07-20T02:32:33.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e659c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 0x7f42e80b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e5d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 0x7f42e81437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39940/0 (socket says 172.21.15.26:39940) 2022-07-20T02:32:33.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 -- 172.21.15.26:0/2775164336 learned_addr learned my addr 172.21.15.26:0/2775164336 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 -- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 msgr2=0x7f42e80b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 0x7f42e80b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 -- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 msgr2=0x7f42e81437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 0x7f42e81437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 -- 172.21.15.26:0/2775164336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0009530 con 0x7f42e80b5ef0 2022-07-20T02:32:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e5d9b700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 0x7f42e81437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e659c700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 0x7f42e80b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e6d9d700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f42e00032f0 tx=0x7f42e0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42e000b870 con 0x7f42e80b5ef0 2022-07-20T02:32:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e0021670 con 0x7f42e80b5ef0 2022-07-20T02:32:33.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e80ab160 con 0x7f42e80b5ef0 2022-07-20T02:32:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42e00219d0 con 0x7f42e80b5ef0 2022-07-20T02:32:33.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e80ab650 con 0x7f42e80b5ef0 2022-07-20T02:32:33.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.033+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e80a9c80 con 0x7f42e80b5ef0 2022-07-20T02:32:33.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.037+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f42e000f070 con 0x7f42e80b5ef0 2022-07-20T02:32:33.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.037+0000 7f42d77fe700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 0x7f42c4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.037+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f42e006cf50 con 0x7f42e80b5ef0 2022-07-20T02:32:33.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.037+0000 7f42e659c700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 0x7f42c4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.037+0000 7f42e659c700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 0x7f42c4055ed0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f42dc00b000 tx=0x7f42dc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.041+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42e0044100 con 0x7f42e80b5ef0 2022-07-20T02:32:33.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.169+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f42e814b2a0 con 0x7f42e80b5ef0 2022-07-20T02:32:33.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42d77fe700 1 -- 172.21.15.26:0/2775164336 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+143 (secure 0 0 0) 0x7f42e0015030 con 0x7f42e80b5ef0 2022-07-20T02:32:33.188 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:33.188 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:32:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:32:33.189 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:32:33.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:33.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:32:33.193 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:32:33.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:33.194 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:32:33.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 msgr2=0x7f42c4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 0x7f42c4055ed0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f42dc00b000 tx=0x7f42dc006df0).stop 2022-07-20T02:32:33.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 msgr2=0x7f42e8143ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f42e00032f0 tx=0x7f42e0016000).stop 2022-07-20T02:32:33.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 shutdown_connections 2022-07-20T02:32:33.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42e80b5ef0 0x7f42e8143ca0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f42c4053a10 0x7f42c4055ed0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e80a8c40 0x7f42e81437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 --2- 172.21.15.26:0/2775164336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f42e80a7fc0 0x7f42e80b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 >> 172.21.15.26:0/2775164336 conn(0x7f42e8060310 msgr2=0x7f42e805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:33.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 shutdown_connections 2022-07-20T02:32:33.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.173+0000 7f42e759e700 1 -- 172.21.15.26:0/2775164336 wait complete. 2022-07-20T02:32:33.217 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:32:33.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:32:33.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 Processor -- start 2022-07-20T02:32:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 -- start start 2022-07-20T02:32:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71d40b3380 0x7f71d414b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d4064880 con 0x7f71d4064450 2022-07-20T02:32:33.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d40b85f0 con 0x7f71d40b3380 2022-07-20T02:32:33.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.329+0000 7f71d359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40b80b0 con 0x7f71d40670c0 2022-07-20T02:32:33.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d1d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f71d4064450 0x7f71d4064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39228/0 (socket says 172.21.15.26:39228) 2022-07-20T02:32:33.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d1d9b700 1 -- 172.21.15.26:0/1198704137 learned_addr learned my addr 172.21.15.26:0/1198704137 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:33.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 --2- 172.21.15.26:0/1198704137 >> v2:172.21.15.120:3300/0 conn(0x7f71d40b3380 0x7f71d414b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d259c700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d159a700 1 -- 172.21.15.26:0/1198704137 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381576587 0 0) 0x7f71d4064880 con 0x7f71d4064450 2022-07-20T02:32:33.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d159a700 1 -- 172.21.15.26:0/1198704137 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b4003260 con 0x7f71d4064450 2022-07-20T02:32:33.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 -- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 msgr2=0x7f71d40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:33.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 -- 172.21.15.26:0/1198704137 >> v1:172.21.15.26:6789/0 conn(0x7f71d4064450 legacy=0x7f71d4064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 -- 172.21.15.26:0/1198704137 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40b8350 con 0x7f71d40b3380 2022-07-20T02:32:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d2d9d700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d40b3380 0x7f71d414b840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f71cc0032f0 tx=0x7f71cc0091d0).ready entity=mon.1 client_cookie=96840bdb143ee43d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d159a700 1 -- 172.21.15.26:0/1198704137 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc00b600 con 0x7f71d40b3380 2022-07-20T02:32:33.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d159a700 1 -- 172.21.15.26:0/1198704137 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc00b760 con 0x7f71d40b3380 2022-07-20T02:32:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d159a700 1 -- 172.21.15.26:0/1198704137 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc00b8d0 con 0x7f71d40b3380 2022-07-20T02:32:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 -- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d40b3380 msgr2=0x7f71d414b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d40b3380 0x7f71d414b840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f71cc0032f0 tx=0x7f71cc0091d0).stop 2022-07-20T02:32:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 -- 172.21.15.26:0/1198704137 shutdown_connections 2022-07-20T02:32:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d40b3380 0x7f71d414b840 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 --2- 172.21.15.26:0/1198704137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 -- 172.21.15.26:0/1198704137 >> 172.21.15.26:0/1198704137 conn(0x7f71d405f840 msgr2=0x7f71d405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:33.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 -- 172.21.15.26:0/1198704137 shutdown_connections 2022-07-20T02:32:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.333+0000 7f71d359e700 1 -- 172.21.15.26:0/1198704137 wait complete. 2022-07-20T02:32:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 Processor -- start 2022-07-20T02:32:33.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- start start 2022-07-20T02:32:33.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d413f540 0x7f71d41506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40b0080 con 0x7f71d414fef0 2022-07-20T02:32:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40b0320 con 0x7f71d413f540 2022-07-20T02:32:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d4153620 con 0x7f71d40670c0 2022-07-20T02:32:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39954/0 (socket says 172.21.15.26:39954) 2022-07-20T02:32:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 -- 172.21.15.26:0/259356587 learned_addr learned my addr 172.21.15.26:0/259356587 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d259c700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d2d9d700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d413f540 0x7f71d41506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 -- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 msgr2=0x7f71d40af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 -- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d413f540 msgr2=0x7f71d41506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d413f540 0x7f71d41506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 -- 172.21.15.26:0/259356587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc009470 con 0x7f71d414fef0 2022-07-20T02:32:33.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d1d9b700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f71c4001110 tx=0x7f71c4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c400ce50 con 0x7f71d414fef0 2022-07-20T02:32:33.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c40072e0 con 0x7f71d414fef0 2022-07-20T02:32:33.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d4138b70 con 0x7f71d414fef0 2022-07-20T02:32:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c4005800 con 0x7f71d414fef0 2022-07-20T02:32:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.337+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d4146f30 con 0x7f71d414fef0 2022-07-20T02:32:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d4150ed0 con 0x7f71d414fef0 2022-07-20T02:32:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f71c4010070 con 0x7f71d414fef0 2022-07-20T02:32:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71cb7fe700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 0x7f71b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f71c406b520 con 0x7f71d414fef0 2022-07-20T02:32:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71d259c700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 0x7f71b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.341+0000 7f71d259c700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 0x7f71b8055ea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f71bc00b000 tx=0x7f71bc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.349+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71c4042580 con 0x7f71d414fef0 2022-07-20T02:32:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.473+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f71d4064ab0 con 0x7f71d414fef0 2022-07-20T02:32:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71cb7fe700 1 -- 172.21.15.26:0/259356587 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f71c4045ba0 con 0x7f71d414fef0 2022-07-20T02:32:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 msgr2=0x7f71b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 0x7f71b8055ea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f71bc00b000 tx=0x7f71bc006df0).stop 2022-07-20T02:32:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 msgr2=0x7f71d413f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f71c4001110 tx=0x7f71c4007af0).stop 2022-07-20T02:32:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 shutdown_connections 2022-07-20T02:32:33.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f71d414fef0 0x7f71d413f050 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f71b80539e0 0x7f71b8055ea0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d413f540 0x7f71d41506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 --2- 172.21.15.26:0/259356587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d40670c0 0x7f71d40af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 >> 172.21.15.26:0/259356587 conn(0x7f71d405f840 msgr2=0x7f71d405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 shutdown_connections 2022-07-20T02:32:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.717+0000 7f71d359e700 1 -- 172.21.15.26:0/259356587 wait complete. 2022-07-20T02:32:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:32:33.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:32:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 Processor -- start 2022-07-20T02:32:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 -- start start 2022-07-20T02:32:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c05ce30 0x7f8b1c066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c146bb0 con 0x7f8b1c060c80 2022-07-20T02:32:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1c146e50 con 0x7f8b1c0ac6f0 2022-07-20T02:32:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.889+0000 7f8b1bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c1470f0 con 0x7f8b1c05ce30 2022-07-20T02:32:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39960/0 (socket says 172.21.15.26:39960) 2022-07-20T02:32:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 -- 172.21.15.26:0/475138637 learned_addr learned my addr 172.21.15.26:0/475138637 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1affd700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c05ce30 0x7f8b1c066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 -- 172.21.15.26:0/475138637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c05ce30 msgr2=0x7f8b1c066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c05ce30 0x7f8b1c066c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 -- 172.21.15.26:0/475138637 >> v1:172.21.15.120:6789/0 conn(0x7f8b1c060c80 legacy=0x7f8b1c061110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 -- 172.21.15.26:0/475138637 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c147390 con 0x7f8b1c0ac6f0 2022-07-20T02:32:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1a7fc700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f8b04002f70 tx=0x7f8b04006eb0).ready entity=mon.0 client_cookie=4a32b6e6664224d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b19ffb700 1 -- 172.21.15.26:0/475138637 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0400d570 con 0x7f8b1c0ac6f0 2022-07-20T02:32:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b19ffb700 1 -- 172.21.15.26:0/475138637 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04000e90 con 0x7f8b1c0ac6f0 2022-07-20T02:32:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b19ffb700 1 -- 172.21.15.26:0/475138637 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0400d900 con 0x7f8b1c0ac6f0 2022-07-20T02:32:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 -- 172.21.15.26:0/475138637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c0ac6f0 msgr2=0x7f8b1c0acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f8b04002f70 tx=0x7f8b04006eb0).stop 2022-07-20T02:32:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 -- 172.21.15.26:0/475138637 shutdown_connections 2022-07-20T02:32:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0acb10 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/475138637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c05ce30 0x7f8b1c066c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 -- 172.21.15.26:0/475138637 >> 172.21.15.26:0/475138637 conn(0x7f8b1c0b7090 msgr2=0x7f8b1c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 -- 172.21.15.26:0/475138637 shutdown_connections 2022-07-20T02:32:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.893+0000 7f8b1bfff700 1 -- 172.21.15.26:0/475138637 wait complete. 2022-07-20T02:32:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 Processor -- start 2022-07-20T02:32:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- start start 2022-07-20T02:32:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c060c80 0x7f8b1c0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0aec00 con 0x7f8b1c05ce30 2022-07-20T02:32:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0660a0 con 0x7f8b1c0ac6f0 2022-07-20T02:32:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0aeee0 con 0x7f8b1c060c80 2022-07-20T02:32:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39964/0 (socket says 172.21.15.26:39964) 2022-07-20T02:32:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 -- 172.21.15.26:0/2977450210 learned_addr learned my addr 172.21.15.26:0/2977450210 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1a7fc700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c060c80 0x7f8b1c0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1b7fe700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 -- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c060c80 msgr2=0x7f8b1c0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c060c80 0x7f8b1c0b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 -- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b1c0ac6f0 msgr2=0x7f8b1c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 -- 172.21.15.26:0/2977450210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0400b1e0 con 0x7f8b1c05ce30 2022-07-20T02:32:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1affd700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f8b0c00c000 tx=0x7f8b0c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c00c6a0 con 0x7f8b1c05ce30 2022-07-20T02:32:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00c800 con 0x7f8b1c05ce30 2022-07-20T02:32:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1c0af180 con 0x7f8b1c05ce30 2022-07-20T02:32:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c00cb60 con 0x7f8b1c05ce30 2022-07-20T02:32:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.897+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c065360 con 0x7f8b1c05ce30 2022-07-20T02:32:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c061820 con 0x7f8b1c05ce30 2022-07-20T02:32:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8b0c022070 con 0x7f8b1c05ce30 2022-07-20T02:32:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b13fff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 0x7f8b00055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f8b0c06c4b0 con 0x7f8b1c05ce30 2022-07-20T02:32:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b1a7fc700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 0x7f8b00055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.901+0000 7f8b1a7fc700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 0x7f8b00055f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f8b0401c000 tx=0x7f8b04015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:33.909+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b0c043440 con 0x7f8b1c05ce30 2022-07-20T02:32:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.037+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f8b1c0574e0 con 0x7f8b1c05ce30 2022-07-20T02:32:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.265+0000 7f8b13fff700 1 -- 172.21.15.26:0/2977450210 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v85) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8b0c046a60 con 0x7f8b1c05ce30 2022-07-20T02:32:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 msgr2=0x7f8b00055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 0x7f8b00055f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f8b0401c000 tx=0x7f8b04015000).stop 2022-07-20T02:32:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 msgr2=0x7f8b1c0b5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f8b0c00c000 tx=0x7f8b0c0091d0).stop 2022-07-20T02:32:34.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 shutdown_connections 2022-07-20T02:32:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8b00053a70 0x7f8b00055f30 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0x7f8b0401c000 tx=0x7f8b04015000).stop 2022-07-20T02:32:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b1c0ac6f0 0x7f8b1c0b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b1c060c80 0x7f8b1c0b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 --2- 172.21.15.26:0/2977450210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b1c05ce30 0x7f8b1c0b5bc0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.269+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 >> 172.21.15.26:0/2977450210 conn(0x7f8b1c0b7090 msgr2=0x7f8b1c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.273+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 shutdown_connections 2022-07-20T02:32:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.273+0000 7f8b1bfff700 1 -- 172.21.15.26:0/2977450210 wait complete. 2022-07-20T02:32:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:32:34.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:32:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.509+0000 7f838d7bc700 1 Processor -- start 2022-07-20T02:32:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.509+0000 7f838d7bc700 1 -- start start 2022-07-20T02:32:34.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.509+0000 7f838d7bc700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83881432e0 0x7f838812c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.509+0000 7f838d7bc700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8388061e00 con 0x7f8388061a20 2022-07-20T02:32:34.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838812cc40 con 0x7f83881468c0 2022-07-20T02:32:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8388147040 con 0x7f83881432e0 2022-07-20T02:32:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8388061a20 0x7f838812c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39252/0 (socket says 172.21.15.26:39252) 2022-07-20T02:32:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 -- 172.21.15.26:0/603897753 learned_addr learned my addr 172.21.15.26:0/603897753 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f83867fc700 1 --2- 172.21.15.26:0/603897753 >> v2:172.21.15.120:3300/0 conn(0x7f83881432e0 0x7f838812c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620891264 0 0) 0x7f8388061e00 con 0x7f8388061a20 2022-07-20T02:32:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8368003260 con 0x7f8388061a20 2022-07-20T02:32:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259546989 0 0) 0x7f838812cc40 con 0x7f83881468c0 2022-07-20T02:32:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8388061e00 con 0x7f83881468c0 2022-07-20T02:32:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468942850 0 0) 0x7f8368003260 con 0x7f8388061a20 2022-07-20T02:32:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 >> v2:172.21.15.120:3300/0 conn(0x7f83881432e0 msgr2=0x7f838812c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 --2- 172.21.15.26:0/603897753 >> v2:172.21.15.120:3300/0 conn(0x7f83881432e0 0x7f838812c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 >> v1:172.21.15.120:6789/0 conn(0x7f83881468c0 legacy=0x7f83880b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83881472e0 con 0x7f8388061a20 2022-07-20T02:32:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f83800019e0 con 0x7f8388061a20 2022-07-20T02:32:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 >> v1:172.21.15.26:6789/0 conn(0x7f8388061a20 legacy=0x7f838812c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 0x7f8368003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8368004500 0x7f8368004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8368004e70 0x7f836800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388147040 con 0x7f83680038e0 2022-07-20T02:32:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836800bcf0 con 0x7f8368004500 2022-07-20T02:32:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836800beb0 con 0x7f8368004e70 2022-07-20T02:32:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 0x7f8368003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f83867fc700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8368004500 0x7f8368004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f83877fe700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8368004e70 0x7f836800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 -- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8368004e70 msgr2=0x7f836800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8368004e70 0x7f836800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 -- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8368004500 msgr2=0x7f8368004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8368004500 0x7f8368004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 -- 172.21.15.26:0/603897753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368003260 con 0x7f83680038e0 2022-07-20T02:32:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f83877fe700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8368004e70 0x7f836800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f83877fe700 1 -- 172.21.15.26:0/603897753 reap_dead start 2022-07-20T02:32:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8386ffd700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 0x7f8368003ce0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f83800017b0 tx=0x7f8388147470).ready entity=mon.0 client_cookie=768df4a5693b2a31 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838000cbe0 con 0x7f83680038e0 2022-07-20T02:32:34.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838000cd40 con 0x7f83680038e0 2022-07-20T02:32:34.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f8385ffb700 1 -- 172.21.15.26:0/603897753 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8380005b80 con 0x7f83680038e0 2022-07-20T02:32:34.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 msgr2=0x7f8368003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 0x7f8368003ce0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f83800017b0 tx=0x7f8388147470).stop 2022-07-20T02:32:34.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- 172.21.15.26:0/603897753 shutdown_connections 2022-07-20T02:32:34.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 --2- 172.21.15.26:0/603897753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83680038e0 0x7f8368003ce0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- 172.21.15.26:0/603897753 >> 172.21.15.26:0/603897753 conn(0x7f838805faf0 msgr2=0x7f8388066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:34.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- 172.21.15.26:0/603897753 shutdown_connections 2022-07-20T02:32:34.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.513+0000 7f838d7bc700 1 -- 172.21.15.26:0/603897753 wait complete. 2022-07-20T02:32:34.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 Processor -- start 2022-07-20T02:32:34.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- start start 2022-07-20T02:32:34.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 0x7f838812d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880af6d0 0x7f838812b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838812bcb0 con 0x7f838812b820 2022-07-20T02:32:34.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880ae3a0 con 0x7f83680038e0 2022-07-20T02:32:34.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880ad8a0 con 0x7f83880af6d0 2022-07-20T02:32:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f8386ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 0x7f838812d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39988/0 (socket says 172.21.15.26:39988) 2022-07-20T02:32:34.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 -- 172.21.15.26:0/1395750735 learned_addr learned my addr 172.21.15.26:0/1395750735 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83867fc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880af6d0 0x7f838812b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 -- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880af6d0 msgr2=0x7f838812b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880af6d0 0x7f838812b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 -- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 msgr2=0x7f838812d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 0x7f838812d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 -- 172.21.15.26:0/1395750735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880ae650 con 0x7f838812b820 2022-07-20T02:32:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f8386ffd700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 0x7f838812d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f83877fe700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f83880ae4e0 tx=0x7f8378009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f837800dd30 con 0x7f838812b820 2022-07-20T02:32:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837800de90 con 0x7f838812b820 2022-07-20T02:32:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880adad0 con 0x7f838812b820 2022-07-20T02:32:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8378013660 con 0x7f838812b820 2022-07-20T02:32:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.517+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880afe50 con 0x7f838812b820 2022-07-20T02:32:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388147580 con 0x7f838812b820 2022-07-20T02:32:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f837801a030 con 0x7f838812b820 2022-07-20T02:32:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f837ffff700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 0x7f836c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f837806b650 con 0x7f838812b820 2022-07-20T02:32:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f8386ffd700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 0x7f836c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.521+0000 7f8386ffd700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 0x7f836c055f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8380006000 tx=0x7f8380015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.525+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8378042490 con 0x7f838812b820 2022-07-20T02:32:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.689+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f83880574e0 con 0x7f838812b820 2022-07-20T02:32:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.697+0000 7f837ffff700 1 -- 172.21.15.26:0/1395750735 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v42) v1 ==== 189+0+0 (secure 0 0 0) 0x7f8378045ab0 con 0x7f838812b820 2022-07-20T02:32:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.709+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 msgr2=0x7f836c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 0x7f836c055f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8380006000 tx=0x7f8380015000).stop 2022-07-20T02:32:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 msgr2=0x7f83881525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f83880ae4e0 tx=0x7f8378009040).stop 2022-07-20T02:32:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 shutdown_connections 2022-07-20T02:32:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f838812b820 0x7f83881525b0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f836c053a70 0x7f836c055f30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83880af6d0 0x7f838812b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 --2- 172.21.15.26:0/1395750735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83680038e0 0x7f838812d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 >> 172.21.15.26:0/1395750735 conn(0x7f838805faf0 msgr2=0x7f838805cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 shutdown_connections 2022-07-20T02:32:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.713+0000 7f838d7bc700 1 -- 172.21.15.26:0/1395750735 wait complete. 2022-07-20T02:32:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 15 and data pool 16 2022-07-20T02:32:34.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:34.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.893+0000 7f8644b24700 1 Processor -- start 2022-07-20T02:32:34.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.893+0000 7f8644b24700 1 -- start start 2022-07-20T02:32:34.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 0x7f8640067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:34.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86400b1bc0 con 0x7f86401484a0 2022-07-20T02:32:34.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640147700 con 0x7f86400b2920 2022-07-20T02:32:34.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864014b040 con 0x7f8640061050 2022-07-20T02:32:34.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863dd9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86401484a0 0x7f8640066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39284/0 (socket says 172.21.15.26:39284) 2022-07-20T02:32:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863dd9b700 1 -- 172.21.15.26:0/1919127516 learned_addr learned my addr 172.21.15.26:0/1919127516 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863ed9d700 1 --2- 172.21.15.26:0/1919127516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 0x7f8640067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128523340 0 0) 0x7f86400b1bc0 con 0x7f86401484a0 2022-07-20T02:32:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624003260 con 0x7f86401484a0 2022-07-20T02:32:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641240635 0 0) 0x7f8640147700 con 0x7f86400b2920 2022-07-20T02:32:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86400b1bc0 con 0x7f86400b2920 2022-07-20T02:32:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863ed9d700 1 -- 172.21.15.26:0/1919127516 >> v1:172.21.15.120:6789/0 conn(0x7f86400b2920 legacy=0x7f8640147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863ed9d700 1 -- 172.21.15.26:0/1919127516 >> v1:172.21.15.26:6789/0 conn(0x7f86401484a0 legacy=0x7f8640066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863ed9d700 1 -- 172.21.15.26:0/1919127516 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8640147930 con 0x7f8640061050 2022-07-20T02:32:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863ed9d700 1 --2- 172.21.15.26:0/1919127516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 0x7f8640067190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f86380032f0 tx=0x7f86380091d0).ready entity=mon.2 client_cookie=e91474d057b3f557 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863800b640 con 0x7f8640061050 2022-07-20T02:32:34.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863800b7a0 con 0x7f8640061050 2022-07-20T02:32:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f863d59a700 1 -- 172.21.15.26:0/1919127516 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863800b910 con 0x7f8640061050 2022-07-20T02:32:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- 172.21.15.26:0/1919127516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 msgr2=0x7f8640067190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 --2- 172.21.15.26:0/1919127516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 0x7f8640067190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f86380032f0 tx=0x7f86380091d0).stop 2022-07-20T02:32:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- 172.21.15.26:0/1919127516 shutdown_connections 2022-07-20T02:32:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 --2- 172.21.15.26:0/1919127516 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8640061050 0x7f8640067190 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- 172.21.15.26:0/1919127516 >> 172.21.15.26:0/1919127516 conn(0x7f86400b60a0 msgr2=0x7f86400b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- 172.21.15.26:0/1919127516 shutdown_connections 2022-07-20T02:32:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.897+0000 7f8644b24700 1 -- 172.21.15.26:0/1919127516 wait complete. 2022-07-20T02:32:35.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 Processor -- start 2022-07-20T02:32:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- start start 2022-07-20T02:32:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 0x7f864014a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86400b2920 0x7f8640066340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640064290 con 0x7f86401484a0 2022-07-20T02:32:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640064990 con 0x7f8640061050 2022-07-20T02:32:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8640064db0 con 0x7f86400b2920 2022-07-20T02:32:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40014/0 (socket says 172.21.15.26:40014) 2022-07-20T02:32:35.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 -- 172.21.15.26:0/1704496042 learned_addr learned my addr 172.21.15.26:0/1704496042 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863dd9b700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86400b2920 0x7f8640066340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863e59c700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 0x7f864014a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 -- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86400b2920 msgr2=0x7f8640066340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86400b2920 0x7f8640066340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 -- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 msgr2=0x7f864014a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 0x7f864014a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 -- 172.21.15.26:0/1704496042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638009470 con 0x7f86401484a0 2022-07-20T02:32:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863e59c700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 0x7f864014a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f863ed9d700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f86380032f0 tx=0x7f8638016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863800b910 con 0x7f86401484a0 2022-07-20T02:32:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863801f3f0 con 0x7f86401484a0 2022-07-20T02:32:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8640065050 con 0x7f86401484a0 2022-07-20T02:32:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863800b640 con 0x7f86401484a0 2022-07-20T02:32:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.901+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400644f0 con 0x7f86401484a0 2022-07-20T02:32:35.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86400646b0 con 0x7f86401484a0 2022-07-20T02:32:35.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8638026030 con 0x7f86401484a0 2022-07-20T02:32:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f862f7fe700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 0x7f861c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f863806ce60 con 0x7f86401484a0 2022-07-20T02:32:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f863e59c700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 0x7f861c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.905+0000 7f863e59c700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 0x7f861c055fc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f863400b000 tx=0x7f8634009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:34.913+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8638043b10 con 0x7f86401484a0 2022-07-20T02:32:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.041+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f864014ada0 con 0x7f86401484a0 2022-07-20T02:32:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.041+0000 7f862f7fe700 1 -- 172.21.15.26:0/1704496042 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16335 (secure 0 0 0) 0x7f8638024070 con 0x7f86401484a0 2022-07-20T02:32:35.056 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:35.056 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":86,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:34.693710+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:33.535657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:34.094152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:35.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 msgr2=0x7f861c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 0x7f861c055fc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f863400b000 tx=0x7f8634009040).stop 2022-07-20T02:32:35.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 msgr2=0x7f8640145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f86380032f0 tx=0x7f8638016000).stop 2022-07-20T02:32:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 shutdown_connections 2022-07-20T02:32:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86401484a0 0x7f8640145890 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f861c053b00 0x7f861c055fc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86400b2920 0x7f8640066340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 --2- 172.21.15.26:0/1704496042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8640061050 0x7f864014a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 >> 172.21.15.26:0/1704496042 conn(0x7f86400b60a0 msgr2=0x7f86400b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 shutdown_connections 2022-07-20T02:32:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.045+0000 7f8644b24700 1 -- 172.21.15.26:0/1704496042 wait complete. 2022-07-20T02:32:35.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:35.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 Processor -- start 2022-07-20T02:32:35.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 -- start start 2022-07-20T02:32:35.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd980b1c60 0x7fcd980a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd98098320 0x7fcd98060020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd98098800 con 0x7fcd98060510 2022-07-20T02:32:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd98098aa0 con 0x7fcd98098320 2022-07-20T02:32:35.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcda015a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9805f8e0 con 0x7fcd980b1c60 2022-07-20T02:32:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd98098320 0x7fcd98060020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd98098320 0x7fcd98060020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40020/0 (socket says 172.21.15.26:40020) 2022-07-20T02:32:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9e6f7700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcd98060510 0x7fcd98143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39302/0 (socket says 172.21.15.26:39302) 2022-07-20T02:32:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 -- 172.21.15.26:0/3151124820 learned_addr learned my addr 172.21.15.26:0/3151124820 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9def6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd980b1c60 0x7fcd980a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 -- 172.21.15.26:0/3151124820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd980b1c60 msgr2=0x7fcd980a9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 --2- 172.21.15.26:0/3151124820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd980b1c60 0x7fcd980a9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 -- 172.21.15.26:0/3151124820 >> v1:172.21.15.26:6789/0 conn(0x7fcd98060510 legacy=0x7fcd98143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9cef4700 1 -- 172.21.15.26:0/3151124820 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200202940 0 0) 0x7fcd98098800 con 0x7fcd98060510 2022-07-20T02:32:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 -- 172.21.15.26:0/3151124820 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9805fb80 con 0x7fcd98098320 2022-07-20T02:32:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.189+0000 7fcd9d6f5700 1 --2- 172.21.15.26:0/3151124820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd98098320 0x7fcd98060020 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fcd98061770 tx=0x7fcd88006f40).ready entity=mon.0 client_cookie=beec70eaae23b5f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcd9cef4700 1 -- 172.21.15.26:0/3151124820 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8800da40 con 0x7fcd98098320 2022-07-20T02:32:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcd9cef4700 1 -- 172.21.15.26:0/3151124820 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88000e90 con 0x7fcd98098320 2022-07-20T02:32:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcd9cef4700 1 -- 172.21.15.26:0/3151124820 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8800dd90 con 0x7fcd98098320 2022-07-20T02:32:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 -- 172.21.15.26:0/3151124820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd98098320 msgr2=0x7fcd98060020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 --2- 172.21.15.26:0/3151124820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd98098320 0x7fcd98060020 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fcd98061770 tx=0x7fcd88006f40).stop 2022-07-20T02:32:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 -- 172.21.15.26:0/3151124820 shutdown_connections 2022-07-20T02:32:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 --2- 172.21.15.26:0/3151124820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd98098320 0x7fcd98060020 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 --2- 172.21.15.26:0/3151124820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd980b1c60 0x7fcd980a9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 -- 172.21.15.26:0/3151124820 >> 172.21.15.26:0/3151124820 conn(0x7fcd980b5e70 msgr2=0x7fcd980b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 -- 172.21.15.26:0/3151124820 shutdown_connections 2022-07-20T02:32:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.193+0000 7fcda015a700 1 -- 172.21.15.26:0/3151124820 wait complete. 2022-07-20T02:32:35.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 Processor -- start 2022-07-20T02:32:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- start start 2022-07-20T02:32:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd98060510 0x7fcd980b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd98098320 0x7fcd9813db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980aecc0 con 0x7fcd980b1c60 2022-07-20T02:32:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98151dc0 con 0x7fcd98098320 2022-07-20T02:32:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98152060 con 0x7fcd98060510 2022-07-20T02:32:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40028/0 (socket says 172.21.15.26:40028) 2022-07-20T02:32:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 -- 172.21.15.26:0/3395617340 learned_addr learned my addr 172.21.15.26:0/3395617340 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9def6700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd98060510 0x7fcd980b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9d6f5700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd98098320 0x7fcd9813db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 -- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd98060510 msgr2=0x7fcd980b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd98060510 0x7fcd980b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 -- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd98098320 msgr2=0x7fcd9813db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd98098320 0x7fcd9813db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 -- 172.21.15.26:0/3395617340 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8800b460 con 0x7fcd980b1c60 2022-07-20T02:32:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd9e6f7700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fcd90000f40 tx=0x7fcd90007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd90005ae0 con 0x7fcd980b1c60 2022-07-20T02:32:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd90005c40 con 0x7fcd980b1c60 2022-07-20T02:32:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd98152300 con 0x7fcd980b1c60 2022-07-20T02:32:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd900070d0 con 0x7fcd980b1c60 2022-07-20T02:32:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.197+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9813e4e0 con 0x7fcd980b1c60 2022-07-20T02:32:35.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98151b80 con 0x7fcd980b1c60 2022-07-20T02:32:35.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd9001a060 con 0x7fcd980b1c60 2022-07-20T02:32:35.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcd86ffd700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 0x7fcd78055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcd9006bb70 con 0x7fcd980b1c60 2022-07-20T02:32:35.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcd9def6700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 0x7fcd78055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.201+0000 7fcd9def6700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 0x7fcd78055fc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcd9400b000 tx=0x7fcd94006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.205+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd98151b80 con 0x7fcd980b1c60 2022-07-20T02:32:35.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.333+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd98151b80 con 0x7fcd980b1c60 2022-07-20T02:32:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.333+0000 7fcd86ffd700 1 -- 172.21.15.26:0/3395617340 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16335 (secure 0 0 0) 0x7fcd98151b80 con 0x7fcd980b1c60 2022-07-20T02:32:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:35.339 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":86,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:34.693710+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:33.535657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:34.094152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 msgr2=0x7fcd78055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 0x7fcd78055fc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcd9400b000 tx=0x7fcd94006df0).stop 2022-07-20T02:32:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 msgr2=0x7fcd9813dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fcd90000f40 tx=0x7fcd90007ef0).stop 2022-07-20T02:32:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 shutdown_connections 2022-07-20T02:32:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd980b1c60 0x7fcd9813dff0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd78053b00 0x7fcd78055fc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd98098320 0x7fcd9813db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 --2- 172.21.15.26:0/3395617340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd98060510 0x7fcd980b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 >> 172.21.15.26:0/3395617340 conn(0x7fcd980b5e70 msgr2=0x7fcd98098f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 shutdown_connections 2022-07-20T02:32:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.337+0000 7fcda015a700 1 -- 172.21.15.26:0/3395617340 wait complete. 2022-07-20T02:32:35.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:32:35.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.529+0000 7fda5bfff700 1 Processor -- start 2022-07-20T02:32:35.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.529+0000 7fda5bfff700 1 -- start start 2022-07-20T02:32:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda5c0670c0 0x7fda5c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 0x7fda5c14b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5c064880 con 0x7fda5c064450 2022-07-20T02:32:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c0b85f0 con 0x7fda5c0670c0 2022-07-20T02:32:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0b80b0 con 0x7fda5c0b3380 2022-07-20T02:32:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5a7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fda5c064450 0x7fda5c064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39312/0 (socket says 172.21.15.26:39312) 2022-07-20T02:32:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5a7fc700 1 -- 172.21.15.26:0/1232552712 learned_addr learned my addr 172.21.15.26:0/1232552712 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 --2- 172.21.15.26:0/1232552712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 0x7fda5c14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5affd700 1 --2- 172.21.15.26:0/1232552712 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0670c0 0x7fda5c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038454960 0 0) 0x7fda5c064880 con 0x7fda5c064450 2022-07-20T02:32:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3c003260 con 0x7fda5c064450 2022-07-20T02:32:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265789191 0 0) 0x7fda3c003260 con 0x7fda5c064450 2022-07-20T02:32:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 -- 172.21.15.26:0/1232552712 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0670c0 msgr2=0x7fda5c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 --2- 172.21.15.26:0/1232552712 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0670c0 0x7fda5c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 -- 172.21.15.26:0/1232552712 >> v1:172.21.15.26:6789/0 conn(0x7fda5c064450 legacy=0x7fda5c064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 -- 172.21.15.26:0/1232552712 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0b8350 con 0x7fda5c0b3380 2022-07-20T02:32:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5b7fe700 1 --2- 172.21.15.26:0/1232552712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 0x7fda5c14b840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fda44001820 tx=0x7fda540091d0).ready entity=mon.2 client_cookie=73839818bae080f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda44001680 con 0x7fda5c0b3380 2022-07-20T02:32:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5400f510 con 0x7fda5c0b3380 2022-07-20T02:32:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda59ffb700 1 -- 172.21.15.26:0/1232552712 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda5400f680 con 0x7fda5c0b3380 2022-07-20T02:32:35.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 -- 172.21.15.26:0/1232552712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 msgr2=0x7fda5c14b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.533+0000 7fda5bfff700 1 --2- 172.21.15.26:0/1232552712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 0x7fda5c14b840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fda44001820 tx=0x7fda540091d0).stop 2022-07-20T02:32:35.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- 172.21.15.26:0/1232552712 shutdown_connections 2022-07-20T02:32:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 --2- 172.21.15.26:0/1232552712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c0b3380 0x7fda5c14b840 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 --2- 172.21.15.26:0/1232552712 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0670c0 0x7fda5c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- 172.21.15.26:0/1232552712 >> 172.21.15.26:0/1232552712 conn(0x7fda5c05f840 msgr2=0x7fda5c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- 172.21.15.26:0/1232552712 shutdown_connections 2022-07-20T02:32:35.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- 172.21.15.26:0/1232552712 wait complete. 2022-07-20T02:32:35.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 Processor -- start 2022-07-20T02:32:35.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- start start 2022-07-20T02:32:35.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 0x7fda5c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c138df0 0x7fda5c139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c13a5e0 con 0x7fda5c0670c0 2022-07-20T02:32:35.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c060df0 con 0x7fda5c1408c0 2022-07-20T02:32:35.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c13a810 con 0x7fda5c138df0 2022-07-20T02:32:35.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40038/0 (socket says 172.21.15.26:40038) 2022-07-20T02:32:35.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5affd700 1 -- 172.21.15.26:0/585563179 learned_addr learned my addr 172.21.15.26:0/585563179 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5a7fc700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 0x7fda5c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.537+0000 7fda5b7fe700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c138df0 0x7fda5c139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 -- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c138df0 msgr2=0x7fda5c139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c138df0 0x7fda5c139270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 -- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 msgr2=0x7fda5c13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 0x7fda5c13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 -- 172.21.15.26:0/585563179 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54009470 con 0x7fda5c0670c0 2022-07-20T02:32:35.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5a7fc700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 0x7fda5c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:35.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5affd700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fda4c00c000 tx=0x7fda4c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4c010780 con 0x7fda5c0670c0 2022-07-20T02:32:35.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c0108e0 con 0x7fda5c0670c0 2022-07-20T02:32:35.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c061020 con 0x7fda5c0670c0 2022-07-20T02:32:35.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4c010c40 con 0x7fda5c0670c0 2022-07-20T02:32:35.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c13b950 con 0x7fda5c0670c0 2022-07-20T02:32:35.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c140ca0 con 0x7fda5c0670c0 2022-07-20T02:32:35.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fda4c01c460 con 0x7fda5c0670c0 2022-07-20T02:32:35.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 0x7fda40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda4c06c4a0 con 0x7fda5c0670c0 2022-07-20T02:32:35.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.541+0000 7fda5a7fc700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 0x7fda40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.545+0000 7fda5a7fc700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 0x7fda40055f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fda4400d000 tx=0x7fda44001010).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.549+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fda4c043160 con 0x7fda5c0670c0 2022-07-20T02:32:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.713+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fda5c062570 con 0x7fda5c0670c0 2022-07-20T02:32:35.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.725+0000 7fda53fff700 1 -- 172.21.15.26:0/585563179 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7fda4c046780 con 0x7fda5c0670c0 2022-07-20T02:32:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 msgr2=0x7fda40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 0x7fda40055f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fda4400d000 tx=0x7fda44001010).stop 2022-07-20T02:32:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 msgr2=0x7fda5c146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fda4c00c000 tx=0x7fda4c006df0).stop 2022-07-20T02:32:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 shutdown_connections 2022-07-20T02:32:35.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda40053a70 0x7fda40055f30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c138df0 0x7fda5c139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c1408c0 0x7fda5c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 --2- 172.21.15.26:0/585563179 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fda5c0670c0 0x7fda5c146830 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 >> 172.21.15.26:0/585563179 conn(0x7fda5c05f840 msgr2=0x7fda5c0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 shutdown_connections 2022-07-20T02:32:35.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.729+0000 7fda5bfff700 1 -- 172.21.15.26:0/585563179 wait complete. 2022-07-20T02:32:35.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 Processor -- start 2022-07-20T02:32:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 -- start start 2022-07-20T02:32:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5640061a20 0x7f56401432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 0x7f564012c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5640061e00 con 0x7f564012c530 2022-07-20T02:32:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5640147040 con 0x7f5640061a20 2022-07-20T02:32:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564866f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56401472e0 con 0x7f56401468c0 2022-07-20T02:32:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5640061a20 0x7f56401432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5640061a20 0x7f56401432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40046/0 (socket says 172.21.15.26:40046) 2022-07-20T02:32:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5646c0c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f564012c530 0x7f56400b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39328/0 (socket says 172.21.15.26:39328) 2022-07-20T02:32:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 -- 172.21.15.26:0/3687528947 learned_addr learned my addr 172.21.15.26:0/3687528947 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645c0a700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 0x7f564012c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 -- 172.21.15.26:0/3687528947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 msgr2=0x7f564012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 0x7f564012c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 -- 172.21.15.26:0/3687528947 >> v1:172.21.15.26:6789/0 conn(0x7f564012c530 legacy=0x7f56400b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645409700 1 -- 172.21.15.26:0/3687528947 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047809934 0 0) 0x7f5640061e00 con 0x7f564012c530 2022-07-20T02:32:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 -- 172.21.15.26:0/3687528947 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5640147580 con 0x7f5640061a20 2022-07-20T02:32:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645c0a700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 0x7f564012c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f564640b700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5640061a20 0x7f56401432e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f56300032f0 tx=0x7f5630009290).ready entity=mon.0 client_cookie=13c01347ff63c0b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645409700 1 -- 172.21.15.26:0/3687528947 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563000b8a0 con 0x7f5640061a20 2022-07-20T02:32:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645409700 1 -- 172.21.15.26:0/3687528947 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000ba00 con 0x7f5640061a20 2022-07-20T02:32:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.873+0000 7f5645409700 1 -- 172.21.15.26:0/3687528947 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563000bb70 con 0x7f5640061a20 2022-07-20T02:32:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- 172.21.15.26:0/3687528947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5640061a20 msgr2=0x7f56401432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5640061a20 0x7f56401432e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f56300032f0 tx=0x7f5630009290).stop 2022-07-20T02:32:35.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- 172.21.15.26:0/3687528947 shutdown_connections 2022-07-20T02:32:35.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f56401468c0 0x7f564012c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 --2- 172.21.15.26:0/3687528947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5640061a20 0x7f56401432e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- 172.21.15.26:0/3687528947 >> 172.21.15.26:0/3687528947 conn(0x7f564005faf0 msgr2=0x7f5640066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:35.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- 172.21.15.26:0/3687528947 shutdown_connections 2022-07-20T02:32:35.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- 172.21.15.26:0/3687528947 wait complete. 2022-07-20T02:32:35.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 Processor -- start 2022-07-20T02:32:35.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.877+0000 7f564866f700 1 -- start start 2022-07-20T02:32:35.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5640061a20 0x7f56400adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f564012c530 0x7f56400b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640155c80 con 0x7f56401468c0 2022-07-20T02:32:35.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640155f20 con 0x7f564012c530 2022-07-20T02:32:35.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56401561c0 con 0x7f5640061a20 2022-07-20T02:32:35.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40056/0 (socket says 172.21.15.26:40056) 2022-07-20T02:32:35.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 -- 172.21.15.26:0/3795391244 learned_addr learned my addr 172.21.15.26:0/3795391244 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:35.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564640b700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5640061a20 0x7f56400adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5645c0a700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f564012c530 0x7f56400b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 -- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5640061a20 msgr2=0x7f56400adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5640061a20 0x7f56400adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 -- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f564012c530 msgr2=0x7f56400b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f564012c530 0x7f56400b84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 -- 172.21.15.26:0/3795391244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009530 con 0x7f56401468c0 2022-07-20T02:32:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f5646c0c700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f5638000fe0 tx=0x7f5638007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56380070d0 con 0x7f56401468c0 2022-07-20T02:32:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638007230 con 0x7f56401468c0 2022-07-20T02:32:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56400b6290 con 0x7f56401468c0 2022-07-20T02:32:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5638005740 con 0x7f56401468c0 2022-07-20T02:32:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56400b9050 con 0x7f56401468c0 2022-07-20T02:32:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640146d20 con 0x7f56401468c0 2022-07-20T02:32:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.881+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f563801a060 con 0x7f56401468c0 2022-07-20T02:32:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.885+0000 7f563f7fe700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 0x7f5624055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.885+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f563806b770 con 0x7f56401468c0 2022-07-20T02:32:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.885+0000 7f564640b700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 0x7f5624055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.885+0000 7f564640b700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 0x7f5624055f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f563001a000 tx=0x7f5630016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:35.889+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5638042530 con 0x7f56401468c0 2022-07-20T02:32:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.053+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56400574e0 con 0x7f56401468c0 2022-07-20T02:32:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.053+0000 7f563f7fe700 1 -- 172.21.15.26:0/3795391244 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f5638018070 con 0x7f56401468c0 2022-07-20T02:32:36.070 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:36.071 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 msgr2=0x7f5624055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 0x7f5624055f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f563001a000 tx=0x7f5630016000).stop 2022-07-20T02:32:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 msgr2=0x7f56400b89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f5638000fe0 tx=0x7f5638007ad0).stop 2022-07-20T02:32:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 shutdown_connections 2022-07-20T02:32:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f56401468c0 0x7f56400b89c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5624053a70 0x7f5624055f30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f564012c530 0x7f56400b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 --2- 172.21.15.26:0/3795391244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5640061a20 0x7f56400adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 >> 172.21.15.26:0/3795391244 conn(0x7f564005faf0 msgr2=0x7f5640146030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 shutdown_connections 2022-07-20T02:32:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.057+0000 7f564866f700 1 -- 172.21.15.26:0/3795391244 wait complete. 2022-07-20T02:32:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:36.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 Processor -- start 2022-07-20T02:32:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 -- start start 2022-07-20T02:32:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8744061860 0x7f8744067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874405ff70 con 0x7f87440980d0 2022-07-20T02:32:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87440606e0 con 0x7f87440b3260 2022-07-20T02:32:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.177+0000 7f874a4ce700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440b3640 con 0x7f8744061860 2022-07-20T02:32:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8749ccd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f87440980d0 0x7f8744142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39342/0 (socket says 172.21.15.26:39342) 2022-07-20T02:32:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8749ccd700 1 -- 172.21.15.26:0/910899504 learned_addr learned my addr 172.21.15.26:0/910899504 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:36.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8748ccb700 1 --2- 172.21.15.26:0/910899504 >> v2:172.21.15.26:3300/0 conn(0x7f8744061860 0x7f8744067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8748ccb700 1 -- 172.21.15.26:0/910899504 >> v1:172.21.15.120:6789/0 conn(0x7f87440b3260 legacy=0x7f87440979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8748ccb700 1 -- 172.21.15.26:0/910899504 >> v1:172.21.15.26:6789/0 conn(0x7f87440980d0 legacy=0x7f8744142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8748ccb700 1 -- 172.21.15.26:0/910899504 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87440601a0 con 0x7f8744061860 2022-07-20T02:32:36.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f8748ccb700 1 --2- 172.21.15.26:0/910899504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8744061860 0x7f8744067900 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f8734002f70 tx=0x7f8734006df0).ready entity=mon.0 client_cookie=2ccf8a2b27b4aff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f873bfff700 1 -- 172.21.15.26:0/910899504 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873400d610 con 0x7f8744061860 2022-07-20T02:32:36.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f873bfff700 1 -- 172.21.15.26:0/910899504 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734000e90 con 0x7f8744061860 2022-07-20T02:32:36.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f873bfff700 1 -- 172.21.15.26:0/910899504 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873400d960 con 0x7f8744061860 2022-07-20T02:32:36.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- 172.21.15.26:0/910899504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8744061860 msgr2=0x7f8744067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 --2- 172.21.15.26:0/910899504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8744061860 0x7f8744067900 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f8734002f70 tx=0x7f8734006df0).stop 2022-07-20T02:32:36.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- 172.21.15.26:0/910899504 shutdown_connections 2022-07-20T02:32:36.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 --2- 172.21.15.26:0/910899504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8744061860 0x7f8744067900 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- 172.21.15.26:0/910899504 >> 172.21.15.26:0/910899504 conn(0x7f87440b67f0 msgr2=0x7f87440b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- 172.21.15.26:0/910899504 shutdown_connections 2022-07-20T02:32:36.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- 172.21.15.26:0/910899504 wait complete. 2022-07-20T02:32:36.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 Processor -- start 2022-07-20T02:32:36.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.181+0000 7f874a4ce700 1 -- start start 2022-07-20T02:32:36.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8744061860 0x7f87440aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87440b3260 0x7f87440998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87440ab740 con 0x7f87440980d0 2022-07-20T02:32:36.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744099de0 con 0x7f8744061860 2022-07-20T02:32:36.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8744145de0 con 0x7f87440b3260 2022-07-20T02:32:36.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40066/0 (socket says 172.21.15.26:40066) 2022-07-20T02:32:36.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 -- 172.21.15.26:0/1875045557 learned_addr learned my addr 172.21.15.26:0/1875045557 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:36.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8749ccd700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87440b3260 0x7f87440998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f87494cc700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8744061860 0x7f87440aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 -- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87440b3260 msgr2=0x7f87440998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87440b3260 0x7f87440998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 -- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8744061860 msgr2=0x7f87440aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8744061860 0x7f87440aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 -- 172.21.15.26:0/1875045557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873400b180 con 0x7f87440980d0 2022-07-20T02:32:36.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f8748ccb700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f8744099f20 tx=0x7f8734015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873400d990 con 0x7f87440980d0 2022-07-20T02:32:36.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734011b70 con 0x7f87440980d0 2022-07-20T02:32:36.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8744095630 con 0x7f87440980d0 2022-07-20T02:32:36.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873401e460 con 0x7f87440980d0 2022-07-20T02:32:36.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87440a95a0 con 0x7f87440980d0 2022-07-20T02:32:36.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8744095810 con 0x7f87440980d0 2022-07-20T02:32:36.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.185+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8734025030 con 0x7f87440980d0 2022-07-20T02:32:36.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.189+0000 7f873a7fc700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 0x7f872c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.189+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f873406d320 con 0x7f87440980d0 2022-07-20T02:32:36.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.189+0000 7f87494cc700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 0x7f872c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.189+0000 7f87494cc700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 0x7f872c055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f874000d000 tx=0x7f8740007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.193+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8744095810 con 0x7f87440980d0 2022-07-20T02:32:36.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87440957c0 con 0x7f87440980d0 2022-07-20T02:32:36.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f873a7fc700 1 -- 172.21.15.26:0/1875045557 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f8734023020 con 0x7f87440980d0 2022-07-20T02:32:36.328 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:36.328 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":87,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:35.696918+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:33.535657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:34.094152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:36.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 msgr2=0x7f872c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 0x7f872c055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f874000d000 tx=0x7f8740007b90).stop 2022-07-20T02:32:36.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 msgr2=0x7f8744099400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.317+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f8744099f20 tx=0x7f8734015000).stop 2022-07-20T02:32:36.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 shutdown_connections 2022-07-20T02:32:36.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f87440980d0 0x7f8744099400 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f872c053a70 0x7f872c055f30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f87440b3260 0x7f87440998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 --2- 172.21.15.26:0/1875045557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8744061860 0x7f87440aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 >> 172.21.15.26:0/1875045557 conn(0x7f87440b67f0 msgr2=0x7f8744141180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 shutdown_connections 2022-07-20T02:32:36.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.321+0000 7f874a4ce700 1 -- 172.21.15.26:0/1875045557 wait complete. 2022-07-20T02:32:36.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:32:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 Processor -- start 2022-07-20T02:32:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- start start 2022-07-20T02:32:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e405cfd0 0x7f73e4143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73e4147000 0x7f73e40b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e41474e0 con 0x7f73e4060f00 2022-07-20T02:32:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e405d3b0 con 0x7f73e4147000 2022-07-20T02:32:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e41431e0 con 0x7f73e405cfd0 2022-07-20T02:32:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73e4147000 0x7f73e40b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73e4147000 0x7f73e40b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40076/0 (socket says 172.21.15.26:40076) 2022-07-20T02:32:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 -- 172.21.15.26:0/3121701449 learned_addr learned my addr 172.21.15.26:0/3121701449 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e9f9e700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e405cfd0 0x7f73e4143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 -- 172.21.15.26:0/3121701449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e405cfd0 msgr2=0x7f73e4143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:36.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e405cfd0 0x7f73e4143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 -- 172.21.15.26:0/3121701449 >> v1:172.21.15.120:6789/0 conn(0x7f73e4060f00 legacy=0x7f73e413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e8f9c700 1 -- 172.21.15.26:0/3121701449 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764806961 0 0) 0x7f73e41474e0 con 0x7f73e4060f00 2022-07-20T02:32:36.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 -- 172.21.15.26:0/3121701449 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e4143480 con 0x7f73e4147000 2022-07-20T02:32:36.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e979d700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4147000 0x7f73e40b4730 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f73d4002f70 tx=0x7f73d4006eb0).ready entity=mon.0 client_cookie=c626014bb9a95e10 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e8f9c700 1 -- 172.21.15.26:0/3121701449 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73d400d570 con 0x7f73e4147000 2022-07-20T02:32:36.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e8f9c700 1 -- 172.21.15.26:0/3121701449 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d4000e90 con 0x7f73e4147000 2022-07-20T02:32:36.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73e8f9c700 1 -- 172.21.15.26:0/3121701449 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73d400d8c0 con 0x7f73e4147000 2022-07-20T02:32:36.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- 172.21.15.26:0/3121701449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4147000 msgr2=0x7f73e40b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4147000 0x7f73e40b4730 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f73d4002f70 tx=0x7f73d4006eb0).stop 2022-07-20T02:32:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 -- 172.21.15.26:0/3121701449 shutdown_connections 2022-07-20T02:32:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.465+0000 7f73eafa0700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4147000 0x7f73e40b4730 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 --2- 172.21.15.26:0/3121701449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e405cfd0 0x7f73e4143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- 172.21.15.26:0/3121701449 >> 172.21.15.26:0/3121701449 conn(0x7f73e40620c0 msgr2=0x7f73e405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- 172.21.15.26:0/3121701449 shutdown_connections 2022-07-20T02:32:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- 172.21.15.26:0/3121701449 wait complete. 2022-07-20T02:32:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 Processor -- start 2022-07-20T02:32:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- start start 2022-07-20T02:32:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73e4147000 0x7f73e4155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e4138250 0x7f73e41386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40af390 con 0x7f73e4060f00 2022-07-20T02:32:36.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e41563e0 con 0x7f73e4147000 2022-07-20T02:32:36.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4138bd0 con 0x7f73e4138250 2022-07-20T02:32:36.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40080/0 (socket says 172.21.15.26:40080) 2022-07-20T02:32:36.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 -- 172.21.15.26:0/1266741723 learned_addr learned my addr 172.21.15.26:0/1266741723 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:36.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e979d700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73e4147000 0x7f73e4155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73ea79f700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e4138250 0x7f73e41386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 -- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e4138250 msgr2=0x7f73e41386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e4138250 0x7f73e41386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 -- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73e4147000 msgr2=0x7f73e4155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73e4147000 0x7f73e4155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 -- 172.21.15.26:0/1266741723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d400b1e0 con 0x7f73e4060f00 2022-07-20T02:32:36.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73e9f9e700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f73e4145c60 tx=0x7f73e0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73e0010780 con 0x7f73e4060f00 2022-07-20T02:32:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e00108e0 con 0x7f73e4060f00 2022-07-20T02:32:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73e0010c40 con 0x7f73e4060f00 2022-07-20T02:32:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e40b3850 con 0x7f73e4060f00 2022-07-20T02:32:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.469+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40b0730 con 0x7f73e4060f00 2022-07-20T02:32:36.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e4145a80 con 0x7f73e4060f00 2022-07-20T02:32:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73e001c460 con 0x7f73e4060f00 2022-07-20T02:32:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73daffd700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 0x7f73cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f73e006c500 con 0x7f73e4060f00 2022-07-20T02:32:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73e979d700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 0x7f73cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.473+0000 7f73e979d700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 0x7f73cc055f60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f73d401c000 tx=0x7f73d4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.481+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73e00430d0 con 0x7f73e4060f00 2022-07-20T02:32:36.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.645+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f73e4143720 con 0x7f73e4060f00 2022-07-20T02:32:36.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.653+0000 7f73daffd700 1 -- 172.21.15.26:0/1266741723 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f73e00466f0 con 0x7f73e4060f00 2022-07-20T02:32:36.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 msgr2=0x7f73cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 0x7f73cc055f60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f73d401c000 tx=0x7f73d4015000).stop 2022-07-20T02:32:36.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 msgr2=0x7f73e40b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f73e4145c60 tx=0x7f73e0006df0).stop 2022-07-20T02:32:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 shutdown_connections 2022-07-20T02:32:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73cc053aa0 0x7f73cc055f60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73e4138250 0x7f73e41386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73e4147000 0x7f73e4155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 --2- 172.21.15.26:0/1266741723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73e4060f00 0x7f73e40b2eb0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 >> 172.21.15.26:0/1266741723 conn(0x7f73e405cfd0 msgr2=0x7f73e4140f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 shutdown_connections 2022-07-20T02:32:36.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.657+0000 7f73eafa0700 1 -- 172.21.15.26:0/1266741723 wait complete. 2022-07-20T02:32:36.683 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:32:36.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:32:36.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 Processor -- start 2022-07-20T02:32:36.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 -- start start 2022-07-20T02:32:36.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d600b2ab0 0x7f5d60061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d600af6d0 0x7f5d600afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d600aa1e0 con 0x7f5d600add30 2022-07-20T02:32:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d600ae110 con 0x7f5d600b2ab0 2022-07-20T02:32:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d667ca700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d600a9f40 con 0x7f5d600af6d0 2022-07-20T02:32:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.797+0000 7f5d64fc7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d600b2ab0 0x7f5d60061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d65fc9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d600af6d0 0x7f5d600afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5d600b2ab0 0x7f5d60061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40090/0 (socket says 172.21.15.26:40090) 2022-07-20T02:32:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 -- 172.21.15.26:0/658708835 learned_addr learned my addr 172.21.15.26:0/658708835 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 -- 172.21.15.26:0/658708835 >> v2:172.21.15.120:3300/0 conn(0x7f5d600af6d0 msgr2=0x7f5d600afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:36.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 --2- 172.21.15.26:0/658708835 >> v2:172.21.15.120:3300/0 conn(0x7f5d600af6d0 0x7f5d600afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 -- 172.21.15.26:0/658708835 >> v1:172.21.15.120:6789/0 conn(0x7f5d600add30 legacy=0x7f5d60060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 -- 172.21.15.26:0/658708835 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d600b1110 con 0x7f5d600b2ab0 2022-07-20T02:32:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d64fc7700 1 --2- 172.21.15.26:0/658708835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600b2ab0 0x7f5d60061060 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f5d50002f70 tx=0x7f5d50006eb0).ready entity=mon.0 client_cookie=25498009e3c488ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d4ffff700 1 -- 172.21.15.26:0/658708835 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5000d5c0 con 0x7f5d600b2ab0 2022-07-20T02:32:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d4ffff700 1 -- 172.21.15.26:0/658708835 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d50000e90 con 0x7f5d600b2ab0 2022-07-20T02:32:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d4ffff700 1 -- 172.21.15.26:0/658708835 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d5000d950 con 0x7f5d600b2ab0 2022-07-20T02:32:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 -- 172.21.15.26:0/658708835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600b2ab0 msgr2=0x7f5d60061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 --2- 172.21.15.26:0/658708835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600b2ab0 0x7f5d60061060 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f5d50002f70 tx=0x7f5d50006eb0).stop 2022-07-20T02:32:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 -- 172.21.15.26:0/658708835 shutdown_connections 2022-07-20T02:32:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 --2- 172.21.15.26:0/658708835 >> v2:172.21.15.120:3300/0 conn(0x7f5d600af6d0 0x7f5d600afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 --2- 172.21.15.26:0/658708835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600b2ab0 0x7f5d60061060 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 -- 172.21.15.26:0/658708835 >> 172.21.15.26:0/658708835 conn(0x7f5d600b3050 msgr2=0x7f5d600b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 -- 172.21.15.26:0/658708835 shutdown_connections 2022-07-20T02:32:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.801+0000 7f5d667ca700 1 -- 172.21.15.26:0/658708835 wait complete. 2022-07-20T02:32:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 Processor -- start 2022-07-20T02:32:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- start start 2022-07-20T02:32:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d600add30 0x7f5d6012f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d600b2ab0 0x7f5d60136e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6013b820 con 0x7f5d600af6d0 2022-07-20T02:32:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6013bac0 con 0x7f5d600b2ab0 2022-07-20T02:32:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6013a3c0 con 0x7f5d600add30 2022-07-20T02:32:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40096/0 (socket says 172.21.15.26:40096) 2022-07-20T02:32:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 -- 172.21.15.26:0/1508482886 learned_addr learned my addr 172.21.15.26:0/1508482886 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d65fc9700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d600b2ab0 0x7f5d60136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d657c8700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d600add30 0x7f5d6012f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 -- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d600add30 msgr2=0x7f5d6012f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d600add30 0x7f5d6012f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 -- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d600b2ab0 msgr2=0x7f5d60136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d600b2ab0 0x7f5d60136e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 -- 172.21.15.26:0/1508482886 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5000b1e0 con 0x7f5d600af6d0 2022-07-20T02:32:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d64fc7700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f5d50002f70 tx=0x7f5d50015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d50011860 con 0x7f5d600af6d0 2022-07-20T02:32:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d500119c0 con 0x7f5d600af6d0 2022-07-20T02:32:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6013a660 con 0x7f5d600af6d0 2022-07-20T02:32:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d50011d20 con 0x7f5d600af6d0 2022-07-20T02:32:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.805+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6012fbc0 con 0x7f5d600af6d0 2022-07-20T02:32:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6013bcc0 con 0x7f5d600af6d0 2022-07-20T02:32:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d5001a040 con 0x7f5d600af6d0 2022-07-20T02:32:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d4e7fc700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 0x7f5d40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5d5006d000 con 0x7f5d600af6d0 2022-07-20T02:32:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d657c8700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 0x7f5d40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.809+0000 7f5d657c8700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 0x7f5d40055f30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5d5c00d000 tx=0x7f5d5c007f10).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.821+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d50043d60 con 0x7f5d600af6d0 2022-07-20T02:32:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:36.993+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5d600b13b0 con 0x7f5d600af6d0 2022-07-20T02:32:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.001+0000 7f5d4e7fc700 1 -- 172.21.15.26:0/1508482886 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5d50023020 con 0x7f5d600af6d0 2022-07-20T02:32:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 msgr2=0x7f5d40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 0x7f5d40055f30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5d5c00d000 tx=0x7f5d5c007f10).stop 2022-07-20T02:32:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 msgr2=0x7f5d60136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f5d50002f70 tx=0x7f5d50015000).stop 2022-07-20T02:32:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 shutdown_connections 2022-07-20T02:32:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d600af6d0 0x7f5d60136910 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d40053a70 0x7f5d40055f30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d600b2ab0 0x7f5d60136e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 --2- 172.21.15.26:0/1508482886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d600add30 0x7f5d6012f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 >> 172.21.15.26:0/1508482886 conn(0x7f5d600b3050 msgr2=0x7f5d6013bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 shutdown_connections 2022-07-20T02:32:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.005+0000 7f5d667ca700 1 -- 172.21.15.26:0/1508482886 wait complete. 2022-07-20T02:32:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:32:37.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:37.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.133+0000 7fb934637700 1 Processor -- start 2022-07-20T02:32:37.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.133+0000 7fb934637700 1 -- start start 2022-07-20T02:32:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 0x7fb92c067480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb92c141000 0x7fb92c141470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92c144670 con 0x7fb92c0b49c0 2022-07-20T02:32:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0b44a0 con 0x7fb92c0b7920 2022-07-20T02:32:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0b4f40 con 0x7fb92c141000 2022-07-20T02:32:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9323d3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 0x7fb92c067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb92c141000 0x7fb92c141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb931bd2700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb92c0b49c0 0x7fb92c144d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:53594/0 (socket says 172.21.15.26:53594) 2022-07-20T02:32:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb931bd2700 1 -- 172.21.15.26:0/3488749838 learned_addr learned my addr 172.21.15.26:0/3488749838 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9323d3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 0x7fb92c067480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40104/0 (socket says 172.21.15.26:40104) 2022-07-20T02:32:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 -- 172.21.15.26:0/3488749838 >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 msgr2=0x7fb92c067480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 --2- 172.21.15.26:0/3488749838 >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 0x7fb92c067480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 -- 172.21.15.26:0/3488749838 >> v1:172.21.15.120:6789/0 conn(0x7fb92c0b49c0 legacy=0x7fb92c144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 -- 172.21.15.26:0/3488749838 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c1448a0 con 0x7fb92c141000 2022-07-20T02:32:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9313d1700 1 -- 172.21.15.26:0/3488749838 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817778165 0 0) 0x7fb92c144670 con 0x7fb92c0b49c0 2022-07-20T02:32:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb932bd4700 1 --2- 172.21.15.26:0/3488749838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c141000 0x7fb92c141470 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb924002f70 tx=0x7fb924006df0).ready entity=mon.1 client_cookie=b8d8041f78901a66 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9313d1700 1 -- 172.21.15.26:0/3488749838 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb92400d570 con 0x7fb92c141000 2022-07-20T02:32:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9313d1700 1 -- 172.21.15.26:0/3488749838 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924000e90 con 0x7fb92c141000 2022-07-20T02:32:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb9313d1700 1 -- 172.21.15.26:0/3488749838 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb92400d8c0 con 0x7fb92c141000 2022-07-20T02:32:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- 172.21.15.26:0/3488749838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c141000 msgr2=0x7fb92c141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 --2- 172.21.15.26:0/3488749838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c141000 0x7fb92c141470 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb924002f70 tx=0x7fb924006df0).stop 2022-07-20T02:32:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- 172.21.15.26:0/3488749838 shutdown_connections 2022-07-20T02:32:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 --2- 172.21.15.26:0/3488749838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c141000 0x7fb92c141470 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 --2- 172.21.15.26:0/3488749838 >> v2:172.21.15.26:3300/0 conn(0x7fb92c0b7920 0x7fb92c067480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- 172.21.15.26:0/3488749838 >> 172.21.15.26:0/3488749838 conn(0x7fb92c064b10 msgr2=0x7fb92c064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- 172.21.15.26:0/3488749838 shutdown_connections 2022-07-20T02:32:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.137+0000 7fb934637700 1 -- 172.21.15.26:0/3488749838 wait complete. 2022-07-20T02:32:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 Processor -- start 2022-07-20T02:32:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- start start 2022-07-20T02:32:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb92c0b49c0 0x7fb92c153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 0x7fb92c140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c154790 con 0x7fb92c141000 2022-07-20T02:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c150eb0 con 0x7fb92c0b7920 2022-07-20T02:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c151150 con 0x7fb92c0b49c0 2022-07-20T02:32:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb931bd2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 0x7fb92c140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb9323d3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb92c0b49c0 0x7fb92c153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40112/0 (socket says 172.21.15.26:40112) 2022-07-20T02:32:37.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb931bd2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 0x7fb92c140a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59800/0 (socket says 172.21.15.26:59800) 2022-07-20T02:32:37.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 -- 172.21.15.26:0/666095366 learned_addr learned my addr 172.21.15.26:0/666095366 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:37.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 -- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb92c0b49c0 msgr2=0x7fb92c153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb92c0b49c0 0x7fb92c153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 -- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 msgr2=0x7fb92c140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 0x7fb92c140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 -- 172.21.15.26:0/666095366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92400b180 con 0x7fb92c141000 2022-07-20T02:32:37.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb932bd4700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fb924002f70 tx=0x7fb924014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:37.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb924016070 con 0x7fb92c141000 2022-07-20T02:32:37.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92400de10 con 0x7fb92c141000 2022-07-20T02:32:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c0ae120 con 0x7fb92c141000 2022-07-20T02:32:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb924013e90 con 0x7fb92c141000 2022-07-20T02:32:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.141+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c0b6440 con 0x7fb92c141000 2022-07-20T02:32:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb924025030 con 0x7fb92c141000 2022-07-20T02:32:37.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c145450 con 0x7fb92c141000 2022-07-20T02:32:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb91f7fe700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 0x7fb90c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb9323d3700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 0x7fb90c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb92406d100 con 0x7fb92c141000 2022-07-20T02:32:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.145+0000 7fb9323d3700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 0x7fb90c055fc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb92000b000 tx=0x7fb920006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:37.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.153+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb924043db0 con 0x7fb92c141000 2022-07-20T02:32:37.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.313+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb92c0574e0 con 0x7fb92c141000 2022-07-20T02:32:37.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.313+0000 7fb91f7fe700 1 -- 172.21.15.26:0/666095366 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7fb924023090 con 0x7fb92c141000 2022-07-20T02:32:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:37.326 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 msgr2=0x7fb90c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 0x7fb90c055fc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb92000b000 tx=0x7fb920006df0).stop 2022-07-20T02:32:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 msgr2=0x7fb92c0b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fb924002f70 tx=0x7fb924014000).stop 2022-07-20T02:32:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 shutdown_connections 2022-07-20T02:32:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb92c141000 0x7fb92c0b5f50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb90c053b00 0x7fb90c055fc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb92c0b7920 0x7fb92c140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 --2- 172.21.15.26:0/666095366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb92c0b49c0 0x7fb92c153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 >> 172.21.15.26:0/666095366 conn(0x7fb92c064b10 msgr2=0x7fb92c13dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:37.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 shutdown_connections 2022-07-20T02:32:37.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.317+0000 7fb934637700 1 -- 172.21.15.26:0/666095366 wait complete. 2022-07-20T02:32:37.371 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:37.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:34.694079+0000', 'modified': '2022-07-20T02:32:35.719385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6699': {'gid': 6699, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2208502036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2208502036}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2208502036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:32:37.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:32:37.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:32:37.841 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6699,"want_state":"up:active","state":"up:active","rank_uptime":3.1317583939999998,"mdsmap_epoch":43,"osdmap_epoch":87,"osdmap_epoch_barrier":87,"uptime":7.5822848389999997} 2022-07-20T02:32:37.856 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6699, "mdsmap_epoch": 43, "osdmap_epoch": 87, "osdmap_epoch_barrier": 87, "rank_uptime": 3.131758394, "state": "up:active", "uptime": 7.582284839, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:32:37.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:37.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.957+0000 7f88b25a4700 1 Processor -- start 2022-07-20T02:32:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- start start 2022-07-20T02:32:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 0x7f88ac066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f88ac061050 0x7f88ac066f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac0b1bc0 con 0x7f88ac0b2920 2022-07-20T02:32:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ac14b040 con 0x7f88ac061050 2022-07-20T02:32:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0a8860 con 0x7f88ac1484a0 2022-07-20T02:32:37.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f88ac061050 0x7f88ac066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f88ac061050 0x7f88ac066f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40120/0 (socket says 172.21.15.26:40120) 2022-07-20T02:32:37.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 -- 172.21.15.26:0/82273879 learned_addr learned my addr 172.21.15.26:0/82273879 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:37.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88abfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f88ac0b2920 0x7f88ac147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39398/0 (socket says 172.21.15.26:39398) 2022-07-20T02:32:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88a3fff700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 0x7f88ac066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 -- 172.21.15.26:0/82273879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 msgr2=0x7f88ac066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:37.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 0x7f88ac066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 -- 172.21.15.26:0/82273879 >> v1:172.21.15.26:6789/0 conn(0x7f88ac0b2920 legacy=0x7f88ac147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 -- 172.21.15.26:0/82273879 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac147700 con 0x7f88ac061050 2022-07-20T02:32:37.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88a3fff700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 0x7f88ac066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:37.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b0b41700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac061050 0x7f88ac066f70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f88a40032f0 tx=0x7f88a4009290).ready entity=mon.0 client_cookie=bb420fd6ef36666b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:37.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88ab7fe700 1 -- 172.21.15.26:0/82273879 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a400b8a0 con 0x7f88ac061050 2022-07-20T02:32:37.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88ab7fe700 1 -- 172.21.15.26:0/82273879 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a400ba00 con 0x7f88ac061050 2022-07-20T02:32:37.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88ab7fe700 1 -- 172.21.15.26:0/82273879 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a400bb70 con 0x7f88ac061050 2022-07-20T02:32:37.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- 172.21.15.26:0/82273879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac061050 msgr2=0x7f88ac066f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac061050 0x7f88ac066f70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f88a40032f0 tx=0x7f88a4009290).stop 2022-07-20T02:32:37.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- 172.21.15.26:0/82273879 shutdown_connections 2022-07-20T02:32:37.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac061050 0x7f88ac066f70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 --2- 172.21.15.26:0/82273879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac1484a0 0x7f88ac066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- 172.21.15.26:0/82273879 >> 172.21.15.26:0/82273879 conn(0x7f88ac0b60a0 msgr2=0x7f88ac0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:37.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- 172.21.15.26:0/82273879 shutdown_connections 2022-07-20T02:32:37.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.961+0000 7f88b25a4700 1 -- 172.21.15.26:0/82273879 wait complete. 2022-07-20T02:32:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 Processor -- start 2022-07-20T02:32:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 -- start start 2022-07-20T02:32:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 0x7f88ac14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88ac1484a0 0x7f88ac156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac159df0 con 0x7f88ac0b2920 2022-07-20T02:32:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac15a020 con 0x7f88ac1484a0 2022-07-20T02:32:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.965+0000 7f88b25a4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac15a2c0 con 0x7f88ac061050 2022-07-20T02:32:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40124/0 (socket says 172.21.15.26:40124) 2022-07-20T02:32:37.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 -- 172.21.15.26:0/2302110140 learned_addr learned my addr 172.21.15.26:0/2302110140 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:37.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88b0b41700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88ac1484a0 0x7f88ac156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88abfff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 0x7f88ac14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:37.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 -- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 msgr2=0x7f88ac14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:37.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 0x7f88ac14c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:37.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 -- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88ac1484a0 msgr2=0x7f88ac156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88ac1484a0 0x7f88ac156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 -- 172.21.15.26:0/2302110140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4009530 con 0x7f88ac0b2920 2022-07-20T02:32:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88abfff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 0x7f88ac14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a3fff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f889400a000 tx=0x7f8894008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8894011780 con 0x7f88ac0b2920 2022-07-20T02:32:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940118e0 con 0x7f88ac0b2920 2022-07-20T02:32:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac0a9d00 con 0x7f88ac0b2920 2022-07-20T02:32:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8894011c40 con 0x7f88ac0b2920 2022-07-20T02:32:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac0ad8a0 con 0x7f88ac0b2920 2022-07-20T02:32:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ac14ada0 con 0x7f88ac0b2920 2022-07-20T02:32:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.969+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8894012420 con 0x7f88ac0b2920 2022-07-20T02:32:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.973+0000 7f88a9ffb700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 0x7f8890055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.973+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f889406bde0 con 0x7f88ac0b2920 2022-07-20T02:32:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.973+0000 7f88abfff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 0x7f8890055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.973+0000 7f88abfff700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 0x7f8890055ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f889c00b000 tx=0x7f889c0070e0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:37.977+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8894042b40 con 0x7f88ac0b2920 2022-07-20T02:32:38.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.105+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88ac0574e0 con 0x7f88ac0b2920 2022-07-20T02:32:38.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.105+0000 7f88a9ffb700 1 -- 172.21.15.26:0/2302110140 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f8894046160 con 0x7f88ac0b2920 2022-07-20T02:32:38.114 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:38.114 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":87,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:35.696918+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:33.535657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:34.094152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 msgr2=0x7f8890055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 0x7f8890055ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f889c00b000 tx=0x7f889c0070e0).stop 2022-07-20T02:32:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 msgr2=0x7f88ac156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f889400a000 tx=0x7f8894008040).stop 2022-07-20T02:32:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 shutdown_connections 2022-07-20T02:32:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88ac0b2920 0x7f88ac156820 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8890053a10 0x7f8890055ed0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88ac1484a0 0x7f88ac156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 --2- 172.21.15.26:0/2302110140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88ac061050 0x7f88ac14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 >> 172.21.15.26:0/2302110140 conn(0x7f88ac0b60a0 msgr2=0x7f88ac149d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 shutdown_connections 2022-07-20T02:32:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.109+0000 7f88b25a4700 1 -- 172.21.15.26:0/2302110140 wait complete. 2022-07-20T02:32:38.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:32:33.535657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:32:34.094152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:32:38.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:32:38.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.249+0000 7f25f8e52700 1 Processor -- start 2022-07-20T02:32:38.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.249+0000 7f25f8e52700 1 -- start start 2022-07-20T02:32:38.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 0x7f25f40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25f40a7fc0 0x7f25f40af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f40b62d0 con 0x7f25f40b5ef0 2022-07-20T02:32:38.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f414aac0 con 0x7f25f40a7fc0 2022-07-20T02:32:38.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f414ad60 con 0x7f25f4147be0 2022-07-20T02:32:38.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25f40b5ef0 0x7f25f41474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39410/0 (socket says 172.21.15.26:39410) 2022-07-20T02:32:38.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f37fe700 1 -- 172.21.15.26:0/3157385332 learned_addr learned my addr 172.21.15.26:0/3157385332 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:38.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 --2- 172.21.15.26:0/3157385332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 0x7f25f40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f3fff700 1 --2- 172.21.15.26:0/3157385332 >> v2:172.21.15.120:3300/0 conn(0x7f25f40a7fc0 0x7f25f40af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f27fc700 1 -- 172.21.15.26:0/3157385332 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542937198 0 0) 0x7f25f40b62d0 con 0x7f25f40b5ef0 2022-07-20T02:32:38.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f27fc700 1 -- 172.21.15.26:0/3157385332 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0003260 con 0x7f25f40b5ef0 2022-07-20T02:32:38.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 -- 172.21.15.26:0/3157385332 >> v2:172.21.15.120:3300/0 conn(0x7f25f40a7fc0 msgr2=0x7f25f40af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 --2- 172.21.15.26:0/3157385332 >> v2:172.21.15.120:3300/0 conn(0x7f25f40a7fc0 0x7f25f40af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 -- 172.21.15.26:0/3157385332 >> v1:172.21.15.26:6789/0 conn(0x7f25f40b5ef0 legacy=0x7f25f41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 -- 172.21.15.26:0/3157385332 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f414b000 con 0x7f25f4147be0 2022-07-20T02:32:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f2ffd700 1 --2- 172.21.15.26:0/3157385332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 0x7f25f40aee40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f25e4002f70 tx=0x7f25e4006df0).ready entity=mon.2 client_cookie=6932c920659b790c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f27fc700 1 -- 172.21.15.26:0/3157385332 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e400d4a0 con 0x7f25f4147be0 2022-07-20T02:32:38.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f27fc700 1 -- 172.21.15.26:0/3157385332 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e4000e90 con 0x7f25f4147be0 2022-07-20T02:32:38.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f27fc700 1 -- 172.21.15.26:0/3157385332 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e400d7f0 con 0x7f25f4147be0 2022-07-20T02:32:38.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- 172.21.15.26:0/3157385332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 msgr2=0x7f25f40aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 --2- 172.21.15.26:0/3157385332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 0x7f25f40aee40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f25e4002f70 tx=0x7f25e4006df0).stop 2022-07-20T02:32:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- 172.21.15.26:0/3157385332 shutdown_connections 2022-07-20T02:32:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 --2- 172.21.15.26:0/3157385332 >> v2:172.21.15.120:3300/0 conn(0x7f25f40a7fc0 0x7f25f40af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 --2- 172.21.15.26:0/3157385332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f4147be0 0x7f25f40aee40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- 172.21.15.26:0/3157385332 >> 172.21.15.26:0/3157385332 conn(0x7f25f4060310 msgr2=0x7f25f40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- 172.21.15.26:0/3157385332 shutdown_connections 2022-07-20T02:32:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.253+0000 7f25f8e52700 1 -- 172.21.15.26:0/3157385332 wait complete. 2022-07-20T02:32:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 Processor -- start 2022-07-20T02:32:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 -- start start 2022-07-20T02:32:38.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f40a7fc0 0x7f25f40b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f40b5ef0 0x7f25f4159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4064350 con 0x7f25f4147be0 2022-07-20T02:32:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4157220 con 0x7f25f40b5ef0 2022-07-20T02:32:38.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f41574c0 con 0x7f25f40a7fc0 2022-07-20T02:32:38.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40140/0 (socket says 172.21.15.26:40140) 2022-07-20T02:32:38.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f40a7fc0 0x7f25f40b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 -- 172.21.15.26:0/1697574450 learned_addr learned my addr 172.21.15.26:0/1697574450 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:38.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f2ffd700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f40b5ef0 0x7f25f4159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 -- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f40a7fc0 msgr2=0x7f25f40b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f40a7fc0 0x7f25f40b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 -- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f40b5ef0 msgr2=0x7f25f4159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f40b5ef0 0x7f25f4159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 -- 172.21.15.26:0/1697574450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e400b180 con 0x7f25f4147be0 2022-07-20T02:32:38.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f3fff700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f25ec006e00 tx=0x7f25ec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25ec00c640 con 0x7f25f4147be0 2022-07-20T02:32:38.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f40ab360 con 0x7f25f4147be0 2022-07-20T02:32:38.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec00c7a0 con 0x7f25f4147be0 2022-07-20T02:32:38.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.257+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25ec00cb00 con 0x7f25f4147be0 2022-07-20T02:32:38.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f40b4ed0 con 0x7f25f4147be0 2022-07-20T02:32:38.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f414b2a0 con 0x7f25f4147be0 2022-07-20T02:32:38.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f25ec022070 con 0x7f25f4147be0 2022-07-20T02:32:38.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f0ff9700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 0x7f25d4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f25ec06c6d0 con 0x7f25f4147be0 2022-07-20T02:32:38.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f37fe700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 0x7f25d4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.261+0000 7f25f37fe700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 0x7f25d4055ea0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f25e800d000 tx=0x7f25e8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.269+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25ec0433f0 con 0x7f25f4147be0 2022-07-20T02:32:38.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.393+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f25f40574e0 con 0x7f25f4147be0 2022-07-20T02:32:38.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f0ff9700 1 -- 172.21.15.26:0/1697574450 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v87) v1 ==== 110+0+10 (secure 0 0 0) 0x7f25ec046a10 con 0x7f25f4147be0 2022-07-20T02:32:38.412 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:32:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 msgr2=0x7f25d4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 0x7f25d4055ea0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f25e800d000 tx=0x7f25e8009040).stop 2022-07-20T02:32:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 msgr2=0x7f25f415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f25ec006e00 tx=0x7f25ec0091d0).stop 2022-07-20T02:32:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 shutdown_connections 2022-07-20T02:32:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25f4147be0 0x7f25f415a130 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f25d40539e0 0x7f25d4055ea0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25f40b5ef0 0x7f25f4159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 --2- 172.21.15.26:0/1697574450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25f40a7fc0 0x7f25f40b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 >> 172.21.15.26:0/1697574450 conn(0x7f25f4060310 msgr2=0x7f25f40aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 shutdown_connections 2022-07-20T02:32:38.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.397+0000 7f25f8e52700 1 -- 172.21.15.26:0/1697574450 wait complete. 2022-07-20T02:32:38.431 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:32:38.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 Processor -- start 2022-07-20T02:32:38.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 -- start start 2022-07-20T02:32:38.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc73005c8c0 0x7fc7300b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc7300afe40 0x7fc730141790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73005cca0 con 0x7fc7300b0da0 2022-07-20T02:32:38.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7300ae910 con 0x7fc73005c8c0 2022-07-20T02:32:38.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72f59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7300aefe0 con 0x7fc7300afe40 2022-07-20T02:32:38.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc73005c8c0 0x7fc7300b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc73005c8c0 0x7fc7300b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40144/0 (socket says 172.21.15.26:40144) 2022-07-20T02:32:38.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc725d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc7300b0da0 0x7fc7300ae200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39428/0 (socket says 172.21.15.26:39428) 2022-07-20T02:32:38.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 -- 172.21.15.26:0/3699845498 learned_addr learned my addr 172.21.15.26:0/3699845498 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:38.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72ed9d700 1 --2- 172.21.15.26:0/3699845498 >> v2:172.21.15.120:3300/0 conn(0x7fc7300afe40 0x7fc730141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 -- 172.21.15.26:0/3699845498 >> v2:172.21.15.120:3300/0 conn(0x7fc7300afe40 msgr2=0x7fc730141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:38.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 --2- 172.21.15.26:0/3699845498 >> v2:172.21.15.120:3300/0 conn(0x7fc7300afe40 0x7fc730141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 -- 172.21.15.26:0/3699845498 >> v1:172.21.15.26:6789/0 conn(0x7fc7300b0da0 legacy=0x7fc7300ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72dd9b700 1 -- 172.21.15.26:0/3699845498 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155543145 0 0) 0x7fc73005cca0 con 0x7fc7300b0da0 2022-07-20T02:32:38.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 -- 172.21.15.26:0/3699845498 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300acac0 con 0x7fc73005c8c0 2022-07-20T02:32:38.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.541+0000 7fc72e59c700 1 --2- 172.21.15.26:0/3699845498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc73005c8c0 0x7fc7300b04e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fc720002f70 tx=0x7fc720006eb0).ready entity=mon.0 client_cookie=5f0d31573a14c76 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72dd9b700 1 -- 172.21.15.26:0/3699845498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc72000d5c0 con 0x7fc73005c8c0 2022-07-20T02:32:38.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72dd9b700 1 -- 172.21.15.26:0/3699845498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc720000e90 con 0x7fc73005c8c0 2022-07-20T02:32:38.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72dd9b700 1 -- 172.21.15.26:0/3699845498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc72000d910 con 0x7fc73005c8c0 2022-07-20T02:32:38.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- 172.21.15.26:0/3699845498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc73005c8c0 msgr2=0x7fc7300b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- 172.21.15.26:0/3699845498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc73005c8c0 0x7fc7300b04e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fc720002f70 tx=0x7fc720006eb0).stop 2022-07-20T02:32:38.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- 172.21.15.26:0/3699845498 shutdown_connections 2022-07-20T02:32:38.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- 172.21.15.26:0/3699845498 >> v2:172.21.15.120:3300/0 conn(0x7fc7300afe40 0x7fc730141790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- 172.21.15.26:0/3699845498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc73005c8c0 0x7fc7300b04e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- 172.21.15.26:0/3699845498 >> 172.21.15.26:0/3699845498 conn(0x7fc730060310 msgr2=0x7fc7300b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- 172.21.15.26:0/3699845498 shutdown_connections 2022-07-20T02:32:38.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- 172.21.15.26:0/3699845498 wait complete. 2022-07-20T02:32:38.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 Processor -- start 2022-07-20T02:32:38.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- start start 2022-07-20T02:32:38.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc73005c8c0 0x7fc73013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7300b0da0 0x7fc7300652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73013d4f0 con 0x7fc7300afe40 2022-07-20T02:32:38.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc730063e70 con 0x7fc7300b0da0 2022-07-20T02:32:38.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72f59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc730063930 con 0x7fc73005c8c0 2022-07-20T02:32:38.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc725d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc725d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40152/0 (socket says 172.21.15.26:40152) 2022-07-20T02:32:38.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc725d9b700 1 -- 172.21.15.26:0/1118558301 learned_addr learned my addr 172.21.15.26:0/1118558301 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:38.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.545+0000 7fc72e59c700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc73005c8c0 0x7fc73013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc72ed9d700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7300b0da0 0x7fc7300652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 -- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc73005c8c0 msgr2=0x7fc73013cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc73005c8c0 0x7fc73013cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 -- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7300b0da0 msgr2=0x7fc7300652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7300b0da0 0x7fc7300652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 -- 172.21.15.26:0/1118558301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72000b1e0 con 0x7fc7300afe40 2022-07-20T02:32:38.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc725d9b700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fc730064000 tx=0x7fc718007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7180070d0 con 0x7fc7300afe40 2022-07-20T02:32:38.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc718007230 con 0x7fc7300afe40 2022-07-20T02:32:38.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc730063bd0 con 0x7fc7300afe40 2022-07-20T02:32:38.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc718005740 con 0x7fc7300afe40 2022-07-20T02:32:38.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc730065db0 con 0x7fc7300afe40 2022-07-20T02:32:38.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc730065080 con 0x7fc7300afe40 2022-07-20T02:32:38.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.549+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc71801a060 con 0x7fc7300afe40 2022-07-20T02:32:38.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.553+0000 7fc727fff700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 0x7fc714055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.553+0000 7fc72e59c700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 0x7fc714055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.553+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc71806b890 con 0x7fc7300afe40 2022-07-20T02:32:38.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.553+0000 7fc72e59c700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 0x7fc714055f30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fc72001c000 tx=0x7fc720015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.557+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc718042580 con 0x7fc7300afe40 2022-07-20T02:32:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.681+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc7300574e0 con 0x7fc7300afe40 2022-07-20T02:32:38.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.681+0000 7fc727fff700 1 -- 172.21.15.26:0/1118558301 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc718018020 con 0x7fc7300afe40 2022-07-20T02:32:38.688 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 msgr2=0x7fc714055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 0x7fc714055f30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fc72001c000 tx=0x7fc720015000).stop 2022-07-20T02:32:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 msgr2=0x7fc7300642d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fc730064000 tx=0x7fc718007b40).stop 2022-07-20T02:32:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 shutdown_connections 2022-07-20T02:32:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7300afe40 0x7fc7300642d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc714053a70 0x7fc714055f30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7300b0da0 0x7fc7300652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 --2- 172.21.15.26:0/1118558301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc73005c8c0 0x7fc73013cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 >> 172.21.15.26:0/1118558301 conn(0x7fc730060310 msgr2=0x7fc7300b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 shutdown_connections 2022-07-20T02:32:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.685+0000 7fc72f59e700 1 -- 172.21.15.26:0/1118558301 wait complete. 2022-07-20T02:32:38.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:32:38.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.817+0000 7fd64e727700 1 Processor -- start 2022-07-20T02:32:38.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.817+0000 7fd64e727700 1 -- start start 2022-07-20T02:32:38.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6480add30 0x7fd6480b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6480af6d0 0x7fd6480afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6480aa1e0 con 0x7fd648060950 2022-07-20T02:32:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480ae110 con 0x7fd6480add30 2022-07-20T02:32:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480a9f40 con 0x7fd6480af6d0 2022-07-20T02:32:38.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6480add30 0x7fd6480b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd6480add30 0x7fd6480b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40158/0 (socket says 172.21.15.26:40158) 2022-07-20T02:32:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 -- 172.21.15.26:0/1685037606 learned_addr learned my addr 172.21.15.26:0/1685037606 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:38.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64df26700 1 --2- 172.21.15.26:0/1685037606 >> v2:172.21.15.120:3300/0 conn(0x7fd6480af6d0 0x7fd6480afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 -- 172.21.15.26:0/1685037606 >> v2:172.21.15.120:3300/0 conn(0x7fd6480af6d0 msgr2=0x7fd6480afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 --2- 172.21.15.26:0/1685037606 >> v2:172.21.15.120:3300/0 conn(0x7fd6480af6d0 0x7fd6480afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 -- 172.21.15.26:0/1685037606 >> v1:172.21.15.120:6789/0 conn(0x7fd648060950 legacy=0x7fd648060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 -- 172.21.15.26:0/1685037606 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480b1110 con 0x7fd6480add30 2022-07-20T02:32:38.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64d725700 1 --2- 172.21.15.26:0/1685037606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6480add30 0x7fd6480b2ab0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fd644002f70 tx=0x7fd644006f40).ready entity=mon.0 client_cookie=1b91e761a1946643 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd63bfff700 1 -- 172.21.15.26:0/1685037606 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64400da80 con 0x7fd6480add30 2022-07-20T02:32:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd63bfff700 1 -- 172.21.15.26:0/1685037606 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd644000e90 con 0x7fd6480add30 2022-07-20T02:32:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd63bfff700 1 -- 172.21.15.26:0/1685037606 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64400de10 con 0x7fd6480add30 2022-07-20T02:32:38.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- 172.21.15.26:0/1685037606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6480add30 msgr2=0x7fd6480b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 --2- 172.21.15.26:0/1685037606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6480add30 0x7fd6480b2ab0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fd644002f70 tx=0x7fd644006f40).stop 2022-07-20T02:32:38.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- 172.21.15.26:0/1685037606 shutdown_connections 2022-07-20T02:32:38.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 --2- 172.21.15.26:0/1685037606 >> v2:172.21.15.120:3300/0 conn(0x7fd6480af6d0 0x7fd6480afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 --2- 172.21.15.26:0/1685037606 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6480add30 0x7fd6480b2ab0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- 172.21.15.26:0/1685037606 >> 172.21.15.26:0/1685037606 conn(0x7fd6480b3050 msgr2=0x7fd6480b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- 172.21.15.26:0/1685037606 shutdown_connections 2022-07-20T02:32:38.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- 172.21.15.26:0/1685037606 wait complete. 2022-07-20T02:32:38.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 Processor -- start 2022-07-20T02:32:38.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.821+0000 7fd64e727700 1 -- start start 2022-07-20T02:32:38.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 0x7fd648064f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6480af6d0 0x7fd648065430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648140ab0 con 0x7fd648060950 2022-07-20T02:32:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64813fd10 con 0x7fd6480af6d0 2022-07-20T02:32:38.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64813ff40 con 0x7fd6480add30 2022-07-20T02:32:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40164/0 (socket says 172.21.15.26:40164) 2022-07-20T02:32:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 -- 172.21.15.26:0/1939478811 learned_addr learned my addr 172.21.15.26:0/1939478811 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:38.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64df26700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6480af6d0 0x7fd648065430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64cf24700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 0x7fd648064f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 -- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 msgr2=0x7fd648064f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 0x7fd648064f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 -- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6480af6d0 msgr2=0x7fd648065430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6480af6d0 0x7fd648065430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 -- 172.21.15.26:0/1939478811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64400b460 con 0x7fd648060950 2022-07-20T02:32:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64cf24700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 0x7fd648064f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64d725700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fd644002f70 tx=0x7fd6480ae860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64401d070 con 0x7fd648060950 2022-07-20T02:32:38.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd644018610 con 0x7fd648060950 2022-07-20T02:32:38.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6480b60d0 con 0x7fd648060950 2022-07-20T02:32:38.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd644018970 con 0x7fd648060950 2022-07-20T02:32:38.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6480649b0 con 0x7fd648060950 2022-07-20T02:32:38.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd644018ad0 con 0x7fd648060950 2022-07-20T02:32:38.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.825+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64813f060 con 0x7fd648060950 2022-07-20T02:32:38.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.829+0000 7fd63a7fc700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 0x7fd628055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.829+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd64406d210 con 0x7fd648060950 2022-07-20T02:32:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.829+0000 7fd64cf24700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 0x7fd628055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.829+0000 7fd64cf24700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 0x7fd628055ff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd63c008000 tx=0x7fd63c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.833+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd644043fd0 con 0x7fd648060950 2022-07-20T02:32:38.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd648064ce0 con 0x7fd648060950 2022-07-20T02:32:38.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd63a7fc700 1 -- 172.21.15.26:0/1939478811 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd6440475f0 con 0x7fd648060950 2022-07-20T02:32:38.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:38.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 msgr2=0x7fd628055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 0x7fd628055ff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd63c008000 tx=0x7fd63c009040).stop 2022-07-20T02:32:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 msgr2=0x7fd6480b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.957+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fd644002f70 tx=0x7fd6480ae860).stop 2022-07-20T02:32:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 shutdown_connections 2022-07-20T02:32:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd628053b30 0x7fd628055ff0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6480af6d0 0x7fd648065430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6480add30 0x7fd648064f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 --2- 172.21.15.26:0/1939478811 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd648060950 0x7fd6480b5be0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 >> 172.21.15.26:0/1939478811 conn(0x7fd6480b3050 msgr2=0x7fd6480b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:38.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 shutdown_connections 2022-07-20T02:32:38.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:38.961+0000 7fd64e727700 1 -- 172.21.15.26:0/1939478811 wait complete. 2022-07-20T02:32:38.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:32:39.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 Processor -- start 2022-07-20T02:32:39.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 -- start start 2022-07-20T02:32:39.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047c061c80 con 0x7f047c143f80 2022-07-20T02:32:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047c0b2810 con 0x7f047c0618a0 2022-07-20T02:32:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0b2ab0 con 0x7f047c0b22d0 2022-07-20T02:32:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047a7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f047c143f80 0x7f047c0b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39454/0 (socket says 172.21.15.26:39454) 2022-07-20T02:32:39.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.113+0000 7f047a7fc700 1 -- 172.21.15.26:0/1555136458 learned_addr learned my addr 172.21.15.26:0/1555136458 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:39.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047b7fe700 1 --2- 172.21.15.26:0/1555136458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294724394 0 0) 0x7f047c061c80 con 0x7f047c143f80 2022-07-20T02:32:39.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c003260 con 0x7f047c143f80 2022-07-20T02:32:39.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425792759 0 0) 0x7f047c0b2810 con 0x7f047c0618a0 2022-07-20T02:32:39.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f047c061c80 con 0x7f047c0618a0 2022-07-20T02:32:39.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256888814 0 0) 0x7f045c003260 con 0x7f047c143f80 2022-07-20T02:32:39.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047b7fe700 1 -- 172.21.15.26:0/1555136458 >> v1:172.21.15.120:6789/0 conn(0x7f047c0618a0 legacy=0x7f047c143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047b7fe700 1 -- 172.21.15.26:0/1555136458 >> v1:172.21.15.26:6789/0 conn(0x7f047c143f80 legacy=0x7f047c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047b7fe700 1 -- 172.21.15.26:0/1555136458 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c0b2d50 con 0x7f047c0b22d0 2022-07-20T02:32:39.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047b7fe700 1 --2- 172.21.15.26:0/1555136458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c147250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0464001800 tx=0x7f04740091d0).ready entity=mon.2 client_cookie=bb283633ad5f1433 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0464001660 con 0x7f047c0b22d0 2022-07-20T02:32:39.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047400f510 con 0x7f047c0b22d0 2022-07-20T02:32:39.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f0479ffb700 1 -- 172.21.15.26:0/1555136458 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047400f680 con 0x7f047c0b22d0 2022-07-20T02:32:39.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 -- 172.21.15.26:0/1555136458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 msgr2=0x7f047c147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 --2- 172.21.15.26:0/1555136458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c147250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0464001800 tx=0x7f04740091d0).stop 2022-07-20T02:32:39.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 -- 172.21.15.26:0/1555136458 shutdown_connections 2022-07-20T02:32:39.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 --2- 172.21.15.26:0/1555136458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c147250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 -- 172.21.15.26:0/1555136458 >> 172.21.15.26:0/1555136458 conn(0x7f047c066f70 msgr2=0x7f047c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:39.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 -- 172.21.15.26:0/1555136458 shutdown_connections 2022-07-20T02:32:39.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.117+0000 7f047bfff700 1 -- 172.21.15.26:0/1555136458 wait complete. 2022-07-20T02:32:39.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 Processor -- start 2022-07-20T02:32:39.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- start start 2022-07-20T02:32:39.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c13e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f047c143f80 0x7f047c146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0af0a0 con 0x7f047c0618a0 2022-07-20T02:32:39.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c153180 con 0x7f047c143f80 2022-07-20T02:32:39.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c153420 con 0x7f047c0b22d0 2022-07-20T02:32:39.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40178/0 (socket says 172.21.15.26:40178) 2022-07-20T02:32:39.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 -- 172.21.15.26:0/2298350665 learned_addr learned my addr 172.21.15.26:0/2298350665 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:39.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047b7fe700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f047c143f80 0x7f047c146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047a7fc700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c13e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 -- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 msgr2=0x7f047c13e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c13e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 -- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f047c143f80 msgr2=0x7f047c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f047c143f80 0x7f047c146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 -- 172.21.15.26:0/2298350665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009470 con 0x7f047c0618a0 2022-07-20T02:32:39.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047a7fc700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c13e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:39.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047affd700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f046c001110 tx=0x7f046c007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f046c00ce50 con 0x7f047c0618a0 2022-07-20T02:32:39.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c00f470 con 0x7f047c0618a0 2022-07-20T02:32:39.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f046c00f7d0 con 0x7f047c0618a0 2022-07-20T02:32:39.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c12d4b0 con 0x7f047c0618a0 2022-07-20T02:32:39.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.121+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c0b07f0 con 0x7f047c0618a0 2022-07-20T02:32:39.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c0af260 con 0x7f047c0618a0 2022-07-20T02:32:39.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f046c019030 con 0x7f047c0618a0 2022-07-20T02:32:39.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f0473fff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 0x7f0460055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f046c06b500 con 0x7f047c0618a0 2022-07-20T02:32:39.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f047a7fc700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 0x7f0460055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.125+0000 7f047a7fc700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 0x7f0460055ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f046400b000 tx=0x7f0464001950).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.137+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f046c0422c0 con 0x7f047c0618a0 2022-07-20T02:32:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f047c152fc0 con 0x7f047c0618a0 2022-07-20T02:32:39.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f0473fff700 1 -- 172.21.15.26:0/2298350665 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+10 (secure 0 0 0) 0x7f046c0458e0 con 0x7f047c0618a0 2022-07-20T02:32:39.272 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:39.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 msgr2=0x7f0460055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 0x7f0460055ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f046400b000 tx=0x7f0464001950).stop 2022-07-20T02:32:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 msgr2=0x7f047c12cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f046c001110 tx=0x7f046c007b60).stop 2022-07-20T02:32:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 shutdown_connections 2022-07-20T02:32:39.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0460053a10 0x7f0460055ed0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f047c143f80 0x7f047c146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f047c0b22d0 0x7f047c13e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 --2- 172.21.15.26:0/2298350665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f047c0618a0 0x7f047c12cfc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 >> 172.21.15.26:0/2298350665 conn(0x7f047c066f70 msgr2=0x7f047c146080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:39.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 shutdown_connections 2022-07-20T02:32:39.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.265+0000 7f047bfff700 1 -- 172.21.15.26:0/2298350665 wait complete. 2022-07-20T02:32:39.291 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:32:39.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:39.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.397+0000 7f0540d50700 1 Processor -- start 2022-07-20T02:32:39.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.397+0000 7f0540d50700 1 -- start start 2022-07-20T02:32:39.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 0x7f053c060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c067dc0 con 0x7f053c0a95a0 2022-07-20T02:32:39.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c0b3640 con 0x7f053c0b3260 2022-07-20T02:32:39.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0615c0 con 0x7f053c095b10 2022-07-20T02:32:39.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053b7fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 0x7f053c060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053b7fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 0x7f053c060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:59878/0 (socket says 172.21.15.26:59878) 2022-07-20T02:32:39.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053b7fe700 1 -- 172.21.15.26:0/1858468693 learned_addr learned my addr 172.21.15.26:0/1858468693 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:39.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0533fff700 1 --1- 172.21.15.26:0/1858468693 >> v1:172.21.15.120:6789/0 conn(0x7f053c0a95a0 0x7f053c0676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:53680/0 (socket says 172.21.15.26:53680) 2022-07-20T02:32:39.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 -- 172.21.15.26:0/1858468693 >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 msgr2=0x7f053c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:39.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 --2- 172.21.15.26:0/1858468693 >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 0x7f053c060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 -- 172.21.15.26:0/1858468693 >> v1:172.21.15.120:6789/0 conn(0x7f053c0a95a0 legacy=0x7f053c0676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 -- 172.21.15.26:0/1858468693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c061860 con 0x7f053c095b10 2022-07-20T02:32:39.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053bfff700 1 --2- 172.21.15.26:0/1858468693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c095f50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f05340032f0 tx=0x7f05340091d0).ready entity=mon.2 client_cookie=9128beafcf5d8f51 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053affd700 1 -- 172.21.15.26:0/1858468693 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f053400b6c0 con 0x7f053c095b10 2022-07-20T02:32:39.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053affd700 1 -- 172.21.15.26:0/1858468693 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053400b820 con 0x7f053c095b10 2022-07-20T02:32:39.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f053affd700 1 -- 172.21.15.26:0/1858468693 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f053400b990 con 0x7f053c095b10 2022-07-20T02:32:39.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- 172.21.15.26:0/1858468693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 msgr2=0x7f053c095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 --2- 172.21.15.26:0/1858468693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c095f50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f05340032f0 tx=0x7f05340091d0).stop 2022-07-20T02:32:39.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- 172.21.15.26:0/1858468693 shutdown_connections 2022-07-20T02:32:39.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 --2- 172.21.15.26:0/1858468693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c095f50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 --2- 172.21.15.26:0/1858468693 >> v2:172.21.15.120:3300/0 conn(0x7f053c0b3260 0x7f053c060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- 172.21.15.26:0/1858468693 >> 172.21.15.26:0/1858468693 conn(0x7f053c0b67f0 msgr2=0x7f053c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:39.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- 172.21.15.26:0/1858468693 shutdown_connections 2022-07-20T02:32:39.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.401+0000 7f0540d50700 1 -- 172.21.15.26:0/1858468693 wait complete. 2022-07-20T02:32:39.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 Processor -- start 2022-07-20T02:32:39.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- start start 2022-07-20T02:32:39.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c092820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 0x7f053c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1538f0 con 0x7f053c0a95a0 2022-07-20T02:32:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0aa420 con 0x7f053c0b3260 2022-07-20T02:32:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0aa6c0 con 0x7f053c095b10 2022-07-20T02:32:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40192/0 (socket says 172.21.15.26:40192) 2022-07-20T02:32:39.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 -- 172.21.15.26:0/1411421117 learned_addr learned my addr 172.21.15.26:0/1411421117 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:39.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f053b7fe700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c092820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f053bfff700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 0x7f053c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 -- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 msgr2=0x7f053c092820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c092820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 -- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 msgr2=0x7f053c154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 0x7f053c154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 -- 172.21.15.26:0/1411421117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534009470 con 0x7f053c0a95a0 2022-07-20T02:32:39.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f053bfff700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 0x7f053c154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:39.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0533fff700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f053c069b90 tx=0x7f0524007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f052400ce30 con 0x7f053c0a95a0 2022-07-20T02:32:39.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05240072e0 con 0x7f053c0a95a0 2022-07-20T02:32:39.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c0aa960 con 0x7f053c0a95a0 2022-07-20T02:32:39.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0524007450 con 0x7f053c0a95a0 2022-07-20T02:32:39.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.405+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c093590 con 0x7f053c0a95a0 2022-07-20T02:32:39.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c061b00 con 0x7f053c0a95a0 2022-07-20T02:32:39.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f052401a070 con 0x7f053c0a95a0 2022-07-20T02:32:39.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f05397fa700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 0x7f0520055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:39.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f052406b120 con 0x7f053c0a95a0 2022-07-20T02:32:39.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f053b7fe700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 0x7f0520055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:39.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.409+0000 7f053b7fe700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 0x7f0520055fc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f052c008000 tx=0x7f052c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:39.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.413+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0524041e80 con 0x7f053c0a95a0 2022-07-20T02:32:39.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.581+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053c0574e0 con 0x7f053c0a95a0 2022-07-20T02:32:39.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.581+0000 7f05397fa700 1 -- 172.21.15.26:0/1411421117 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f05240454a0 con 0x7f053c0a95a0 2022-07-20T02:32:39.592 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:39.592 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:39.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 msgr2=0x7f0520055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 0x7f0520055fc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f052c008000 tx=0x7f052c008040).stop 2022-07-20T02:32:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 msgr2=0x7f053c153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f053c069b90 tx=0x7f0524007c40).stop 2022-07-20T02:32:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 shutdown_connections 2022-07-20T02:32:39.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f053c0a95a0 0x7f053c153d50 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0520053b00 0x7f0520055fc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f053c0b3260 0x7f053c154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 --2- 172.21.15.26:0/1411421117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053c095b10 0x7f053c092820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 >> 172.21.15.26:0/1411421117 conn(0x7f053c0b67f0 msgr2=0x7f053c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 shutdown_connections 2022-07-20T02:32:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:39.585+0000 7f0540d50700 1 -- 172.21.15.26:0/1411421117 wait complete. 2022-07-20T02:32:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:39.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:34.694079+0000', 'modified': '2022-07-20T02:32:35.719385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6699': {'gid': 6699, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2208502036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2208502036}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2208502036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:32:39.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:32:39.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:32:40.033 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6699,"want_state":"up:active","state":"up:active","rank_uptime":5.3261550509999998,"mdsmap_epoch":43,"osdmap_epoch":87,"osdmap_epoch_barrier":87,"uptime":9.7766895770000009} 2022-07-20T02:32:40.033 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6699, "mdsmap_epoch": 43, "osdmap_epoch": 87, "osdmap_epoch_barrier": 87, "rank_uptime": 5.326155051, "state": "up:active", "uptime": 9.776689577, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:32:40.034 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:32:40.035 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:32:40.035 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:32:40.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.036 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:40.047 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:32:40.065 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:40.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:40.085 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:40.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.097 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:32:40.145 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:32:40.147 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:40.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:32:40.170 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:40.171 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:40.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:32:40.224+0000 7efec8089080 -1 init, newargv = 0x55e2b9c5cba0 newargc=9 2022-07-20T02:32:40.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22987]: starting ceph client 2022-07-20T02:32:40.234 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:40.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.238 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:32:40.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22987]: starting fuse 2022-07-20T02:32:40.254 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:32:40.255 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:32:40.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.256 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:32:40.258 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:32:40.259 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:32:40.259 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:32:40.260 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:32:40.261 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:40.262 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:32:40.263 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:32:40.264 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:32:40.265 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:40.265 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:32:40.267 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:32:40.268 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:32:40.269 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:32:40.270 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:40.270 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:32:40.271 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:32:40.272 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:32:40.272 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:32:40.273 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:32:40.274 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:32:40.275 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:32:40.276 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:32:40.373 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.22987.asok 2022-07-20T02:32:40.373 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22987.asok 2022-07-20T02:32:40.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22987.asok status 2022-07-20T02:32:40.478 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:32:40.478 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:32:40.478 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:32:40.479 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:32:40.479 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:32:40.479 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:32:40.479 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:32:40.480 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "22987", 2022-07-20T02:32:40.480 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:32:40.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:40.480 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:32:40.481 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:32:40.482 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 6715, 2022-07-20T02:32:40.482 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:32:40.483 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:32:40.483 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:32:40.484 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 6715 2022-07-20T02:32:40.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:40.487 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:32:40.487 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:32:40.489 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:32:40.489 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2710963461 2022-07-20T02:32:40.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:32:40.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:40.490 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:32:40.490 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:32:40.490 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:32:40.491 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2710963461 2022-07-20T02:32:40.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:32:40.491 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.6715 172.21.15.159:0/2710963461", 2022-07-20T02:32:40.491 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/2710963461", 2022-07-20T02:32:40.491 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:32:40.492 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 43, 2022-07-20T02:32:40.492 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 87, 2022-07-20T02:32:40.492 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:32:40.492 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:32:40.493 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:32:40.493 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:32:40.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.509 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:40.510 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:40.535 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:32:40.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:40.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:40.537 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:40.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:40.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.681+0000 7f0bd9376700 1 Processor -- start 2022-07-20T02:32:40.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.681+0000 7f0bd9376700 1 -- start start 2022-07-20T02:32:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 0x7f0bd40af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd40b62d0 con 0x7f0bd40b5ef0 2022-07-20T02:32:40.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd414aac0 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd414ad60 con 0x7f0bd4147be0 2022-07-20T02:32:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd8b75700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 0x7f0bd40af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd8b75700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 0x7f0bd40af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40204/0 (socket says 172.21.15.26:40204) 2022-07-20T02:32:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd8b75700 1 -- 172.21.15.26:0/3476718720 learned_addr learned my addr 172.21.15.26:0/3476718720 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0bd40b5ef0 0x7f0bd41474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39482/0 (socket says 172.21.15.26:39482) 2022-07-20T02:32:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 --2- 172.21.15.26:0/3476718720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 -- 172.21.15.26:0/3476718720 >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 msgr2=0x7f0bd40af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 --2- 172.21.15.26:0/3476718720 >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 0x7f0bd40af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 -- 172.21.15.26:0/3476718720 >> v1:172.21.15.26:6789/0 conn(0x7f0bd40b5ef0 legacy=0x7f0bd41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 -- 172.21.15.26:0/3476718720 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd414b000 con 0x7f0bd4147be0 2022-07-20T02:32:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bcb7fe700 1 --2- 172.21.15.26:0/3476718720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40aee40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0bc4002f70 tx=0x7f0bc4006df0).ready entity=mon.2 client_cookie=c638194e812dd801 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd37fe700 1 -- 172.21.15.26:0/3476718720 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bc400d570 con 0x7f0bd4147be0 2022-07-20T02:32:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd37fe700 1 -- 172.21.15.26:0/3476718720 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4000e90 con 0x7f0bd4147be0 2022-07-20T02:32:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd37fe700 1 -- 172.21.15.26:0/3476718720 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bc400d8c0 con 0x7f0bd4147be0 2022-07-20T02:32:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- 172.21.15.26:0/3476718720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 msgr2=0x7f0bd40aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 --2- 172.21.15.26:0/3476718720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40aee40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0bc4002f70 tx=0x7f0bc4006df0).stop 2022-07-20T02:32:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- 172.21.15.26:0/3476718720 shutdown_connections 2022-07-20T02:32:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 --2- 172.21.15.26:0/3476718720 >> v2:172.21.15.26:3300/0 conn(0x7f0bd40a7fc0 0x7f0bd40af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 --2- 172.21.15.26:0/3476718720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40aee40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- 172.21.15.26:0/3476718720 >> 172.21.15.26:0/3476718720 conn(0x7f0bd4060310 msgr2=0x7f0bd40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- 172.21.15.26:0/3476718720 shutdown_connections 2022-07-20T02:32:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.685+0000 7f0bd9376700 1 -- 172.21.15.26:0/3476718720 wait complete. 2022-07-20T02:32:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 Processor -- start 2022-07-20T02:32:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 -- start start 2022-07-20T02:32:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 0x7f0bd40abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 0x7f0bd4156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40628b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40abaa0 con 0x7f0bd40b5ef0 2022-07-20T02:32:40.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b4310 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bd9376700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b45b0 con 0x7f0bd4147be0 2022-07-20T02:32:40.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.689+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 0x7f0bd4156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 0x7f0bd4156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40208/0 (socket says 172.21.15.26:40208) 2022-07-20T02:32:40.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bcb7fe700 1 -- 172.21.15.26:0/2208963970 learned_addr learned my addr 172.21.15.26:0/2208963970 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 0x7f0bd40abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd8b75700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 -- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 msgr2=0x7f0bd40628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 -- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 msgr2=0x7f0bd4156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 0x7f0bd4156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 -- 172.21.15.26:0/2208963970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc400b180 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd3fff700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 0x7f0bd40abca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f0bd414b620 tx=0x7f0bbc007b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:40.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bbc0070d0 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc007230 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd40b4850 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bbc005740 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd41565d0 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd414b2a0 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.693+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0bbc010070 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.697+0000 7f0bd1ffb700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 0x7f0bb8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.697+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0bbc06b390 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.697+0000 7f0bcb7fe700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 0x7f0bb8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.697+0000 7f0bcb7fe700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 0x7f0bb8055ff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0bc401a000 tx=0x7f0bc4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.705+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0bbc042150 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.833+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bd40574e0 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd1ffb700 1 -- 172.21.15.26:0/2208963970 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f0bbc045770 con 0x7f0bd40a7fc0 2022-07-20T02:32:40.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:40.841 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":87,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:35.696918+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:33.535657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:34.094152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:40.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 msgr2=0x7f0bb8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 0x7f0bb8055ff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0bc401a000 tx=0x7f0bc4014000).stop 2022-07-20T02:32:40.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 msgr2=0x7f0bd40abca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 0x7f0bd40abca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f0bd414b620 tx=0x7f0bbc007b40).stop 2022-07-20T02:32:40.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 shutdown_connections 2022-07-20T02:32:40.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0bd40b5ef0 0x7f0bd4156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0bb8053b30 0x7f0bb8055ff0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bd4147be0 0x7f0bd40628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 --2- 172.21.15.26:0/2208963970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd40a7fc0 0x7f0bd40abca0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 >> 172.21.15.26:0/2208963970 conn(0x7f0bd4060310 msgr2=0x7f0bd40aeed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:40.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 shutdown_connections 2022-07-20T02:32:40.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.837+0000 7f0bd9376700 1 -- 172.21.15.26:0/2208963970 wait complete. 2022-07-20T02:32:40.853 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:32:33.535657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:32:34.094152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:32:40.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:32:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.953+0000 7f8654d7c700 1 Processor -- start 2022-07-20T02:32:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- start start 2022-07-20T02:32:40.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8650147d70 0x7f86500a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86500b2f10 con 0x7f86500b2b30 2022-07-20T02:32:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86500a7500 con 0x7f8650060f10 2022-07-20T02:32:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500a8e30 con 0x7f8650147d70 2022-07-20T02:32:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864e59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86500b2b30 0x7f8650147660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39494/0 (socket says 172.21.15.26:39494) 2022-07-20T02:32:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864e59c700 1 -- 172.21.15.26:0/29838150 learned_addr learned my addr 172.21.15.26:0/29838150 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:40.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864dd9b700 1 --2- 172.21.15.26:0/29838150 >> v2:172.21.15.120:3300/0 conn(0x7f8650147d70 0x7f86500a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267018920 0 0) 0x7f86500b2f10 con 0x7f86500b2b30 2022-07-20T02:32:40.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8634003260 con 0x7f86500b2b30 2022-07-20T02:32:40.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864dd9b700 1 -- 172.21.15.26:0/29838150 >> v1:172.21.15.120:6789/0 conn(0x7f8650060f10 legacy=0x7f8650066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761300196 0 0) 0x7f86500a7500 con 0x7f8650060f10 2022-07-20T02:32:40.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864dd9b700 1 -- 172.21.15.26:0/29838150 >> v1:172.21.15.26:6789/0 conn(0x7f86500b2b30 legacy=0x7f8650147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864dd9b700 1 -- 172.21.15.26:0/29838150 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500a77a0 con 0x7f8650147d70 2022-07-20T02:32:40.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864dd9b700 1 --2- 172.21.15.26:0/29838150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8650147d70 0x7f86500a7ee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f86400091d0).ready entity=mon.1 client_cookie=2bfd3749ca7d482a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b6c0 con 0x7f8650147d70 2022-07-20T02:32:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000b820 con 0x7f8650147d70 2022-07-20T02:32:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f864d59a700 1 -- 172.21.15.26:0/29838150 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b990 con 0x7f8650147d70 2022-07-20T02:32:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- 172.21.15.26:0/29838150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8650147d70 msgr2=0x7f86500a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 --2- 172.21.15.26:0/29838150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8650147d70 0x7f86500a7ee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f86400091d0).stop 2022-07-20T02:32:40.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- 172.21.15.26:0/29838150 shutdown_connections 2022-07-20T02:32:40.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 --2- 172.21.15.26:0/29838150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8650147d70 0x7f86500a7ee0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.957+0000 7f8654d7c700 1 -- 172.21.15.26:0/29838150 >> 172.21.15.26:0/29838150 conn(0x7f86500b6110 msgr2=0x7f86500b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:40.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- 172.21.15.26:0/29838150 shutdown_connections 2022-07-20T02:32:40.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- 172.21.15.26:0/29838150 wait complete. 2022-07-20T02:32:40.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 Processor -- start 2022-07-20T02:32:40.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- start start 2022-07-20T02:32:40.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86500b2b30 0x7f8650065c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8650147d70 0x7f8650156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:40.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501568e0 con 0x7f8650060f10 2022-07-20T02:32:40.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86500ae580 con 0x7f86500b2b30 2022-07-20T02:32:40.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f8654d7c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500ae7b0 con 0x7f8650147d70 2022-07-20T02:32:40.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40218/0 (socket says 172.21.15.26:40218) 2022-07-20T02:32:40.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 -- 172.21.15.26:0/357399520 learned_addr learned my addr 172.21.15.26:0/357399520 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:40.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864dd9b700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86500b2b30 0x7f8650065c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864ed9d700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8650147d70 0x7f8650156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:40.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 -- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8650147d70 msgr2=0x7f8650156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8650147d70 0x7f8650156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 -- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86500b2b30 msgr2=0x7f8650065c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:40.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86500b2b30 0x7f8650065c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:40.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 -- 172.21.15.26:0/357399520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8640009470 con 0x7f8650060f10 2022-07-20T02:32:40.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.961+0000 7f864e59c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f8644001110 tx=0x7f8644007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:40.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864400ce80 con 0x7f8650060f10 2022-07-20T02:32:40.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86440070d0 con 0x7f8650060f10 2022-07-20T02:32:40.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86500aea50 con 0x7f8650060f10 2022-07-20T02:32:40.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8644007430 con 0x7f8650060f10 2022-07-20T02:32:40.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86500ac9b0 con 0x7f8650060f10 2022-07-20T02:32:40.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86500af480 con 0x7f8650060f10 2022-07-20T02:32:40.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8644010070 con 0x7f8650060f10 2022-07-20T02:32:40.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 0x7f862c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f864406b5b0 con 0x7f8650060f10 2022-07-20T02:32:41.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.965+0000 7f864dd9b700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 0x7f862c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.969+0000 7f864dd9b700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 0x7f862c055e10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f864001b000 tx=0x7f8640016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:40.973+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86500af480 con 0x7f8650060f10 2022-07-20T02:32:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.101+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f86500af480 con 0x7f8650060f10 2022-07-20T02:32:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.101+0000 7f863f7fe700 1 -- 172.21.15.26:0/357399520 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v87) v1 ==== 110+0+10 (secure 0 0 0) 0x7f86500af480 con 0x7f8650060f10 2022-07-20T02:32:41.106 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:32:41.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 msgr2=0x7f862c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 0x7f862c055e10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f864001b000 tx=0x7f8640016000).stop 2022-07-20T02:32:41.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 msgr2=0x7f865015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f8644001110 tx=0x7f8644007aa0).stop 2022-07-20T02:32:41.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 shutdown_connections 2022-07-20T02:32:41.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f862c053950 0x7f862c055e10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8650147d70 0x7f8650156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86500b2b30 0x7f8650065c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 --2- 172.21.15.26:0/357399520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8650060f10 0x7f865015a9c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 >> 172.21.15.26:0/357399520 conn(0x7f86500b6110 msgr2=0x7f8650066a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 shutdown_connections 2022-07-20T02:32:41.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.105+0000 7f8654d7c700 1 -- 172.21.15.26:0/357399520 wait complete. 2022-07-20T02:32:41.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:32:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 Processor -- start 2022-07-20T02:32:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 -- start start 2022-07-20T02:32:41.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f048c0ace00 0x7f048c061390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c146bb0 con 0x7f048c05ce30 2022-07-20T02:32:41.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c146e50 con 0x7f048c066c00 2022-07-20T02:32:41.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04934a5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048c1470f0 con 0x7f048c0ace00 2022-07-20T02:32:41.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f048c05ce30 0x7f048c0ac6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39506/0 (socket says 172.21.15.26:39506) 2022-07-20T02:32:41.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 -- 172.21.15.26:0/1101276669 learned_addr learned my addr 172.21.15.26:0/1101276669 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:41.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f0492ca4700 1 --2- 172.21.15.26:0/1101276669 >> v2:172.21.15.120:3300/0 conn(0x7f048c0ace00 0x7f048c061390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536397523 0 0) 0x7f048c146bb0 con 0x7f048c05ce30 2022-07-20T02:32:41.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474003260 con 0x7f048c05ce30 2022-07-20T02:32:41.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181627659 0 0) 0x7f0474003260 con 0x7f048c05ce30 2022-07-20T02:32:41.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 >> v2:172.21.15.120:3300/0 conn(0x7f048c0ace00 msgr2=0x7f048c061390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:41.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 --2- 172.21.15.26:0/1101276669 >> v2:172.21.15.120:3300/0 conn(0x7f048c0ace00 0x7f048c061390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 >> v1:172.21.15.120:6789/0 conn(0x7f048c066c00 legacy=0x7f048c060c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c147390 con 0x7f048c05ce30 2022-07-20T02:32:41.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f0488001660 con 0x7f048c05ce30 2022-07-20T02:32:41.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 >> v1:172.21.15.26:6789/0 conn(0x7f048c05ce30 legacy=0x7f048c0ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 0x7f0474003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0474004500 0x7f0474004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474004e70 0x7f047400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c1470f0 con 0x7f0474003750 2022-07-20T02:32:41.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047400bd20 con 0x7f0474004500 2022-07-20T02:32:41.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047400bee0 con 0x7f0474004e70 2022-07-20T02:32:41.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 0x7f0474003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f0491ca2700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0474004500 0x7f0474004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f0492ca4700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474004e70 0x7f047400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 -- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474004e70 msgr2=0x7f047400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474004e70 0x7f047400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 -- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0474004500 msgr2=0x7f0474004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:41.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0474004500 0x7f0474004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 -- 172.21.15.26:0/1101276669 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474003260 con 0x7f0474003750 2022-07-20T02:32:41.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f0492ca4700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474004e70 0x7f047400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:32:41.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f0492ca4700 1 -- 172.21.15.26:0/1101276669 reap_dead start 2022-07-20T02:32:41.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.237+0000 7f04924a3700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 0x7f0474003b70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f0488001370 tx=0x7f048c146fe0).ready entity=mon.0 client_cookie=2d7f574dd07cd884 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04880070d0 con 0x7f0474003750 2022-07-20T02:32:41.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488007230 con 0x7f0474003750 2022-07-20T02:32:41.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04914a1700 1 -- 172.21.15.26:0/1101276669 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048800c8f0 con 0x7f0474003750 2022-07-20T02:32:41.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 msgr2=0x7f0474003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 0x7f0474003b70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f0488001370 tx=0x7f048c146fe0).stop 2022-07-20T02:32:41.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- 172.21.15.26:0/1101276669 shutdown_connections 2022-07-20T02:32:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 --2- 172.21.15.26:0/1101276669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0474003750 0x7f0474003b70 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- 172.21.15.26:0/1101276669 >> 172.21.15.26:0/1101276669 conn(0x7f048c0b7090 msgr2=0x7f048c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- 172.21.15.26:0/1101276669 shutdown_connections 2022-07-20T02:32:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- 172.21.15.26:0/1101276669 wait complete. 2022-07-20T02:32:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 Processor -- start 2022-07-20T02:32:41.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- start start 2022-07-20T02:32:41.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474003750 0x7f048c0af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 0x7f048c060f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 0x7f048c0af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c153330 con 0x7f048c0aec10 2022-07-20T02:32:41.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0ae7e0 con 0x7f048c1523b0 2022-07-20T02:32:41.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04934a5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0adef0 con 0x7f0474003750 2022-07-20T02:32:41.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f0492ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 0x7f048c0af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f0492ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 0x7f048c0af0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40242/0 (socket says 172.21.15.26:40242) 2022-07-20T02:32:41.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f0492ca4700 1 -- 172.21.15.26:0/3805311033 learned_addr learned my addr 172.21.15.26:0/3805311033 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:41.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f0491ca2700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 0x7f048c060f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.241+0000 7f04924a3700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474003750 0x7f048c0af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 -- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474003750 msgr2=0x7f048c0af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474003750 0x7f048c0af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 -- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 msgr2=0x7f048c0af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 0x7f048c0af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 -- 172.21.15.26:0/3805311033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0ae190 con 0x7f048c1523b0 2022-07-20T02:32:41.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f0491ca2700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 0x7f048c060f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f047c00b000 tx=0x7f047c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047c014620 con 0x7f048c1523b0 2022-07-20T02:32:41.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c014780 con 0x7f048c1523b0 2022-07-20T02:32:41.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c0666e0 con 0x7f048c1523b0 2022-07-20T02:32:41.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047c014ae0 con 0x7f048c1523b0 2022-07-20T02:32:41.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c0adae0 con 0x7f048c1523b0 2022-07-20T02:32:41.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c0574e0 con 0x7f048c1523b0 2022-07-20T02:32:41.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f047c014c40 con 0x7f048c1523b0 2022-07-20T02:32:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.245+0000 7f047b7fe700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 0x7f046c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.249+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f047c06bf70 con 0x7f048c1523b0 2022-07-20T02:32:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.249+0000 7f04924a3700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 0x7f046c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.249+0000 7f04924a3700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 0x7f046c055f30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0488019000 tx=0x7f0488013000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.257+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f047c042ba0 con 0x7f048c1523b0 2022-07-20T02:32:41.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.385+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f048c0ae920 con 0x7f048c1523b0 2022-07-20T02:32:41.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.385+0000 7f047b7fe700 1 -- 172.21.15.26:0/3805311033 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7f047c0461c0 con 0x7f048c1523b0 2022-07-20T02:32:41.417 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:41.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 msgr2=0x7f046c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 0x7f046c055f30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0488019000 tx=0x7f0488013000).stop 2022-07-20T02:32:41.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 msgr2=0x7f048c060f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 0x7f048c060f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f047c00b000 tx=0x7f047c009040).stop 2022-07-20T02:32:41.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 shutdown_connections 2022-07-20T02:32:41.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f048c0aec10 0x7f048c0af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f046c053a70 0x7f046c055f30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f048c1523b0 0x7f048c060f50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 --2- 172.21.15.26:0/3805311033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0474003750 0x7f048c0af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 >> 172.21.15.26:0/3805311033 conn(0x7f048c0b7090 msgr2=0x7f048c1459b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 shutdown_connections 2022-07-20T02:32:41.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.389+0000 7f04934a5700 1 -- 172.21.15.26:0/3805311033 wait complete. 2022-07-20T02:32:41.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:32:41.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 Processor -- start 2022-07-20T02:32:41.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 -- start start 2022-07-20T02:32:41.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdd84140ed0 0x7fdd84147590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8405d3b0 con 0x7fdd840b1160 2022-07-20T02:32:41.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840b4bc0 con 0x7fdd8405cfd0 2022-07-20T02:32:41.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd89155700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd841435b0 con 0x7fdd84140ed0 2022-07-20T02:32:41.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8259c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdd840b1160 0x7fdd84146e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39528/0 (socket says 172.21.15.26:39528) 2022-07-20T02:32:41.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdd84140ed0 0x7fdd84147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8259c700 1 -- 172.21.15.26:0/2108517398 learned_addr learned my addr 172.21.15.26:0/2108517398 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:41.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd81d9b700 1 -- 172.21.15.26:0/2108517398 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028540834 0 0) 0x7fdd8405d3b0 con 0x7fdd840b1160 2022-07-20T02:32:41.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd81d9b700 1 -- 172.21.15.26:0/2108517398 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd68003260 con 0x7fdd840b1160 2022-07-20T02:32:41.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8359e700 1 -- 172.21.15.26:0/2108517398 >> v1:172.21.15.120:6789/0 conn(0x7fdd8405cfd0 legacy=0x7fdd841407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8359e700 1 -- 172.21.15.26:0/2108517398 >> v1:172.21.15.26:6789/0 conn(0x7fdd840b1160 legacy=0x7fdd84146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8359e700 1 -- 172.21.15.26:0/2108517398 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840b4e60 con 0x7fdd84140ed0 2022-07-20T02:32:41.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd8359e700 1 --2- 172.21.15.26:0/2108517398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd84140ed0 0x7fdd84147590 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fdd7c0032f0 tx=0x7fdd7c0091d0).ready entity=mon.0 client_cookie=827970515b963bca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd81d9b700 1 -- 172.21.15.26:0/2108517398 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd7c00b6c0 con 0x7fdd84140ed0 2022-07-20T02:32:41.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd81d9b700 1 -- 172.21.15.26:0/2108517398 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c00b820 con 0x7fdd84140ed0 2022-07-20T02:32:41.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.537+0000 7fdd81d9b700 1 -- 172.21.15.26:0/2108517398 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd7c00b990 con 0x7fdd84140ed0 2022-07-20T02:32:41.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- 172.21.15.26:0/2108517398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd84140ed0 msgr2=0x7fdd84147590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 --2- 172.21.15.26:0/2108517398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd84140ed0 0x7fdd84147590 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fdd7c0032f0 tx=0x7fdd7c0091d0).stop 2022-07-20T02:32:41.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- 172.21.15.26:0/2108517398 shutdown_connections 2022-07-20T02:32:41.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 --2- 172.21.15.26:0/2108517398 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd84140ed0 0x7fdd84147590 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- 172.21.15.26:0/2108517398 >> 172.21.15.26:0/2108517398 conn(0x7fdd84066f70 msgr2=0x7fdd8405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- 172.21.15.26:0/2108517398 shutdown_connections 2022-07-20T02:32:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- 172.21.15.26:0/2108517398 wait complete. 2022-07-20T02:32:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 Processor -- start 2022-07-20T02:32:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.541+0000 7fdd89155700 1 -- start start 2022-07-20T02:32:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd8405cfd0 0x7fdd840b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd84140ed0 0x7fdd841560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd84153620 con 0x7fdd840b1160 2022-07-20T02:32:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840b6ff0 con 0x7fdd84140ed0 2022-07-20T02:32:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd84153380 con 0x7fdd8405cfd0 2022-07-20T02:32:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40254/0 (socket says 172.21.15.26:40254) 2022-07-20T02:32:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 -- 172.21.15.26:0/769684908 learned_addr learned my addr 172.21.15.26:0/769684908 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8359e700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd84140ed0 0x7fdd841560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd82d9d700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd8405cfd0 0x7fdd840b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 -- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd8405cfd0 msgr2=0x7fdd840b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd8405cfd0 0x7fdd840b66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 -- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd84140ed0 msgr2=0x7fdd841560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd84140ed0 0x7fdd841560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 -- 172.21.15.26:0/769684908 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c009470 con 0x7fdd840b1160 2022-07-20T02:32:41.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd8259c700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fdd78001110 tx=0x7fdd78007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd780072e0 con 0x7fdd840b1160 2022-07-20T02:32:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78007440 con 0x7fdd840b1160 2022-07-20T02:32:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd84148e80 con 0x7fdd840b1160 2022-07-20T02:32:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd78005800 con 0x7fdd840b1160 2022-07-20T02:32:41.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd841530a0 con 0x7fdd840b1160 2022-07-20T02:32:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.545+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840574e0 con 0x7fdd840b1160 2022-07-20T02:32:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.549+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdd7801a060 con 0x7fdd840b1160 2022-07-20T02:32:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.549+0000 7fdd73fff700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 0x7fdd60055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.549+0000 7fdd82d9d700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 0x7fdd60055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.549+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdd7806b9f0 con 0x7fdd840b1160 2022-07-20T02:32:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.549+0000 7fdd82d9d700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 0x7fdd60055fc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdd7400b000 tx=0x7fdd74009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.557+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd7806de60 con 0x7fdd840b1160 2022-07-20T02:32:41.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd84060030 con 0x7fdd840b1160 2022-07-20T02:32:41.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd73fff700 1 -- 172.21.15.26:0/769684908 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdd780180a0 con 0x7fdd840b1160 2022-07-20T02:32:41.693 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:41.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 msgr2=0x7fdd60055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 0x7fdd60055fc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdd7400b000 tx=0x7fdd74009040).stop 2022-07-20T02:32:41.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 msgr2=0x7fdd84155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fdd78001110 tx=0x7fdd78007b80).stop 2022-07-20T02:32:41.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 shutdown_connections 2022-07-20T02:32:41.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd840b1160 0x7fdd84155bd0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd60053b00 0x7fdd60055fc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd84140ed0 0x7fdd841560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 --2- 172.21.15.26:0/769684908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd8405cfd0 0x7fdd840b66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 >> 172.21.15.26:0/769684908 conn(0x7fdd84066f70 msgr2=0x7fdd840b99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 shutdown_connections 2022-07-20T02:32:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.685+0000 7fdd89155700 1 -- 172.21.15.26:0/769684908 wait complete. 2022-07-20T02:32:41.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:32:41.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.821+0000 7fdd96ad1700 1 Processor -- start 2022-07-20T02:32:41.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.821+0000 7fdd96ad1700 1 -- start start 2022-07-20T02:32:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdd900a9ef0 0x7fdd9014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd900b12b0 con 0x7fdd900b2d40 2022-07-20T02:32:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd90066e70 con 0x7fdd900add60 2022-07-20T02:32:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd90067110 con 0x7fdd900a9ef0 2022-07-20T02:32:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd9506e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdd900a9ef0 0x7fdd9014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdd900b2d40 0x7fdd900aff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39542/0 (socket says 172.21.15.26:39542) 2022-07-20T02:32:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8ffff700 1 -- 172.21.15.26:0/1615578155 learned_addr learned my addr 172.21.15.26:0/1615578155 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8f7fe700 1 -- 172.21.15.26:0/1615578155 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227962667 0 0) 0x7fdd900b12b0 con 0x7fdd900b2d40 2022-07-20T02:32:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd9506e700 1 -- 172.21.15.26:0/1615578155 >> v1:172.21.15.120:6789/0 conn(0x7fdd900add60 legacy=0x7fdd900af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd9506e700 1 -- 172.21.15.26:0/1615578155 >> v1:172.21.15.26:6789/0 conn(0x7fdd900b2d40 legacy=0x7fdd900aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd9506e700 1 -- 172.21.15.26:0/1615578155 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90066af0 con 0x7fdd900a9ef0 2022-07-20T02:32:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd9506e700 1 --2- 172.21.15.26:0/1615578155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900a9ef0 0x7fdd9014abb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fdd880032f0 tx=0x7fdd880091d0).ready entity=mon.0 client_cookie=99696e314bdf97b7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8f7fe700 1 -- 172.21.15.26:0/1615578155 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd88009ea0 con 0x7fdd900a9ef0 2022-07-20T02:32:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8f7fe700 1 -- 172.21.15.26:0/1615578155 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8800b8a0 con 0x7fdd900a9ef0 2022-07-20T02:32:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd8f7fe700 1 -- 172.21.15.26:0/1615578155 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd8800ba10 con 0x7fdd900a9ef0 2022-07-20T02:32:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1615578155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900a9ef0 msgr2=0x7fdd9014abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1615578155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900a9ef0 0x7fdd9014abb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fdd880032f0 tx=0x7fdd880091d0).stop 2022-07-20T02:32:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1615578155 shutdown_connections 2022-07-20T02:32:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1615578155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900a9ef0 0x7fdd9014abb0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1615578155 >> 172.21.15.26:0/1615578155 conn(0x7fdd900b66e0 msgr2=0x7fdd900b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1615578155 shutdown_connections 2022-07-20T02:32:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.825+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1615578155 wait complete. 2022-07-20T02:32:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 Processor -- start 2022-07-20T02:32:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 -- start start 2022-07-20T02:32:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 0x7fdd900b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd900add60 0x7fdd9013d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9005d4f0 con 0x7fdd900b2d40 2022-07-20T02:32:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90065ae0 con 0x7fdd900a9ef0 2022-07-20T02:32:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd96ad1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90065840 con 0x7fdd900add60 2022-07-20T02:32:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd9506e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd9506e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40268/0 (socket says 172.21.15.26:40268) 2022-07-20T02:32:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd9506e700 1 -- 172.21.15.26:0/1635432968 learned_addr learned my addr 172.21.15.26:0/1635432968 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd9486d700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 0x7fdd900b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.829+0000 7fdd8ffff700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd900add60 0x7fdd9013d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 -- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd900add60 msgr2=0x7fdd9013d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd900add60 0x7fdd9013d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 -- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 msgr2=0x7fdd900b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 0x7fdd900b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 -- 172.21.15.26:0/1635432968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009470 con 0x7fdd900b2d40 2022-07-20T02:32:41.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9486d700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 0x7fdd900b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd9506e700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fdd900a9730 tx=0x7fdd88016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd88009ea0 con 0x7fdd900b2d40 2022-07-20T02:32:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd88011e60 con 0x7fdd900b2d40 2022-07-20T02:32:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd90063a30 con 0x7fdd900b2d40 2022-07-20T02:32:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd8801f5e0 con 0x7fdd900b2d40 2022-07-20T02:32:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd900ad520 con 0x7fdd900b2d40 2022-07-20T02:32:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd900ae140 con 0x7fdd900b2d40 2022-07-20T02:32:41.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdd88026030 con 0x7fdd900b2d40 2022-07-20T02:32:41.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.833+0000 7fdd8dffb700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 0x7fdd70055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:41.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.837+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdd8806cda0 con 0x7fdd900b2d40 2022-07-20T02:32:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.837+0000 7fdd9486d700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 0x7fdd70055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.837+0000 7fdd9486d700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 0x7fdd70055f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdd8400d000 tx=0x7fdd84009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.841+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd88043b60 con 0x7fdd900b2d40 2022-07-20T02:32:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.969+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd900574e0 con 0x7fdd900b2d40 2022-07-20T02:32:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.969+0000 7fdd8dffb700 1 -- 172.21.15.26:0/1635432968 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdd880240a0 con 0x7fdd900b2d40 2022-07-20T02:32:41.977 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:41.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 msgr2=0x7fdd70055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 0x7fdd70055f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdd8400d000 tx=0x7fdd84009040).stop 2022-07-20T02:32:41.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 msgr2=0x7fdd90064170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:41.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fdd900a9730 tx=0x7fdd88016000).stop 2022-07-20T02:32:41.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 shutdown_connections 2022-07-20T02:32:41.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd900b2d40 0x7fdd90064170 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd70053a70 0x7fdd70055f30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd900add60 0x7fdd9013d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 --2- 172.21.15.26:0/1635432968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd900a9ef0 0x7fdd900b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:41.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 >> 172.21.15.26:0/1635432968 conn(0x7fdd900b66e0 msgr2=0x7fdd9005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:41.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 shutdown_connections 2022-07-20T02:32:41.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:41.973+0000 7fdd96ad1700 1 -- 172.21.15.26:0/1635432968 wait complete. 2022-07-20T02:32:41.996 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:32:41.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 Processor -- start 2022-07-20T02:32:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- start start 2022-07-20T02:32:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e4067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc2e4140320 0x7fc2e4140760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e4146750 con 0x7fc2e4143070 2022-07-20T02:32:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e41469f0 con 0x7fc2e4140320 2022-07-20T02:32:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e4146c90 con 0x7fc2e40b5d50 2022-07-20T02:32:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e1d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2e4143070 0x7fc2e4143490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39554/0 (socket says 172.21.15.26:39554) 2022-07-20T02:32:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e1d9b700 1 -- 172.21.15.26:0/3149358799 learned_addr learned my addr 172.21.15.26:0/3149358799 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e2d9d700 1 --2- 172.21.15.26:0/3149358799 >> v2:172.21.15.26:3300/0 conn(0x7fc2e4140320 0x7fc2e4140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3149358799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e4067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e159a700 1 -- 172.21.15.26:0/3149358799 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926056471 0 0) 0x7fc2e4146750 con 0x7fc2e4143070 2022-07-20T02:32:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e159a700 1 -- 172.21.15.26:0/3149358799 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c4003260 con 0x7fc2e4143070 2022-07-20T02:32:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 -- 172.21.15.26:0/3149358799 >> v2:172.21.15.26:3300/0 conn(0x7fc2e4140320 msgr2=0x7fc2e4140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3149358799 >> v2:172.21.15.26:3300/0 conn(0x7fc2e4140320 0x7fc2e4140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 -- 172.21.15.26:0/3149358799 >> v1:172.21.15.26:6789/0 conn(0x7fc2e4143070 legacy=0x7fc2e4143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 -- 172.21.15.26:0/3149358799 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4146f30 con 0x7fc2e40b5d50 2022-07-20T02:32:42.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3149358799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e4067490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fc2d4002f70 tx=0x7fc2d4006df0).ready entity=mon.2 client_cookie=497b272a42a5221c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e159a700 1 -- 172.21.15.26:0/3149358799 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2d400d4a0 con 0x7fc2e40b5d50 2022-07-20T02:32:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e159a700 1 -- 172.21.15.26:0/3149358799 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4000e90 con 0x7fc2e40b5d50 2022-07-20T02:32:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e159a700 1 -- 172.21.15.26:0/3149358799 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2d400d7f0 con 0x7fc2e40b5d50 2022-07-20T02:32:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- 172.21.15.26:0/3149358799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 msgr2=0x7fc2e4067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3149358799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e4067490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fc2d4002f70 tx=0x7fc2d4006df0).stop 2022-07-20T02:32:42.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- 172.21.15.26:0/3149358799 shutdown_connections 2022-07-20T02:32:42.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3149358799 >> v2:172.21.15.26:3300/0 conn(0x7fc2e4140320 0x7fc2e4140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3149358799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e4067490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- 172.21.15.26:0/3149358799 >> 172.21.15.26:0/3149358799 conn(0x7fc2e4060310 msgr2=0x7fc2e40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- 172.21.15.26:0/3149358799 shutdown_connections 2022-07-20T02:32:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.101+0000 7fc2e359e700 1 -- 172.21.15.26:0/3149358799 wait complete. 2022-07-20T02:32:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 Processor -- start 2022-07-20T02:32:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 -- start start 2022-07-20T02:32:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e40abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e4143070 0x7fc2e40ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41456f0 con 0x7fc2e4140320 2022-07-20T02:32:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40ab680 con 0x7fc2e4143070 2022-07-20T02:32:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41460d0 con 0x7fc2e40b5d50 2022-07-20T02:32:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40282/0 (socket says 172.21.15.26:40282) 2022-07-20T02:32:42.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 -- 172.21.15.26:0/3730008667 learned_addr learned my addr 172.21.15.26:0/3730008667 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:42.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e40abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e2d9d700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e4143070 0x7fc2e40ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 -- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 msgr2=0x7fc2e40abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e40abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 -- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e4143070 msgr2=0x7fc2e40ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e4143070 0x7fc2e40ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 -- 172.21.15.26:0/3730008667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d400b180 con 0x7fc2e4140320 2022-07-20T02:32:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.105+0000 7fc2e1d9b700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7fc2cc001110 tx=0x7fc2cc007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:42.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2cc0070d0 con 0x7fc2e4140320 2022-07-20T02:32:42.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc007230 con 0x7fc2e4140320 2022-07-20T02:32:42.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e41451b0 con 0x7fc2e4140320 2022-07-20T02:32:42.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2cc005610 con 0x7fc2e4140320 2022-07-20T02:32:42.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e4144910 con 0x7fc2e4140320 2022-07-20T02:32:42.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4144ad0 con 0x7fc2e4140320 2022-07-20T02:32:42.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2cc01a060 con 0x7fc2e4140320 2022-07-20T02:32:42.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 0x7fc2c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:42.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc2cc06b3a0 con 0x7fc2e4140320 2022-07-20T02:32:42.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 0x7fc2c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.109+0000 7fc2e259c700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 0x7fc2c8055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc2d401a000 tx=0x7fc2d4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.117+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2cc042160 con 0x7fc2e4140320 2022-07-20T02:32:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.277+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2e4140c50 con 0x7fc2e4140320 2022-07-20T02:32:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2db7fe700 1 -- 172.21.15.26:0/3730008667 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7fc2cc018030 con 0x7fc2e4140320 2022-07-20T02:32:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:42.293 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:42.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 msgr2=0x7fc2c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 0x7fc2c8055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc2d401a000 tx=0x7fc2d4014000).stop 2022-07-20T02:32:42.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 msgr2=0x7fc2e40aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:42.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7fc2cc001110 tx=0x7fc2cc007b80).stop 2022-07-20T02:32:42.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 shutdown_connections 2022-07-20T02:32:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2e4140320 0x7fc2e40aaca0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc2c8053a70 0x7fc2c8055f30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2e4143070 0x7fc2e40ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 --2- 172.21.15.26:0/3730008667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2e40b5d50 0x7fc2e40abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:42.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 >> 172.21.15.26:0/3730008667 conn(0x7fc2e4060310 msgr2=0x7fc2e405ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:42.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 shutdown_connections 2022-07-20T02:32:42.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:42.281+0000 7fc2e359e700 1 -- 172.21.15.26:0/3730008667 wait complete. 2022-07-20T02:32:42.324 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:42.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:34.694079+0000', 'modified': '2022-07-20T02:32:35.719385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6699': {'gid': 6699, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2208502036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2208502036}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2208502036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:32:42.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:32:42.635 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:32:42.755 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":6699,"want_state":"up:active","state":"up:active","rank_uptime":8.0481516170000003,"mdsmap_epoch":43,"osdmap_epoch":87,"osdmap_epoch_barrier":87,"uptime":12.498684110999999} 2022-07-20T02:32:42.756 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 6699, "mdsmap_epoch": 43, "osdmap_epoch": 87, "osdmap_epoch_barrier": 87, "rank_uptime": 8.048151617, "state": "up:active", "uptime": 12.498684111, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:32:42.757 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:32:42.757 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:32:42.758 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:32:42.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:42.761 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:42.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:32:42.803 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:42.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:42.864 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:42.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:42.867 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:32:42.874 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:32:42.875 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:42.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:32:42.952 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:32:42.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:32:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:32:42.995+0000 7fa34f330080 -1 init, newargv = 0x55c3a917ba10 newargc=9 2022-07-20T02:32:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22611]: starting ceph client 2022-07-20T02:32:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:32:43.022 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:32:43.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.024 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:32:43.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22611]: starting fuse 2022-07-20T02:32:43.030 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:32:43.031 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:32:43.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:32:43.034 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:32:43.035 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:32:43.036 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:32:43.037 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:32:43.038 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:32:43.039 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:32:43.039 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:32:43.040 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:32:43.040 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:32:43.041 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:32:43.042 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:32:43.042 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:32:43.043 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:32:43.043 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:32:43.043 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:32:43.044 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:32:43.044 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:32:43.045 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:32:43.046 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:32:43.046 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:32:43.047 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:32:43.047 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:32:43.103 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22611.asok 2022-07-20T02:32:43.103 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22611.asok 2022-07-20T02:32:43.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22611.asok status 2022-07-20T02:32:43.257 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:32:43.257 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:32:43.258 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:32:43.258 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:32:43.259 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:32:43.259 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:32:43.265 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:32:43.265 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22611", 2022-07-20T02:32:43.266 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:32:43.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:32:43.267 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:32:43.267 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:32:43.267 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 6757, 2022-07-20T02:32:43.268 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:32:43.268 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:32:43.269 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:32:43.270 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 6757 2022-07-20T02:32:43.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:32:43.271 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:32:43.271 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:32:43.271 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:32:43.272 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 936151184 2022-07-20T02:32:43.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:32:43.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:32:43.272 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:32:43.272 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:32:43.273 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:32:43.273 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 936151184 2022-07-20T02:32:43.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:32:43.295 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.6757 172.21.15.176:0/936151184", 2022-07-20T02:32:43.296 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/936151184", 2022-07-20T02:32:43.296 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:32:43.296 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 43, 2022-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 87, 2022-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:32:43.297 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:32:43.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.300 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:43.300 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:43.325 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:32:43.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:32:43.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.327 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:43.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 Processor -- start 2022-07-20T02:32:43.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 -- start start 2022-07-20T02:32:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa6681465b0 0x7fa668066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:43.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680b1ae0 con 0x7fa6680672a0 2022-07-20T02:32:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680b1d80 con 0x7fa6680b24c0 2022-07-20T02:32:43.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.529+0000 7fa66d05c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6680b2e20 con 0x7fa6681465b0 2022-07-20T02:32:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66c85b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa6680672a0 0x7fa668146990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39572/0 (socket says 172.21.15.26:39572) 2022-07-20T02:32:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66c85b700 1 -- 172.21.15.26:0/4186337381 learned_addr learned my addr 172.21.15.26:0/4186337381 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:43.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa667fff700 1 --2- 172.21.15.26:0/4186337381 >> v2:172.21.15.120:3300/0 conn(0x7fa6681465b0 0x7fa668066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611131531 0 0) 0x7fa6680b1ae0 con 0x7fa6680672a0 2022-07-20T02:32:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa648003260 con 0x7fa6680672a0 2022-07-20T02:32:43.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297264740 0 0) 0x7fa6680b1d80 con 0x7fa6680b24c0 2022-07-20T02:32:43.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6680b1ae0 con 0x7fa6680b24c0 2022-07-20T02:32:43.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa667fff700 1 -- 172.21.15.26:0/4186337381 >> v1:172.21.15.120:6789/0 conn(0x7fa6680b24c0 legacy=0x7fa6680604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa667fff700 1 -- 172.21.15.26:0/4186337381 >> v1:172.21.15.26:6789/0 conn(0x7fa6680672a0 legacy=0x7fa668146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa667fff700 1 -- 172.21.15.26:0/4186337381 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680b2020 con 0x7fa6681465b0 2022-07-20T02:32:43.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa667fff700 1 --2- 172.21.15.26:0/4186337381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6681465b0 0x7fa668066db0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa658002f70 tx=0x7fa658006df0).ready entity=mon.1 client_cookie=3a8b8a75d4312776 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:43.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa65800d540 con 0x7fa6681465b0 2022-07-20T02:32:43.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658000e90 con 0x7fa6681465b0 2022-07-20T02:32:43.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa6677fe700 1 -- 172.21.15.26:0/4186337381 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa65800d890 con 0x7fa6681465b0 2022-07-20T02:32:43.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 -- 172.21.15.26:0/4186337381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6681465b0 msgr2=0x7fa668066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 --2- 172.21.15.26:0/4186337381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6681465b0 0x7fa668066db0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa658002f70 tx=0x7fa658006df0).stop 2022-07-20T02:32:43.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 -- 172.21.15.26:0/4186337381 shutdown_connections 2022-07-20T02:32:43.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 --2- 172.21.15.26:0/4186337381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6681465b0 0x7fa668066db0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 -- 172.21.15.26:0/4186337381 >> 172.21.15.26:0/4186337381 conn(0x7fa6680b3050 msgr2=0x7fa6680b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:43.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 -- 172.21.15.26:0/4186337381 shutdown_connections 2022-07-20T02:32:43.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.533+0000 7fa66d05c700 1 -- 172.21.15.26:0/4186337381 wait complete. 2022-07-20T02:32:43.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 Processor -- start 2022-07-20T02:32:43.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- start start 2022-07-20T02:32:43.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6680672a0 0x7fa6680648a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:43.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:43.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668063b60 0x7fa668155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:43.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6681566c0 con 0x7fa6681465b0 2022-07-20T02:32:43.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680aef90 con 0x7fa6680672a0 2022-07-20T02:32:43.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680af1c0 con 0x7fa668063b60 2022-07-20T02:32:43.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:43.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40298/0 (socket says 172.21.15.26:40298) 2022-07-20T02:32:43.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 -- 172.21.15.26:0/67781791 learned_addr learned my addr 172.21.15.26:0/67781791 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:43.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa667fff700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6680672a0 0x7fa6680648a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:43.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66c85b700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668063b60 0x7fa668155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:43.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 -- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668063b60 msgr2=0x7fa668155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668063b60 0x7fa668155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 -- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6680672a0 msgr2=0x7fa6680648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6680672a0 0x7fa6680648a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 -- 172.21.15.26:0/67781791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65800b180 con 0x7fa6681465b0 2022-07-20T02:32:43.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa65f7fe700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fa650001110 tx=0x7fa650007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:43.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa650012400 con 0x7fa6681465b0 2022-07-20T02:32:43.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650012560 con 0x7fa6681465b0 2022-07-20T02:32:43.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6500128c0 con 0x7fa6681465b0 2022-07-20T02:32:43.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa668064d90 con 0x7fa6681465b0 2022-07-20T02:32:43.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.537+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6680afb50 con 0x7fa6681465b0 2022-07-20T02:32:43.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa650010070 con 0x7fa6681465b0 2022-07-20T02:32:43.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6681561c0 con 0x7fa6681465b0 2022-07-20T02:32:43.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa665ffb700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 0x7fa64c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:43.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa65006b1d0 con 0x7fa6681465b0 2022-07-20T02:32:43.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa667fff700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 0x7fa64c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:43.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.541+0000 7fa667fff700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 0x7fa64c055f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa658018000 tx=0x7fa658014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:43.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.545+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa650041f90 con 0x7fa6681465b0 2022-07-20T02:32:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.709+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6680af380 con 0x7fa6681465b0 2022-07-20T02:32:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.709+0000 7fa665ffb700 1 -- 172.21.15.26:0/67781791 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7fa6500455b0 con 0x7fa6681465b0 2022-07-20T02:32:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:43.723 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 msgr2=0x7fa64c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 0x7fa64c055f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa658018000 tx=0x7fa658014000).stop 2022-07-20T02:32:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 msgr2=0x7fa6680af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fa650001110 tx=0x7fa650007b20).stop 2022-07-20T02:32:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 shutdown_connections 2022-07-20T02:32:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa6681465b0 0x7fa6680af660 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa64c053a70 0x7fa64c055f30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa668063b60 0x7fa668155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 --2- 172.21.15.26:0/67781791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6680672a0 0x7fa6680648a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 >> 172.21.15.26:0/67781791 conn(0x7fa6680b3050 msgr2=0x7fa6680b53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 shutdown_connections 2022-07-20T02:32:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:43.713+0000 7fa66d05c700 1 -- 172.21.15.26:0/67781791 wait complete. 2022-07-20T02:32:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:43.737 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 1 2022-07-20T02:32:43.849 INFO:teuthology.orchestra.run.smithi026.stdout:{"path":"/","ino":1,"rdev":0,"ctime":"2022-07-20T02:32:40.570274+0000","btime":"2022-07-20T02:32:34.699495+0000","mode":17407,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":16,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":0,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-20T02:32:34.699495+0000","atime":"0.000000","time_warp_seq":0,"change_attr":1,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-20T02:32:40.570274+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-20T02:32:34.699495+0000"},"version":3,"file_data_version":0,"xattr_version":1,"backtrace_version":3,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":3,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"authpin":1},"nref":9,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"xlockdone","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{"gather_set":[],"state":"sync","is_leased":false,"num_rdlocks":1,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent"],"client_caps":[{"client_id":6715,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2},{"client_id":6757,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[]} 2022-07-20T02:32:43.853 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "2022-07-20T02:32:34.699495+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "0.000000", "auth_pins": 2, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "xlockdone", "xlock_by": {} }, "backtrace_version": 3, "btime": "2022-07-20T02:32:34.699495+0000", "change_attr": 1, "client_caps": [ { "client_id": 6715, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" }, { "client_id": 6757, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-20T02:32:40.570274+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 16, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 17407, "mtime": "2022-07-20T02:32:34.699495+0000", "nestlock": {}, "nlink": 1, "nref": 9, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/", "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 3, "request": 1 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2022-07-20T02:32:40.570274+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": { "gather_set": [], "is_leased": false, "num_rdlocks": 1, "num_wrlocks": 0, "num_xlocks": 0, "state": "sync", "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 0, "truncate_size": 18446744073709551615, "uid": 0, "version": 3, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } 2022-07-20T02:32:43.855 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:32:43.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:43.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:43.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:43.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:43.983 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:32:43.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:43.986 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:43.987 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:43.997 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:32:44.000 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:44.001 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:32:44.002 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:32:44.020 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:44.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[22987]: fuse finished with error 0 and tester_r 0 2022-07-20T02:32:44.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:44.066 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:44.067 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:44.100 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:32:44.102 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:32:44.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:32:44.229 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:44.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:44.247 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:44.247 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:44.272 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:32:44.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:32:44.274 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:32:44.275 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:32:44.284 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:44.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22611]: fuse finished with error 0 and tester_r 0 2022-07-20T02:32:44.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:44.322 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:44.322 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:44.358 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:32:44.358 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:32:44.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:32:44.397 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:44.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:32:44.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 Processor -- start 2022-07-20T02:32:44.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 -- start start 2022-07-20T02:32:44.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8147000 0x7f18b8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f18b8060f00 0x7f18b813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b81474e0 con 0x7f18b805cfd0 2022-07-20T02:32:44.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80b4e40 con 0x7f18b8060f00 2022-07-20T02:32:44.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.517+0000 7f18be571700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b805d3b0 con 0x7f18b8147000 2022-07-20T02:32:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f18b8060f00 0x7f18b813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f18b8060f00 0x7f18b813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40306/0 (socket says 172.21.15.26:40306) 2022-07-20T02:32:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 -- 172.21.15.26:0/1637366334 learned_addr learned my addr 172.21.15.26:0/1637366334 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bd56f700 1 --1- 172.21.15.26:0/1637366334 >> v1:172.21.15.26:6789/0 conn(0x7f18b805cfd0 0x7f18b80b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39582/0 (socket says 172.21.15.26:39582) 2022-07-20T02:32:44.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bcd6e700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8147000 0x7f18b8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 -- 172.21.15.26:0/1637366334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8147000 msgr2=0x7f18b8143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:44.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8147000 0x7f18b8143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:44.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 -- 172.21.15.26:0/1637366334 >> v1:172.21.15.26:6789/0 conn(0x7f18b805cfd0 legacy=0x7f18b80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:44.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18affff700 1 -- 172.21.15.26:0/1637366334 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217816095 0 0) 0x7f18b81474e0 con 0x7f18b805cfd0 2022-07-20T02:32:44.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 -- 172.21.15.26:0/1637366334 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b81431e0 con 0x7f18b8060f00 2022-07-20T02:32:44.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18bdd70700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8060f00 0x7f18b813c9b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f18b0002f70 tx=0x7f18b0006eb0).ready entity=mon.0 client_cookie=93c321d610baacf8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:44.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18affff700 1 -- 172.21.15.26:0/1637366334 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b000d570 con 0x7f18b8060f00 2022-07-20T02:32:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18affff700 1 -- 172.21.15.26:0/1637366334 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b0000e90 con 0x7f18b8060f00 2022-07-20T02:32:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18affff700 1 -- 172.21.15.26:0/1637366334 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b000d8c0 con 0x7f18b8060f00 2022-07-20T02:32:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- 172.21.15.26:0/1637366334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8060f00 msgr2=0x7f18b813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:44.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8060f00 0x7f18b813c9b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f18b0002f70 tx=0x7f18b0006eb0).stop 2022-07-20T02:32:44.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- 172.21.15.26:0/1637366334 shutdown_connections 2022-07-20T02:32:44.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8060f00 0x7f18b813c9b0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:44.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 --2- 172.21.15.26:0/1637366334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8147000 0x7f18b8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:44.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- 172.21.15.26:0/1637366334 >> 172.21.15.26:0/1637366334 conn(0x7f18b80620c0 msgr2=0x7f18b805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:44.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- 172.21.15.26:0/1637366334 shutdown_connections 2022-07-20T02:32:44.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- 172.21.15.26:0/1637366334 wait complete. 2022-07-20T02:32:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 Processor -- start 2022-07-20T02:32:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.521+0000 7f18be571700 1 -- start start 2022-07-20T02:32:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8060f00 0x7f18b80ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8141480 0x7f18b80af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b814fe80 con 0x7f18b8147000 2022-07-20T02:32:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8152cc0 con 0x7f18b8141480 2022-07-20T02:32:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8152f60 con 0x7f18b8060f00 2022-07-20T02:32:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40308/0 (socket says 172.21.15.26:40308) 2022-07-20T02:32:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 -- 172.21.15.26:0/2188909810 learned_addr learned my addr 172.21.15.26:0/2188909810 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bd56f700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8060f00 0x7f18b80ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bdd70700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8141480 0x7f18b80af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 -- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8060f00 msgr2=0x7f18b80ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:44.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8060f00 0x7f18b80ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:44.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 -- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8141480 msgr2=0x7f18b80af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:44.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8141480 0x7f18b80af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 -- 172.21.15.26:0/2188909810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b000b1e0 con 0x7f18b8147000 2022-07-20T02:32:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18bcd6e700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f18b400c000 tx=0x7f18b40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b400c6a0 con 0x7f18b8147000 2022-07-20T02:32:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b400c800 con 0x7f18b8147000 2022-07-20T02:32:44.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b80af480 con 0x7f18b8147000 2022-07-20T02:32:44.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b400cb60 con 0x7f18b8147000 2022-07-20T02:32:44.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.525+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b80b3740 con 0x7f18b8147000 2022-07-20T02:32:44.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b80af610 con 0x7f18b8147000 2022-07-20T02:32:44.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f18b4022070 con 0x7f18b8147000 2022-07-20T02:32:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18ae7fc700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 0x7f1898055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f18b406c840 con 0x7f18b8147000 2022-07-20T02:32:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18bd56f700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 0x7f1898055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:44.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.529+0000 7f18bd56f700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 0x7f1898055f30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f18a800d000 tx=0x7f18a8007b80).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:44.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.537+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f18b4043470 con 0x7f18b8147000 2022-07-20T02:32:44.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.669+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f18b8153120 con 0x7f18b8147000 2022-07-20T02:32:44.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18ae7fc700 1 -- 172.21.15.26:0/2188909810 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v87) v1 ==== 81+0+34088 (secure 0 0 0) 0x7f18b4046a90 con 0x7f18b8147000 2022-07-20T02:32:44.677 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:44.677 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:44.677 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 87, 2022-07-20T02:32:44.678 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:32:44.679 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:32:44.679 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:32:35.696918+0000", 2022-07-20T02:32:44.679 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:44.680 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:44.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:32:44.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:32:44.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:32:44.681 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:32:44.682 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:32:44.682 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:32:44.682 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:32:44.684 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:44.685 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:32:44.685 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:32:44.688 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:32:44.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:32:44.731 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:32:44.731 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 16, 2022-07-20T02:32:44.732 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:32:44.732 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:32:44.733 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:32:44.796 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:32:44.796 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:32:44.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:44.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:32:44.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:32:44.801 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:32:44.801 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:44.802 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:44.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:44.802 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:44.807 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:44.808 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:44.815 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:44.815 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:44.816 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:32:44.816 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:32:44.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:32:44.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:32:44.820 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:32:44.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:44.821 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:44.821 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:44.822 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:44.824 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:44.824 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:44.832 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:44.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.833 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:32:44.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:44.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:44.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:44.835 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:44.835 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:44.836 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:44.836 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:44.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:44.838 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:44.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:44.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:44.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:44.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:44.840 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:44.840 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:44.841 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:44.842 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:44.843 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:44.843 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:44.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:44.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:44.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:44.845 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:44.845 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:44.846 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:44.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:44.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.848 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:44.848 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:44.848 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:44.849 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:44.849 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:44.850 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:44.850 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:44.850 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:44.851 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:44.851 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:44.852 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:44.852 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:44.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:32:44.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.853 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:44.854 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:32:44.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:44.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:44.856 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:32:44.856 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:32:44.856 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:32:44.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:44.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:44.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:44.859 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:44.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:44.860 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:44.860 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:44.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:44.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:44.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:44.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:44.868 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:44.872 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:44.873 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:44.873 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:44.874 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:44.874 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:44.875 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:44.875 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:44.876 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:44.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:32:44.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:44.878 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:44.878 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:44.879 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:44.879 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:44.880 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:44.891 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:44.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:44.893 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:44.893 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:44.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:44.894 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:44.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:44.895 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:44.896 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:44.896 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:44.897 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:44.897 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:44.898 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:44.898 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:44.898 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:44.899 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:44.899 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:44.900 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:44.901 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:44.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:44.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.902 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:44.902 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:44.903 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:44.903 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:44.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:44.904 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:44.905 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:44.905 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:44.905 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:44.906 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:44.907 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:44.907 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:44.908 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:44.908 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:32:44.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:44.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:44.910 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 15, 2022-07-20T02:32:44.911 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:32:44.911 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:32:33.535657+0000", 2022-07-20T02:32:44.911 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:44.912 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:44.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:44.913 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:44.913 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:44.914 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:44.914 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:44.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:44.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:44.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:44.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:44.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:44.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:44.920 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:44.920 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:44.920 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:44.921 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:44.921 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:44.922 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:44.922 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:44.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "86", 2022-07-20T02:32:44.924 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:44.924 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:44.924 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:44.925 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:44.925 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:44.926 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:44.926 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:44.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:44.927 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:44.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:44.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:44.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:44.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:44.930 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:44.930 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:44.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:44.931 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:44.932 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:44.933 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:44.934 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:44.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:44.936 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:44.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:44.938 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:44.939 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:44.940 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:32:44.941 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:32:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:32:44.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:44.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:32:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:32:44.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:44.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 16, 2022-07-20T02:32:44.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:32:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:32:34.094152+0000", 2022-07-20T02:32:44.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:44.984 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:44.985 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:44.986 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:44.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:44.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:45.015 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:45.016 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:45.017 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:45.018 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.019 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "87", 2022-07-20T02:32:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:45.020 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:45.021 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:45.022 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:45.025 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:45.026 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:45.027 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:45.028 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:45.057 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:45.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.059 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:45.060 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:45.061 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:45.062 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:45.062 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:45.063 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:45.063 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:45.064 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:45.064 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:45.065 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:45.065 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:32:45.066 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:32:45.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.108 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:45.109 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:32:45.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.110 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:45.111 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:32:45.112 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.112 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.112 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.113 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.113 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.114 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.119 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.127 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.137 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:32:45.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:32:45.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:32:45.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:32:45.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.203 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:32:45.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:32:45.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.229 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:32:45.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:32:45.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:45.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.244 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:32:45.245 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:32:45.245 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:32:45.245 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:32:45.245 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.246 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.246 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:45.247 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:32:45.271 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.271 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.272 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.272 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.274 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.274 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.275 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.288 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 84, 2022-07-20T02:32:45.289 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.290 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.290 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:32:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:32:45.293 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.303 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:32:45.306 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:32:45.310 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.317 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:32:45.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:32:45.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.329 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:32:45.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:32:45.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:45.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:32:45.358 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:32:45.359 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:32:45.359 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:32:45.359 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.360 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.360 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:32:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.369 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.369 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.369 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.369 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.370 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.370 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.370 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.371 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:32:45.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:32:45.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.377 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:32:45.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:32:45.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.387 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:32:45.389 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:32:45.392 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.393 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.395 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:32:45.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:32:45.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:45.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.398 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:32:45.398 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:32:45.398 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:32:45.399 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:32:45.399 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.399 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.399 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.400 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:45.400 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:32:45.401 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.401 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.401 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.401 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.402 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.403 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.414 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.414 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.415 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.415 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.415 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:32:45.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:32:45.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:32:45.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:32:45.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.423 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:32:45.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:32:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:32:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:32:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:45.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:32:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:32:45.431 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:32:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:32:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.432 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.433 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:32:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.434 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.436 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:32:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:32:45.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.440 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:32:45.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:32:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:32:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:32:45.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.447 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:32:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:32:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:45.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:32:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:32:45.451 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:32:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:32:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.452 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:45.453 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:32:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.454 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.455 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:32:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:32:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.459 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.460 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:32:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:32:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:32:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:32:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:32:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:32:45.469 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:32:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:32:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:32:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:32:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.471 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:32:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.473 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.474 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.475 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:32:45.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:32:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.479 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:32:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:32:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:32:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:32:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:32:45.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:32:45.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:32:45.490 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:32:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:32:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:32:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.491 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:32:45.494 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:45.495 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:45.496 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:32:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:32:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:32:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:32:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.536 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:45.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:32:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:32:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:32:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:32:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:32:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:32:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:32:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:32:45.607 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:45.607 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:32:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.633 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:32:45.633 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:45.635 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.635 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:32:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.649 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:45.649 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.650 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.650 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.652 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.652 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:32:45.653 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.654 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:45.655 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.655 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.656 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.656 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.657 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:32:45.658 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:45.661 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.662 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.687 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.687 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:32:45.687 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.688 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:45.689 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.689 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.690 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.690 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.701 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.702 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:32:45.702 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:45.703 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.704 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.705 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.705 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.706 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:32:45.707 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:45.708 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:45.708 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:45.709 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:45.709 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:45.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:45.710 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:45.710 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:32:45.710 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:45.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.711 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:45.711 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:32:45.716 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:32:45.717 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:32:45.717 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:32:45.718 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:32:45.718 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:32:45.719 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:32:45.719 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:32:45.720 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:45.723 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:32:45.723 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:32:45.744 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:45.745 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:32:45.745 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:45.763 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:45.764 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:32:45.765 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:32:45.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.766 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:32:45.767 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:45.767 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:45.768 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:32:45.768 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:45.769 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:45.769 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:32:45.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:45.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:45.773 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:32:45.773 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:32:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:32:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:32:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:32:45.786 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 msgr2=0x7f1898055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 0x7f1898055f30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f18a800d000 tx=0x7f18a8007b80).stop 2022-07-20T02:32:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 msgr2=0x7f18b81419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f18b400c000 tx=0x7f18b40091d0).stop 2022-07-20T02:32:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 shutdown_connections 2022-07-20T02:32:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8147000 0x7f18b81419e0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:45.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1898053a70 0x7f1898055f30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:45.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8141480 0x7f18b80af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 --2- 172.21.15.26:0/2188909810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8060f00 0x7f18b80ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 >> 172.21.15.26:0/2188909810 conn(0x7f18b805cfd0 msgr2=0x7f18b80b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 shutdown_connections 2022-07-20T02:32:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:44.673+0000 7f18be571700 1 -- 172.21.15.26:0/2188909810 wait complete. 2022-07-20T02:32:45.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.937+0000 7f3fc583a700 1 Processor -- start 2022-07-20T02:32:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.937+0000 7f3fc583a700 1 -- start start 2022-07-20T02:32:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc0060920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc00aa340 con 0x7f3fc00b0070 2022-07-20T02:32:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc0062590 con 0x7f3fc0141560 2022-07-20T02:32:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0067270 con 0x7f3fc00b2c70 2022-07-20T02:32:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbeffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3fc00b0070 0x7f3fc0060210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39596/0 (socket says 172.21.15.26:39596) 2022-07-20T02:32:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbeffd700 1 -- 172.21.15.26:0/4193308199 learned_addr learned my addr 172.21.15.26:0/4193308199 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fb7fff700 1 --2- 172.21.15.26:0/4193308199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc0060920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199960674 0 0) 0x7f3fc00aa340 con 0x7f3fc00b0070 2022-07-20T02:32:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa0003260 con 0x7f3fc00b0070 2022-07-20T02:32:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fb7fff700 1 -- 172.21.15.26:0/4193308199 >> v1:172.21.15.120:6789/0 conn(0x7f3fc0141560 legacy=0x7f3fc01419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160522270 0 0) 0x7f3fc0062590 con 0x7f3fc0141560 2022-07-20T02:32:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fb7fff700 1 -- 172.21.15.26:0/4193308199 >> v1:172.21.15.26:6789/0 conn(0x7f3fc00b0070 legacy=0x7f3fc0060210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fb7fff700 1 -- 172.21.15.26:0/4193308199 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc00b0450 con 0x7f3fc00b2c70 2022-07-20T02:32:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fb7fff700 1 --2- 172.21.15.26:0/4193308199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc0060920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3fa8002f70 tx=0x7f3fa8006df0).ready entity=mon.2 client_cookie=548ceb568d2aa7d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa800d570 con 0x7f3fc00b2c70 2022-07-20T02:32:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa8000e90 con 0x7f3fc00b2c70 2022-07-20T02:32:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fbe7fc700 1 -- 172.21.15.26:0/4193308199 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa800d8c0 con 0x7f3fc00b2c70 2022-07-20T02:32:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 -- 172.21.15.26:0/4193308199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 msgr2=0x7f3fc0060920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.941+0000 7f3fc583a700 1 --2- 172.21.15.26:0/4193308199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc0060920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3fa8002f70 tx=0x7f3fa8006df0).stop 2022-07-20T02:32:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- 172.21.15.26:0/4193308199 shutdown_connections 2022-07-20T02:32:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 --2- 172.21.15.26:0/4193308199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc0060920 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- 172.21.15.26:0/4193308199 >> 172.21.15.26:0/4193308199 conn(0x7f3fc00b66e0 msgr2=0x7f3fc00b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- 172.21.15.26:0/4193308199 shutdown_connections 2022-07-20T02:32:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- 172.21.15.26:0/4193308199 wait complete. 2022-07-20T02:32:45.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 Processor -- start 2022-07-20T02:32:45.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- start start 2022-07-20T02:32:45.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fc00b0070 0x7f3fc00af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:45.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc00ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:45.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:45.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01426b0 con 0x7f3fc0141560 2022-07-20T02:32:45.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00ade30 con 0x7f3fc00b0070 2022-07-20T02:32:45.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fc583a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00ae0d0 con 0x7f3fc00b2c70 2022-07-20T02:32:45.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:45.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40326/0 (socket says 172.21.15.26:40326) 2022-07-20T02:32:45.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fbf7fe700 1 -- 172.21.15.26:0/3605161957 learned_addr learned my addr 172.21.15.26:0/3605161957 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:45.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fbeffd700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fc00b0070 0x7f3fc00af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:45.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.945+0000 7f3fb7fff700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc00ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:45.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 -- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 msgr2=0x7f3fc00ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc00ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 -- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fc00b0070 msgr2=0x7f3fc00af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fc00b0070 0x7f3fc00af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 -- 172.21.15.26:0/3605161957 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa800b180 con 0x7f3fc0141560 2022-07-20T02:32:46.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbf7fe700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f3fb8000f40 tx=0x7f3fb8007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:46.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fb800f470 con 0x7f3fc0141560 2022-07-20T02:32:46.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb800f5d0 con 0x7f3fc0141560 2022-07-20T02:32:46.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc00ae370 con 0x7f3fc0141560 2022-07-20T02:32:46.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fb800f970 con 0x7f3fc0141560 2022-07-20T02:32:46.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc01522c0 con 0x7f3fc0141560 2022-07-20T02:32:46.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc0152510 con 0x7f3fc0141560 2022-07-20T02:32:46.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3fb8019030 con 0x7f3fc0141560 2022-07-20T02:32:46.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.949+0000 7f3fbcff9700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 0x7f3fa4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.953+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3fb806b570 con 0x7f3fc0141560 2022-07-20T02:32:46.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.953+0000 7f3fbeffd700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 0x7f3fa4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.953+0000 7f3fbeffd700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 0x7f3fa4055e10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3fb000d000 tx=0x7f3fb0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:46.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:45.957+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3fc0152510 con 0x7f3fc0141560 2022-07-20T02:32:46.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.121+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc0152510 con 0x7f3fc0141560 2022-07-20T02:32:46.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.121+0000 7f3fbcff9700 1 -- 172.21.15.26:0/3605161957 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f3fc0152510 con 0x7f3fc0141560 2022-07-20T02:32:46.206 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:46.207 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":43,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:35.719385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2208502036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2208502036},{"type":"v1","addr":"172.21.15.26:6837","nonce":2208502036}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:46.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 msgr2=0x7f3fa4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 0x7f3fa4055e10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3fb000d000 tx=0x7f3fb0009040).stop 2022-07-20T02:32:46.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 msgr2=0x7f3fc01527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f3fb8000f40 tx=0x7f3fb8007ef0).stop 2022-07-20T02:32:46.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 shutdown_connections 2022-07-20T02:32:46.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fc0141560 0x7f3fc01527d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fa4053950 0x7f3fa4055e10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fc00b2c70 0x7f3fc00ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 --2- 172.21.15.26:0/3605161957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fc00b0070 0x7f3fc00af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 >> 172.21.15.26:0/3605161957 conn(0x7f3fc00b66e0 msgr2=0x7f3fc00ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:46.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 shutdown_connections 2022-07-20T02:32:46.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.125+0000 7f3fc583a700 1 -- 172.21.15.26:0/3605161957 wait complete. 2022-07-20T02:32:46.226 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 43 2022-07-20T02:32:46.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:32:46.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 Processor -- start 2022-07-20T02:32:46.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 -- start start 2022-07-20T02:32:46.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc840b5ef0 0x7fdc841474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc840a7fc0 0x7fdc841479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc840aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc84147eb0 con 0x7fdc840a7fc0 2022-07-20T02:32:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc840b62d0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc89122700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8414aac0 con 0x7fdc840a8c40 2022-07-20T02:32:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc837fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc840a7fc0 0x7fdc841479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc837fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdc840a7fc0 0x7fdc841479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40332/0 (socket says 172.21.15.26:40332) 2022-07-20T02:32:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.337+0000 7fdc837fe700 1 -- 172.21.15.26:0/3000586222 learned_addr learned my addr 172.21.15.26:0/3000586222 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc88921700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc840aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc83fff700 1 --2- 172.21.15.26:0/3000586222 >> v2:172.21.15.120:3300/0 conn(0x7fdc840b5ef0 0x7fdc841474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 -- 172.21.15.26:0/3000586222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 msgr2=0x7fdc840aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc840aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 -- 172.21.15.26:0/3000586222 >> v2:172.21.15.120:3300/0 conn(0x7fdc840b5ef0 msgr2=0x7fdc841474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 --2- 172.21.15.26:0/3000586222 >> v2:172.21.15.120:3300/0 conn(0x7fdc840b5ef0 0x7fdc841474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 -- 172.21.15.26:0/3000586222 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8414ad60 con 0x7fdc840a7fc0 2022-07-20T02:32:46.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc88921700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc840aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc837fe700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840a7fc0 0x7fdc841479c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fdc740032f0 tx=0x7fdc74009290).ready entity=mon.0 client_cookie=a589bee55683142f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc82ffd700 1 -- 172.21.15.26:0/3000586222 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc7400b870 con 0x7fdc840a7fc0 2022-07-20T02:32:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc82ffd700 1 -- 172.21.15.26:0/3000586222 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400b9d0 con 0x7fdc840a7fc0 2022-07-20T02:32:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc82ffd700 1 -- 172.21.15.26:0/3000586222 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc7400bb40 con 0x7fdc840a7fc0 2022-07-20T02:32:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- 172.21.15.26:0/3000586222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840a7fc0 msgr2=0x7fdc841479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840a7fc0 0x7fdc841479c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fdc740032f0 tx=0x7fdc74009290).stop 2022-07-20T02:32:46.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- 172.21.15.26:0/3000586222 shutdown_connections 2022-07-20T02:32:46.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc840aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 --2- 172.21.15.26:0/3000586222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840a7fc0 0x7fdc841479c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 --2- 172.21.15.26:0/3000586222 >> v2:172.21.15.120:3300/0 conn(0x7fdc840b5ef0 0x7fdc841474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- 172.21.15.26:0/3000586222 >> 172.21.15.26:0/3000586222 conn(0x7fdc84060310 msgr2=0x7fdc840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:46.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- 172.21.15.26:0/3000586222 shutdown_connections 2022-07-20T02:32:46.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- 172.21.15.26:0/3000586222 wait complete. 2022-07-20T02:32:46.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 Processor -- start 2022-07-20T02:32:46.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.341+0000 7fdc89122700 1 -- start start 2022-07-20T02:32:46.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc840a7fc0 0x7fdc840b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc84146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840ac390 con 0x7fdc840b5ef0 2022-07-20T02:32:46.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840654a0 con 0x7fdc840a7fc0 2022-07-20T02:32:46.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840abe50 con 0x7fdc840a8c40 2022-07-20T02:32:46.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40340/0 (socket says 172.21.15.26:40340) 2022-07-20T02:32:46.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 -- 172.21.15.26:0/3156399365 learned_addr learned my addr 172.21.15.26:0/3156399365 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:46.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc83fff700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc840a7fc0 0x7fdc840b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc837fe700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc84146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 -- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 msgr2=0x7fdc84146de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc84146de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 -- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc840a7fc0 msgr2=0x7fdc840b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:46.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc840a7fc0 0x7fdc840b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:46.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 -- 172.21.15.26:0/3156399365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74009530 con 0x7fdc840b5ef0 2022-07-20T02:32:46.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc88921700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fdc7c006940 tx=0x7fdc7c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:46.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc7c011780 con 0x7fdc840b5ef0 2022-07-20T02:32:46.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c0118e0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc840ac0f0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc7c011c40 con 0x7fdc840b5ef0 2022-07-20T02:32:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc840aaee0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.345+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc840ab0a0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.349+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc7c01c460 con 0x7fdc840b5ef0 2022-07-20T02:32:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.349+0000 7fdc817fa700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 0x7fdc64055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.349+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdc7c06c480 con 0x7fdc840b5ef0 2022-07-20T02:32:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.349+0000 7fdc83fff700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 0x7fdc64055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.349+0000 7fdc83fff700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 0x7fdc64055f30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fdc7800c000 tx=0x7fdc78008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.353+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc7c0430d0 con 0x7fdc840b5ef0 2022-07-20T02:32:46.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:46.517+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdc8414b000 con 0x7fdc840b5ef0 2022-07-20T02:32:47.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:47.400 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:47.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:47.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:32:47.421+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:32:47.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:32:47.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.429+0000 7fdc817fa700 1 -- 172.21.15.26:0/3156399365 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdc7c0466f0 con 0x7fdc840b5ef0 2022-07-20T02:32:47.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 msgr2=0x7fdc64055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 0x7fdc64055f30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fdc7800c000 tx=0x7fdc78008040).stop 2022-07-20T02:32:47.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 msgr2=0x7fdc841472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fdc7c006940 tx=0x7fdc7c008040).stop 2022-07-20T02:32:47.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 shutdown_connections 2022-07-20T02:32:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdc840b5ef0 0x7fdc841472d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdc64053a70 0x7fdc64055f30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc840a8c40 0x7fdc84146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 --2- 172.21.15.26:0/3156399365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc840a7fc0 0x7fdc840b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 >> 172.21.15.26:0/3156399365 conn(0x7fdc84060310 msgr2=0x7fdc840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 shutdown_connections 2022-07-20T02:32:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.433+0000 7fdc89122700 1 -- 172.21.15.26:0/3156399365 wait complete. 2022-07-20T02:32:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:32:47.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:47.465 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:32:47.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 Processor -- start 2022-07-20T02:32:47.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 -- start start 2022-07-20T02:32:47.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 0x7f12280aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f12280a7fc0 0x7f12280af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280b62d0 con 0x7f12280b5ef0 2022-07-20T02:32:47.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122814aac0 con 0x7f12280a7fc0 2022-07-20T02:32:47.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.569+0000 7f122d3e3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122814ad60 con 0x7f1228147be0 2022-07-20T02:32:47.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1227fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f12280b5ef0 0x7f12281474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39636/0 (socket says 172.21.15.26:39636) 2022-07-20T02:32:47.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1227fff700 1 -- 172.21.15.26:0/1302108462 learned_addr learned my addr 172.21.15.26:0/1302108462 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:47.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 --2- 172.21.15.26:0/1302108462 >> v2:172.21.15.120:3300/0 conn(0x7f12280a7fc0 0x7f12280af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f12277fe700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 0x7f12280aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1226ffd700 1 -- 172.21.15.26:0/1302108462 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658168290 0 0) 0x7f12280b62d0 con 0x7f12280b5ef0 2022-07-20T02:32:47.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1226ffd700 1 -- 172.21.15.26:0/1302108462 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1214003260 con 0x7f12280b5ef0 2022-07-20T02:32:47.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 -- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 msgr2=0x7f12280aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:47.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 0x7f12280aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 -- 172.21.15.26:0/1302108462 >> v1:172.21.15.26:6789/0 conn(0x7f12280b5ef0 legacy=0x7f12281474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 -- 172.21.15.26:0/1302108462 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122814b000 con 0x7f12280a7fc0 2022-07-20T02:32:47.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f12277fe700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 0x7f12280aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:47.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122cbe2700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12280a7fc0 0x7f12280af330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f12200032f0 tx=0x7f12200091d0).ready entity=mon.1 client_cookie=181914aaccd3d4cd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1226ffd700 1 -- 172.21.15.26:0/1302108462 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f122000b600 con 0x7f12280a7fc0 2022-07-20T02:32:47.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1226ffd700 1 -- 172.21.15.26:0/1302108462 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122000b760 con 0x7f12280a7fc0 2022-07-20T02:32:47.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f1226ffd700 1 -- 172.21.15.26:0/1302108462 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f122000b8d0 con 0x7f12280a7fc0 2022-07-20T02:32:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12280a7fc0 msgr2=0x7f12280af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12280a7fc0 0x7f12280af330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f12200032f0 tx=0x7f12200091d0).stop 2022-07-20T02:32:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- 172.21.15.26:0/1302108462 shutdown_connections 2022-07-20T02:32:47.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12280a7fc0 0x7f12280af330 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 --2- 172.21.15.26:0/1302108462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1228147be0 0x7f12280aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- 172.21.15.26:0/1302108462 >> 172.21.15.26:0/1302108462 conn(0x7f1228060310 msgr2=0x7f12280b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:47.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- 172.21.15.26:0/1302108462 shutdown_connections 2022-07-20T02:32:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- 172.21.15.26:0/1302108462 wait complete. 2022-07-20T02:32:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 Processor -- start 2022-07-20T02:32:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.573+0000 7f122d3e3700 1 -- start start 2022-07-20T02:32:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12280a7fc0 0x7f122805cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228147be0 0x7f12280abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228159ba0 con 0x7f12280b5ef0 2022-07-20T02:32:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12280aa7e0 con 0x7f1228147be0 2022-07-20T02:32:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1228062bc0 con 0x7f12280a7fc0 2022-07-20T02:32:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40364/0 (socket says 172.21.15.26:40364) 2022-07-20T02:32:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 -- 172.21.15.26:0/920749184 learned_addr learned my addr 172.21.15.26:0/920749184 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122cbe2700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228147be0 0x7f12280abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f1227fff700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12280a7fc0 0x7f122805cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 -- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12280a7fc0 msgr2=0x7f122805cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12280a7fc0 0x7f122805cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 -- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228147be0 msgr2=0x7f12280abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228147be0 0x7f12280abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 -- 172.21.15.26:0/920749184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121800b180 con 0x7f12280b5ef0 2022-07-20T02:32:47.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12277fe700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f121800c000 tx=0x7f1218006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1218010780 con 0x7f12280b5ef0 2022-07-20T02:32:47.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12180108e0 con 0x7f12280b5ef0 2022-07-20T02:32:47.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1220009470 con 0x7f12280b5ef0 2022-07-20T02:32:47.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1218010c40 con 0x7f12280b5ef0 2022-07-20T02:32:47.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12280aaa10 con 0x7f12280b5ef0 2022-07-20T02:32:47.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122814b2a0 con 0x7f12280b5ef0 2022-07-20T02:32:47.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.577+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f121801c460 con 0x7f12280b5ef0 2022-07-20T02:32:47.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.581+0000 7f12257fa700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 0x7f1208055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.581+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f121806c5c0 con 0x7f12280b5ef0 2022-07-20T02:32:47.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.581+0000 7f1227fff700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 0x7f1208055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.581+0000 7f1227fff700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 0x7f1208055fc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f121c00d000 tx=0x7f121c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.585+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1218043230 con 0x7f12280b5ef0 2022-07-20T02:32:47.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.745+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12280574e0 con 0x7f12280b5ef0 2022-07-20T02:32:47.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.745+0000 7f12257fa700 1 -- 172.21.15.26:0/920749184 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f1218046850 con 0x7f12280b5ef0 2022-07-20T02:32:47.766 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:47.767 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:34.694079+0000","modified":"2022-07-20T02:32:47.426620+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T02:32:47.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 msgr2=0x7f1208055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 0x7f1208055fc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f121c00d000 tx=0x7f121c009040).stop 2022-07-20T02:32:47.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 msgr2=0x7f12280ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f121800c000 tx=0x7f1218006df0).stop 2022-07-20T02:32:47.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 shutdown_connections 2022-07-20T02:32:47.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12280b5ef0 0x7f12280ab6c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1208053b00 0x7f1208055fc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228147be0 0x7f12280abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 --2- 172.21.15.26:0/920749184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12280a7fc0 0x7f122805cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 >> 172.21.15.26:0/920749184 conn(0x7f1228060310 msgr2=0x7f1228148bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:47.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 shutdown_connections 2022-07-20T02:32:47.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.749+0000 7f122d3e3700 1 -- 172.21.15.26:0/920749184 wait complete. 2022-07-20T02:32:47.819 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 44 2022-07-20T02:32:47.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:32:47.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 Processor -- start 2022-07-20T02:32:47.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 -- start start 2022-07-20T02:32:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 0x7f161c0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f161c0a7fc0 0x7f161c0af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161c0b62d0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161c14aac0 con 0x7f161c0a7fc0 2022-07-20T02:32:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c14ad60 con 0x7f161c147be0 2022-07-20T02:32:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161affd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f161c0b5ef0 0x7f161c1474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39652/0 (socket says 172.21.15.26:39652) 2022-07-20T02:32:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161affd700 1 -- 172.21.15.26:0/2688642475 learned_addr learned my addr 172.21.15.26:0/2688642475 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161a7fc700 1 --2- 172.21.15.26:0/2688642475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 0x7f161c0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161b7fe700 1 --2- 172.21.15.26:0/2688642475 >> v2:172.21.15.120:3300/0 conn(0x7f161c0a7fc0 0x7f161c0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f1619ffb700 1 -- 172.21.15.26:0/2688642475 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133369541 0 0) 0x7f161c0b62d0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f1619ffb700 1 -- 172.21.15.26:0/2688642475 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1600003260 con 0x7f161c0b5ef0 2022-07-20T02:32:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161a7fc700 1 -- 172.21.15.26:0/2688642475 >> v2:172.21.15.120:3300/0 conn(0x7f161c0a7fc0 msgr2=0x7f161c0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161a7fc700 1 --2- 172.21.15.26:0/2688642475 >> v2:172.21.15.120:3300/0 conn(0x7f161c0a7fc0 0x7f161c0af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161a7fc700 1 -- 172.21.15.26:0/2688642475 >> v1:172.21.15.26:6789/0 conn(0x7f161c0b5ef0 legacy=0x7f161c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.933+0000 7f161a7fc700 1 -- 172.21.15.26:0/2688642475 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c14b000 con 0x7f161c147be0 2022-07-20T02:32:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161a7fc700 1 --2- 172.21.15.26:0/2688642475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 0x7f161c0aee40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f160c0032f0 tx=0x7f160c0091d0).ready entity=mon.2 client_cookie=8e899fae2080900c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f1619ffb700 1 -- 172.21.15.26:0/2688642475 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f160c00b600 con 0x7f161c147be0 2022-07-20T02:32:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f1619ffb700 1 -- 172.21.15.26:0/2688642475 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c00b760 con 0x7f161c147be0 2022-07-20T02:32:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f1619ffb700 1 -- 172.21.15.26:0/2688642475 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f160c00b8d0 con 0x7f161c147be0 2022-07-20T02:32:47.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- 172.21.15.26:0/2688642475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 msgr2=0x7f161c0aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- 172.21.15.26:0/2688642475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 0x7f161c0aee40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f160c0032f0 tx=0x7f160c0091d0).stop 2022-07-20T02:32:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- 172.21.15.26:0/2688642475 shutdown_connections 2022-07-20T02:32:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- 172.21.15.26:0/2688642475 >> v2:172.21.15.120:3300/0 conn(0x7f161c0a7fc0 0x7f161c0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- 172.21.15.26:0/2688642475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c147be0 0x7f161c0aee40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- 172.21.15.26:0/2688642475 >> 172.21.15.26:0/2688642475 conn(0x7f161c060310 msgr2=0x7f161c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- 172.21.15.26:0/2688642475 shutdown_connections 2022-07-20T02:32:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- 172.21.15.26:0/2688642475 wait complete. 2022-07-20T02:32:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 Processor -- start 2022-07-20T02:32:47.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- start start 2022-07-20T02:32:47.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 0x7f161c067af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 0x7f161c0b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c0ab4f0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c15a7c0 con 0x7f161c147be0 2022-07-20T02:32:47.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c0ab720 con 0x7f161c0a7fc0 2022-07-20T02:32:47.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161a7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40378/0 (socket says 172.21.15.26:40378) 2022-07-20T02:32:47.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.937+0000 7f161a7fc700 1 -- 172.21.15.26:0/1536989450 learned_addr learned my addr 172.21.15.26:0/1536989450 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:47.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161b7fe700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 0x7f161c0b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161affd700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 0x7f161c067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 -- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 msgr2=0x7f161c067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 0x7f161c067af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 -- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 msgr2=0x7f161c0b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:47.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 0x7f161c0b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:47.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 -- 172.21.15.26:0/1536989450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c009470 con 0x7f161c0b5ef0 2022-07-20T02:32:47.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161b7fe700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 0x7f161c0b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:47.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161affd700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 0x7f161c067af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:47.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161a7fc700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f160c0032f0 tx=0x7f160c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f160c011cc0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c011e20 con 0x7f161c0b5ef0 2022-07-20T02:32:47.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f160c01f5e0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c0ab9c0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c0abc40 con 0x7f161c0b5ef0 2022-07-20T02:32:47.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c14b2a0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.941+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f160c026030 con 0x7f161c0b5ef0 2022-07-20T02:32:47.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.945+0000 7f160bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 0x7f15f8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:47.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.945+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f160c06ccc0 con 0x7f161c0b5ef0 2022-07-20T02:32:47.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.945+0000 7f161affd700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 0x7f15f8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:47.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.945+0000 7f161affd700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 0x7f15f8055ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f161000b000 tx=0x7f1610009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:47.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:47.949+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f160c0439d0 con 0x7f161c0b5ef0 2022-07-20T02:32:48.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.113+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f161c15a930 con 0x7f161c0b5ef0 2022-07-20T02:32:48.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.433+0000 7f160bfff700 1 -- 172.21.15.26:0/1536989450 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f160c0150a0 con 0x7f161c0b5ef0 2022-07-20T02:32:48.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 msgr2=0x7f15f8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 0x7f15f8055ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f161000b000 tx=0x7f1610009040).stop 2022-07-20T02:32:48.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 msgr2=0x7f161c0aae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f160c0032f0 tx=0x7f160c016000).stop 2022-07-20T02:32:48.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 shutdown_connections 2022-07-20T02:32:48.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f161c0b5ef0 0x7f161c0aae80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f15f80539e0 0x7f15f8055ea0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c147be0 0x7f161c0b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 --2- 172.21.15.26:0/1536989450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161c0a7fc0 0x7f161c067af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 >> 172.21.15.26:0/1536989450 conn(0x7f161c060310 msgr2=0x7f161c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:48.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 shutdown_connections 2022-07-20T02:32:48.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.437+0000 7f161bfff700 1 -- 172.21.15.26:0/1536989450 wait complete. 2022-07-20T02:32:48.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:32:48.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.569+0000 7f5aa9cc4700 1 Processor -- start 2022-07-20T02:32:48.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa9cc4700 1 -- start start 2022-07-20T02:32:48.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa9cc4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5aa40add30 0x7f5aa40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa9cc4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40aa1e0 con 0x7f5aa4060950 2022-07-20T02:32:48.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa9cc4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40ae110 con 0x7f5aa40af6d0 2022-07-20T02:32:48.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa9cc4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40a9f40 con 0x7f5aa40add30 2022-07-20T02:32:48.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9bfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5aa4060950 0x7f5aa4060dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39668/0 (socket says 172.21.15.26:39668) 2022-07-20T02:32:48.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9bfff700 1 -- 172.21.15.26:0/2764945488 learned_addr learned my addr 172.21.15.26:0/2764945488 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:48.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa8cc2700 1 --2- 172.21.15.26:0/2764945488 >> v2:172.21.15.120:3300/0 conn(0x7f5aa40add30 0x7f5aa40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874830510 0 0) 0x7f5aa40aa1e0 con 0x7f5aa4060950 2022-07-20T02:32:48.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90003260 con 0x7f5aa4060950 2022-07-20T02:32:48.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428973342 0 0) 0x7f5aa40ae110 con 0x7f5aa40af6d0 2022-07-20T02:32:48.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa40aa1e0 con 0x7f5aa40af6d0 2022-07-20T02:32:48.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1893192401 0 0) 0x7f5a90003260 con 0x7f5aa4060950 2022-07-20T02:32:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 >> v2:172.21.15.120:3300/0 conn(0x7f5aa40add30 msgr2=0x7f5aa40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 --2- 172.21.15.26:0/2764945488 >> v2:172.21.15.120:3300/0 conn(0x7f5aa40add30 0x7f5aa40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 >> v1:172.21.15.120:6789/0 conn(0x7f5aa40af6d0 legacy=0x7f5aa40afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40b1110 con 0x7f5aa4060950 2022-07-20T02:32:48.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f5a94001660 con 0x7f5aa4060950 2022-07-20T02:32:48.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 >> v1:172.21.15.26:6789/0 conn(0x7f5aa4060950 legacy=0x7f5aa4060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a900038e0 0x7f5a90003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004500 0x7f5a90004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 0x7f5a9000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40a9f40 con 0x7f5a90004500 2022-07-20T02:32:48.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9000bcf0 con 0x7f5a90004e70 2022-07-20T02:32:48.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9000beb0 con 0x7f5a900038e0 2022-07-20T02:32:48.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.573+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 0x7f5a9000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa8cc2700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a900038e0 0x7f5a90003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5a9bfff700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004500 0x7f5a90004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 -- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a900038e0 msgr2=0x7f5a90003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:48.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a900038e0 0x7f5a90003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 -- 172.21.15.26:0/2764945488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004500 msgr2=0x7f5a90004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:48.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004500 0x7f5a90004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 -- 172.21.15.26:0/2764945488 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90003260 con 0x7f5a90004e70 2022-07-20T02:32:48.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 -- 172.21.15.26:0/2764945488 reap_dead start 2022-07-20T02:32:48.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 0x7f5a9000b750 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f5a900054d0 tx=0x7f5a9c007d80).ready entity=mon.1 client_cookie=a49abd94c3c08075 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:48.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a9c006890 con 0x7f5a90004e70 2022-07-20T02:32:48.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c0069f0 con 0x7f5a90004e70 2022-07-20T02:32:48.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5a9b7fe700 1 -- 172.21.15.26:0/2764945488 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a9c006d50 con 0x7f5a90004e70 2022-07-20T02:32:48.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 msgr2=0x7f5a9000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 0x7f5a9000b750 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f5a900054d0 tx=0x7f5a9c007d80).stop 2022-07-20T02:32:48.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/2764945488 shutdown_connections 2022-07-20T02:32:48.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/2764945488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a90004e70 0x7f5a9000b750 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/2764945488 >> 172.21.15.26:0/2764945488 conn(0x7f5aa40b3050 msgr2=0x7f5aa40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:48.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/2764945488 shutdown_connections 2022-07-20T02:32:48.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/2764945488 wait complete. 2022-07-20T02:32:48.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 Processor -- start 2022-07-20T02:32:48.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.577+0000 7f5aa9cc4700 1 -- start start 2022-07-20T02:32:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa4063800 0x7f5aa40ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 0x7f5aa40abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40640c0 con 0x7f5a90004e70 2022-07-20T02:32:48.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40aac10 con 0x7f5aa40ab800 2022-07-20T02:32:48.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa414afc0 con 0x7f5aa4063800 2022-07-20T02:32:48.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40398/0 (socket says 172.21.15.26:40398) 2022-07-20T02:32:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 -- 172.21.15.26:0/1671411106 learned_addr learned my addr 172.21.15.26:0/1671411106 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5a9bfff700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa4063800 0x7f5aa40ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 0x7f5aa40abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 -- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa4063800 msgr2=0x7f5aa40ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa4063800 0x7f5aa40ab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 -- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 msgr2=0x7f5aa40abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 0x7f5aa40abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:48.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 -- 172.21.15.26:0/1671411106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa413c1b0 con 0x7f5a90004e70 2022-07-20T02:32:48.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa94c3700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 0x7f5aa40abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:48.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa8cc2700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f5aa000b000 tx=0x7f5aa0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:48.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aa0013680 con 0x7f5a90004e70 2022-07-20T02:32:48.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa00137e0 con 0x7f5a90004e70 2022-07-20T02:32:48.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa413cbd0 con 0x7f5a90004e70 2022-07-20T02:32:48.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aa0013b80 con 0x7f5a90004e70 2022-07-20T02:32:48.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa40aae70 con 0x7f5a90004e70 2022-07-20T02:32:48.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.581+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa40b13b0 con 0x7f5a90004e70 2022-07-20T02:32:48.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.585+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5aa0013ce0 con 0x7f5a90004e70 2022-07-20T02:32:48.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.585+0000 7f5a99ffb700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 0x7f5a84055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:48.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.585+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f5aa006c9b0 con 0x7f5a90004e70 2022-07-20T02:32:48.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.585+0000 7f5a9bfff700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 0x7f5a84055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:48.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.585+0000 7f5a9bfff700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 0x7f5a84055ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5a94005000 tx=0x7f5aa40ae860).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:48.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.593+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5aa0043610 con 0x7f5a90004e70 2022-07-20T02:32:48.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:48.713+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5aa40574e0 con 0x7f5a90004e70 2022-07-20T02:32:49.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.501+0000 7f5a99ffb700 1 -- 172.21.15.26:0/1671411106 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 185+0+0 (secure 0 0 0) 0x7f5aa0018020 con 0x7f5a90004e70 2022-07-20T02:32:49.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 msgr2=0x7f5a84055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 0x7f5a84055ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5a94005000 tx=0x7f5aa40ae860).stop 2022-07-20T02:32:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 msgr2=0x7f5aa40b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f5aa000b000 tx=0x7f5aa0006df0).stop 2022-07-20T02:32:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 shutdown_connections 2022-07-20T02:32:49.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a84053b30 0x7f5a84055ff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa40ab800 0x7f5aa40abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa4063800 0x7f5aa40ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 --2- 172.21.15.26:0/1671411106 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a90004e70 0x7f5aa40b44f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 >> 172.21.15.26:0/1671411106 conn(0x7f5aa40b3050 msgr2=0x7f5aa413d780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 shutdown_connections 2022-07-20T02:32:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.505+0000 7f5aa9cc4700 1 -- 172.21.15.26:0/1671411106 wait complete. 2022-07-20T02:32:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:32:49.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:32:49.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 Processor -- start 2022-07-20T02:32:49.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 -- start start 2022-07-20T02:32:49.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70640b3260 0x7f7064060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7064095b10 0x7f7064095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064067dc0 con 0x7f70640a95a0 2022-07-20T02:32:49.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640b3640 con 0x7f70640b3260 2022-07-20T02:32:49.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7063fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640615c0 con 0x7f7064095b10 2022-07-20T02:32:49.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7062ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70640b3260 0x7f7064060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7062ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f70640b3260 0x7f7064060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40406/0 (socket says 172.21.15.26:40406) 2022-07-20T02:32:49.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.641+0000 7f7062ffd700 1 -- 172.21.15.26:0/2751970845 learned_addr learned my addr 172.21.15.26:0/2751970845 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:49.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f70637fe700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7064095b10 0x7f7064095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7062ffd700 1 -- 172.21.15.26:0/2751970845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7064095b10 msgr2=0x7f7064095f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:49.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7062ffd700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7064095b10 0x7f7064095f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7062ffd700 1 -- 172.21.15.26:0/2751970845 >> v1:172.21.15.120:6789/0 conn(0x7f70640a95a0 legacy=0x7f70640676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7062ffd700 1 -- 172.21.15.26:0/2751970845 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064061860 con 0x7f70640b3260 2022-07-20T02:32:49.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7062ffd700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70640b3260 0x7f7064060950 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f7054002f70 tx=0x7f7054006eb0).ready entity=mon.0 client_cookie=6dd3e42a2d032b0a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:49.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7061ffb700 1 -- 172.21.15.26:0/2751970845 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705400d570 con 0x7f70640b3260 2022-07-20T02:32:49.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7061ffb700 1 -- 172.21.15.26:0/2751970845 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054000e90 con 0x7f70640b3260 2022-07-20T02:32:49.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7061ffb700 1 -- 172.21.15.26:0/2751970845 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705400d900 con 0x7f70640b3260 2022-07-20T02:32:49.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 -- 172.21.15.26:0/2751970845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70640b3260 msgr2=0x7f7064060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70640b3260 0x7f7064060950 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f7054002f70 tx=0x7f7054006eb0).stop 2022-07-20T02:32:49.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 -- 172.21.15.26:0/2751970845 shutdown_connections 2022-07-20T02:32:49.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7064095b10 0x7f7064095f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 --2- 172.21.15.26:0/2751970845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70640b3260 0x7f7064060950 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 -- 172.21.15.26:0/2751970845 >> 172.21.15.26:0/2751970845 conn(0x7f70640b67f0 msgr2=0x7f70640b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:49.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 -- 172.21.15.26:0/2751970845 shutdown_connections 2022-07-20T02:32:49.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.645+0000 7f7063fff700 1 -- 172.21.15.26:0/2751970845 wait complete. 2022-07-20T02:32:49.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 Processor -- start 2022-07-20T02:32:49.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- start start 2022-07-20T02:32:49.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 0x7f7064092d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 0x7f7064153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064093ef0 con 0x7f7064095b10 2022-07-20T02:32:49.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640939b0 con 0x7f70640a95a0 2022-07-20T02:32:49.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064093c50 con 0x7f70640b3260 2022-07-20T02:32:49.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40412/0 (socket says 172.21.15.26:40412) 2022-07-20T02:32:49.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 -- 172.21.15.26:0/969609538 learned_addr learned my addr 172.21.15.26:0/969609538 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:49.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f70627fc700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 0x7f7064092d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f70637fe700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 0x7f7064153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 -- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 msgr2=0x7f7064153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 0x7f7064153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 -- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 msgr2=0x7f7064092d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:49.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 0x7f7064092d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:49.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 -- 172.21.15.26:0/969609538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705400b1e0 con 0x7f7064095b10 2022-07-20T02:32:49.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f70627fc700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 0x7f7064092d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f70637fe700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 0x7f7064153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7062ffd700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f7054002f70 tx=0x7f7054015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7054011950 con 0x7f7064095b10 2022-07-20T02:32:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054011ab0 con 0x7f7064095b10 2022-07-20T02:32:49.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7054011e10 con 0x7f7064095b10 2022-07-20T02:32:49.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70641539b0 con 0x7f7064095b10 2022-07-20T02:32:49.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.649+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7064093200 con 0x7f7064095b10 2022-07-20T02:32:49.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f705401a040 con 0x7f7064095b10 2022-07-20T02:32:49.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064061b00 con 0x7f7064095b10 2022-07-20T02:32:49.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f7053fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 0x7f7040055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:49.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f705406cdd0 con 0x7f7064095b10 2022-07-20T02:32:49.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f70627fc700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 0x7f7040055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:49.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.653+0000 7f70627fc700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 0x7f7040055e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f705800b000 tx=0x7f7058009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:49.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.661+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7054043d30 con 0x7f7064095b10 2022-07-20T02:32:49.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:49.785+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f70640574e0 con 0x7f7064095b10 2022-07-20T02:32:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.513+0000 7f7053fff700 1 -- 172.21.15.26:0/969609538 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7054023020 con 0x7f7064095b10 2022-07-20T02:32:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 msgr2=0x7f7040055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 0x7f7040055e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f705800b000 tx=0x7f7058009040).stop 2022-07-20T02:32:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 msgr2=0x7f706405fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f7054002f70 tx=0x7f7054015000).stop 2022-07-20T02:32:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 shutdown_connections 2022-07-20T02:32:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7040053980 0x7f7040055e40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70640b3260 0x7f7064153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70640a95a0 0x7f7064092d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 --2- 172.21.15.26:0/969609538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7064095b10 0x7f706405fc30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 >> 172.21.15.26:0/969609538 conn(0x7f70640b67f0 msgr2=0x7f70640b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 shutdown_connections 2022-07-20T02:32:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.517+0000 7f7063fff700 1 -- 172.21.15.26:0/969609538 wait complete. 2022-07-20T02:32:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:32:50.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:32:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 Processor -- start 2022-07-20T02:32:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- start start 2022-07-20T02:32:50.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0b5ef0 0x7fd02c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd02c0a8c40 0x7fd02c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0b62d0 con 0x7fd02c0a7fc0 2022-07-20T02:32:50.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c14aac0 con 0x7fd02c0a8c40 2022-07-20T02:32:50.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c14ad60 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd02c0a8c40 0x7fd02c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd021d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd02c0a7fc0 0x7fd02c0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39704/0 (socket says 172.21.15.26:39704) 2022-07-20T02:32:50.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd021d9b700 1 -- 172.21.15.26:0/4167074935 learned_addr learned my addr 172.21.15.26:0/4167074935 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:50.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02a59c700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0b5ef0 0x7fd02c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 -- 172.21.15.26:0/4167074935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0b5ef0 msgr2=0x7fd02c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0b5ef0 0x7fd02c1474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 -- 172.21.15.26:0/4167074935 >> v1:172.21.15.26:6789/0 conn(0x7fd02c0a7fc0 legacy=0x7fd02c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 -- 172.21.15.26:0/4167074935 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c14b000 con 0x7fd02c0a8c40 2022-07-20T02:32:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02ad9d700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0a8c40 0x7fd02c0af550 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fd0240032f0 tx=0x7fd0240091d0).ready entity=mon.0 client_cookie=b61f8557c65b5b71 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:50.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd029d9b700 1 -- 172.21.15.26:0/4167074935 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd02400b6c0 con 0x7fd02c0a8c40 2022-07-20T02:32:50.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd029d9b700 1 -- 172.21.15.26:0/4167074935 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02400b820 con 0x7fd02c0a8c40 2022-07-20T02:32:50.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd029d9b700 1 -- 172.21.15.26:0/4167074935 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd02400b990 con 0x7fd02c0a8c40 2022-07-20T02:32:50.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- 172.21.15.26:0/4167074935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0a8c40 msgr2=0x7fd02c0af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0a8c40 0x7fd02c0af550 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fd0240032f0 tx=0x7fd0240091d0).stop 2022-07-20T02:32:50.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- 172.21.15.26:0/4167074935 shutdown_connections 2022-07-20T02:32:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0a8c40 0x7fd02c0af550 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 --2- 172.21.15.26:0/4167074935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0b5ef0 0x7fd02c1474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- 172.21.15.26:0/4167074935 >> 172.21.15.26:0/4167074935 conn(0x7fd02c060310 msgr2=0x7fd02c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- 172.21.15.26:0/4167074935 shutdown_connections 2022-07-20T02:32:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.661+0000 7fd02b59e700 1 -- 172.21.15.26:0/4167074935 wait complete. 2022-07-20T02:32:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 Processor -- start 2022-07-20T02:32:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 -- start start 2022-07-20T02:32:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 0x7fd02c15a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd02c0a8c40 0x7fd02c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c062b70 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c062760 con 0x7fd02c0a8c40 2022-07-20T02:32:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c062d60 con 0x7fd02c0a7fc0 2022-07-20T02:32:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40430/0 (socket says 172.21.15.26:40430) 2022-07-20T02:32:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 -- 172.21.15.26:0/3970913856 learned_addr learned my addr 172.21.15.26:0/3970913856 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02a59c700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 0x7fd02c15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd021d9b700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd02c0a8c40 0x7fd02c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 -- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 msgr2=0x7fd02c15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 0x7fd02c15a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 -- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd02c0a8c40 msgr2=0x7fd02c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd02c0a8c40 0x7fd02c15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02ad9d700 1 -- 172.21.15.26:0/3970913856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024009470 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.665+0000 7fd02a59c700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 0x7fd02c15a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02ad9d700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fd0240032f0 tx=0x7fd024016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd024011d70 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd024021440 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0240217a0 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c063b70 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c0640b0 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c15a5c0 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd024029070 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 0x7fd010055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd02406cd50 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02a59c700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 0x7fd010055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.669+0000 7fd02a59c700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 0x7fd010055fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd01c00b000 tx=0x7fd01c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:50.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.677+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd024043da0 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.809+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd02c14b2a0 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.813+0000 7fd023fff700 1 -- 172.21.15.26:0/3970913856 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd024015020 con 0x7fd02c0b5ef0 2022-07-20T02:32:50.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 msgr2=0x7fd010055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 0x7fd010055fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd01c00b000 tx=0x7fd01c006df0).stop 2022-07-20T02:32:50.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 msgr2=0x7fd02c0a92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fd0240032f0 tx=0x7fd024016000).stop 2022-07-20T02:32:50.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 shutdown_connections 2022-07-20T02:32:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd02c0b5ef0 0x7fd02c0a92b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd010053b00 0x7fd010055fc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd02c0a8c40 0x7fd02c15a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 --2- 172.21.15.26:0/3970913856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02c0a7fc0 0x7fd02c15a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 >> 172.21.15.26:0/3970913856 conn(0x7fd02c060310 msgr2=0x7fd02c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:50.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 shutdown_connections 2022-07-20T02:32:50.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.817+0000 7fd02b59e700 1 -- 172.21.15.26:0/3970913856 wait complete. 2022-07-20T02:32:50.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-20T02:32:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 Processor -- start 2022-07-20T02:32:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- start start 2022-07-20T02:32:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f524805cfd0 0x7f52481407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5248146e80 0x7f5248147290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524805d3b0 con 0x7f52480b1160 2022-07-20T02:32:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52481435b0 con 0x7f524805cfd0 2022-07-20T02:32:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480b4bc0 con 0x7f5248146e80 2022-07-20T02:32:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f524805cfd0 0x7f52481407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f524805cfd0 0x7f52481407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40434/0 (socket says 172.21.15.26:40434) 2022-07-20T02:32:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 -- 172.21.15.26:0/1841776250 learned_addr learned my addr 172.21.15.26:0/1841776250 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f52477fe700 1 --2- 172.21.15.26:0/1841776250 >> v2:172.21.15.120:3300/0 conn(0x7f5248146e80 0x7f5248147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 -- 172.21.15.26:0/1841776250 >> v2:172.21.15.120:3300/0 conn(0x7f5248146e80 msgr2=0x7f5248147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 --2- 172.21.15.26:0/1841776250 >> v2:172.21.15.120:3300/0 conn(0x7f5248146e80 0x7f5248147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 -- 172.21.15.26:0/1841776250 >> v1:172.21.15.120:6789/0 conn(0x7f52480b1160 legacy=0x7f5248140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 -- 172.21.15.26:0/1841776250 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480b4e60 con 0x7f524805cfd0 2022-07-20T02:32:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5246ffd700 1 --2- 172.21.15.26:0/1841776250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f524805cfd0 0x7f52481407c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5230002f70 tx=0x7f5230006eb0).ready entity=mon.0 client_cookie=c5ab323cbe34d63c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5245ffb700 1 -- 172.21.15.26:0/1841776250 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523000d610 con 0x7f524805cfd0 2022-07-20T02:32:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5245ffb700 1 -- 172.21.15.26:0/1841776250 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5230000e90 con 0x7f524805cfd0 2022-07-20T02:32:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f5245ffb700 1 -- 172.21.15.26:0/1841776250 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523000d9a0 con 0x7f524805cfd0 2022-07-20T02:32:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- 172.21.15.26:0/1841776250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f524805cfd0 msgr2=0x7f52481407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 --2- 172.21.15.26:0/1841776250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f524805cfd0 0x7f52481407c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5230002f70 tx=0x7f5230006eb0).stop 2022-07-20T02:32:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 -- 172.21.15.26:0/1841776250 shutdown_connections 2022-07-20T02:32:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 --2- 172.21.15.26:0/1841776250 >> v2:172.21.15.120:3300/0 conn(0x7f5248146e80 0x7f5248147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.941+0000 7f524d873700 1 --2- 172.21.15.26:0/1841776250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f524805cfd0 0x7f52481407c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- 172.21.15.26:0/1841776250 >> 172.21.15.26:0/1841776250 conn(0x7f5248066f70 msgr2=0x7f524805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- 172.21.15.26:0/1841776250 shutdown_connections 2022-07-20T02:32:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- 172.21.15.26:0/1841776250 wait complete. 2022-07-20T02:32:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 Processor -- start 2022-07-20T02:32:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- start start 2022-07-20T02:32:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 0x7f524813bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52480b1160 0x7f5248156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:50.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480b0080 con 0x7f5248146e80 2022-07-20T02:32:50.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480ae620 con 0x7f524805cfd0 2022-07-20T02:32:50.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f524d873700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248155ff0 con 0x7f52480b1160 2022-07-20T02:32:50.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f52477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f52477fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40444/0 (socket says 172.21.15.26:40444) 2022-07-20T02:32:50.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f52477fe700 1 -- 172.21.15.26:0/459423108 learned_addr learned my addr 172.21.15.26:0/459423108 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:50.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.945+0000 7f5246ffd700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 0x7f524813bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52467fc700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52480b1160 0x7f5248156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 -- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52480b1160 msgr2=0x7f5248156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52480b1160 0x7f5248156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 -- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 msgr2=0x7f524813bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:50.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 0x7f524813bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:50.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 -- 172.21.15.26:0/459423108 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523000b1e0 con 0x7f5248146e80 2022-07-20T02:32:50.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f5246ffd700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 0x7f524813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:50.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f52477fe700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f523800a000 tx=0x7f5238008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:50.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52380106e0 con 0x7f5248146e80 2022-07-20T02:32:50.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5238010840 con 0x7f5248146e80 2022-07-20T02:32:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52480b6ac0 con 0x7f5248146e80 2022-07-20T02:32:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5238010ba0 con 0x7f5248146e80 2022-07-20T02:32:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52480ad560 con 0x7f5248146e80 2022-07-20T02:32:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52480574e0 con 0x7f5248146e80 2022-07-20T02:32:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f523801a030 con 0x7f5248146e80 2022-07-20T02:32:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.949+0000 7f523ffff700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 0x7f5220055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.953+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f523806b840 con 0x7f5248146e80 2022-07-20T02:32:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.953+0000 7f5246ffd700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 0x7f5220055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.953+0000 7f5246ffd700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 0x7f5220055f30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f523001c000 tx=0x7f5230015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:51.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:50.957+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f52380428e0 con 0x7f5248146e80 2022-07-20T02:32:51.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.125+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7f52480b3000 con 0x7f5248146e80 2022-07-20T02:32:51.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.513+0000 7f523ffff700 1 -- 172.21.15.26:0/459423108 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v326) v1 ==== 126+0+0 (secure 0 0 0) 0x7f5238045f00 con 0x7f5248146e80 2022-07-20T02:32:51.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 msgr2=0x7f5220055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 0x7f5220055f30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f523001c000 tx=0x7f5230015000).stop 2022-07-20T02:32:51.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 msgr2=0x7f5248156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f523800a000 tx=0x7f5238008040).stop 2022-07-20T02:32:51.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 shutdown_connections 2022-07-20T02:32:51.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5248146e80 0x7f5248156980 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5220053a70 0x7f5220055f30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52480b1160 0x7f5248156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 --2- 172.21.15.26:0/459423108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524805cfd0 0x7f524813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 >> 172.21.15.26:0/459423108 conn(0x7f5248066f70 msgr2=0x7f52480b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:51.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 shutdown_connections 2022-07-20T02:32:51.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.517+0000 7f524d873700 1 -- 172.21.15.26:0/459423108 wait complete. 2022-07-20T02:32:51.544 INFO:tasks.cephfs_test_runner:test_dump_inode (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:32:51.545 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:32:51.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-20T02:32:51.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 Processor -- start 2022-07-20T02:32:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 -- start start 2022-07-20T02:32:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad0805f9e0 0x7fad081473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08060550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080b13f0 con 0x7fad080b2070 2022-07-20T02:32:51.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad08060a40 con 0x7fad0805f9e0 2022-07-20T02:32:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad07fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad080b1690 con 0x7fad080b0f50 2022-07-20T02:32:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad0805f9e0 0x7fad081473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad067fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad0805f9e0 0x7fad081473e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:60140/0 (socket says 172.21.15.26:60140) 2022-07-20T02:32:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 -- 172.21.15.26:0/3814942720 learned_addr learned my addr 172.21.15.26:0/3814942720 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad077fe700 1 --1- 172.21.15.26:0/3814942720 >> v1:172.21.15.120:6789/0 conn(0x7fad080b2070 0x7fad08066c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:53944/0 (socket says 172.21.15.26:53944) 2022-07-20T02:32:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 -- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 msgr2=0x7fad08060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 --2- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.661+0000 7fad06ffd700 1 -- 172.21.15.26:0/3814942720 >> v1:172.21.15.120:6789/0 conn(0x7fad080b2070 legacy=0x7fad08066c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad06ffd700 1 -- 172.21.15.26:0/3814942720 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080b1930 con 0x7fad0805f9e0 2022-07-20T02:32:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad06ffd700 1 --2- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad081473e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7facf80032f0 tx=0x7facf8009290).ready entity=mon.1 client_cookie=57ec28c6c322ff4f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad05ffb700 1 -- 172.21.15.26:0/3814942720 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf800b8a0 con 0x7fad0805f9e0 2022-07-20T02:32:51.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad05ffb700 1 -- 172.21.15.26:0/3814942720 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf800ba00 con 0x7fad0805f9e0 2022-07-20T02:32:51.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad05ffb700 1 -- 172.21.15.26:0/3814942720 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf800bb70 con 0x7fad0805f9e0 2022-07-20T02:32:51.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 msgr2=0x7fad081473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 --2- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad081473e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7facf80032f0 tx=0x7facf8009290).stop 2022-07-20T02:32:51.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- 172.21.15.26:0/3814942720 shutdown_connections 2022-07-20T02:32:51.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 --2- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 --2- 172.21.15.26:0/3814942720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad081473e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- 172.21.15.26:0/3814942720 >> 172.21.15.26:0/3814942720 conn(0x7fad080b3050 msgr2=0x7fad080b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- 172.21.15.26:0/3814942720 shutdown_connections 2022-07-20T02:32:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- 172.21.15.26:0/3814942720 wait complete. 2022-07-20T02:32:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 Processor -- start 2022-07-20T02:32:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.665+0000 7fad07fff700 1 -- start start 2022-07-20T02:32:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad080b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 0x7fad081487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad08066060 con 0x7fad080b2070 2022-07-20T02:32:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad08065530 con 0x7fad0805f9e0 2022-07-20T02:32:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad08157570 con 0x7fad080b0f50 2022-07-20T02:32:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 0x7fad081487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad067fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad067fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:33738/0 (socket says 172.21.15.26:33738) 2022-07-20T02:32:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad06ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad080b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad067fc700 1 -- 172.21.15.26:0/1218835478 learned_addr learned my addr 172.21.15.26:0/1218835478 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:51.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 -- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 msgr2=0x7fad08146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 -- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 msgr2=0x7fad080b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad080b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 -- 172.21.15.26:0/1218835478 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8009530 con 0x7fad080b2070 2022-07-20T02:32:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad067fc700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad077fe700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 0x7fad081487e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fad080667c0 tx=0x7fad00007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:51.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad0000ccb0 con 0x7fad080b2070 2022-07-20T02:32:51.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0000ce10 con 0x7fad080b2070 2022-07-20T02:32:51.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad081568f0 con 0x7fad080b2070 2022-07-20T02:32:51.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad000072c0 con 0x7fad080b2070 2022-07-20T02:32:51.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad081570d0 con 0x7fad080b2070 2022-07-20T02:32:51.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.669+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad08157290 con 0x7fad080b2070 2022-07-20T02:32:51.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.673+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad0001a060 con 0x7fad080b2070 2022-07-20T02:32:51.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.673+0000 7facfffff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 0x7face0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:51.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.673+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fad0006ae20 con 0x7fad080b2070 2022-07-20T02:32:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.673+0000 7fad06ffd700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 0x7face0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.673+0000 7fad06ffd700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 0x7face0055f30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7facf801b000 tx=0x7facf8016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.681+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad00041ec0 con 0x7fad080b2070 2022-07-20T02:32:51.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:51.853+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7fad08156a80 con 0x7fad080b2070 2022-07-20T02:32:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.517+0000 7facfffff700 1 -- 172.21.15.26:0/1218835478 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v327) v1 ==== 141+0+0 (secure 0 0 0) 0x7fad00018070 con 0x7fad080b2070 2022-07-20T02:32:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 msgr2=0x7face0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 0x7face0055f30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7facf801b000 tx=0x7facf8016000).stop 2022-07-20T02:32:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 msgr2=0x7fad081487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 0x7fad081487e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fad080667c0 tx=0x7fad00007b60).stop 2022-07-20T02:32:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 shutdown_connections 2022-07-20T02:32:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad080b2070 0x7fad081487e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7face0053a70 0x7face0055f30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080b0f50 0x7fad08146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 --2- 172.21.15.26:0/1218835478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad0805f9e0 0x7fad080b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 >> 172.21.15.26:0/1218835478 conn(0x7fad080b3050 msgr2=0x7fad0813fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 shutdown_connections 2022-07-20T02:32:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.521+0000 7fad07fff700 1 -- 172.21.15.26:0/1218835478 wait complete. 2022-07-20T02:32:52.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:32:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.661+0000 7f830a91b700 1 Processor -- start 2022-07-20T02:32:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 -- start start 2022-07-20T02:32:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f83040b2aa0 0x7f83041438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8304095800 0x7f8304143db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830405f840 con 0x7f83040a95a0 2022-07-20T02:32:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040b2e80 con 0x7f83040b2aa0 2022-07-20T02:32:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f830a91b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040b0a50 con 0x7f8304095800 2022-07-20T02:32:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f83040b2aa0 0x7f83041438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f83040b2aa0 0x7f83041438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40464/0 (socket says 172.21.15.26:40464) 2022-07-20T02:32:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 -- 172.21.15.26:0/1237589891 learned_addr learned my addr 172.21.15.26:0/1237589891 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309118700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8304095800 0x7f8304143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 -- 172.21.15.26:0/1237589891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8304095800 msgr2=0x7f8304143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8304095800 0x7f8304143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8308917700 1 -- 172.21.15.26:0/1237589891 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129423952 0 0) 0x7f830405f840 con 0x7f83040a95a0 2022-07-20T02:32:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 -- 172.21.15.26:0/1237589891 >> v1:172.21.15.120:6789/0 conn(0x7f83040a95a0 legacy=0x7f83040b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 -- 172.21.15.26:0/1237589891 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830405fa70 con 0x7f83040b2aa0 2022-07-20T02:32:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8309919700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040b2aa0 0x7f83041438c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f8300002f70 tx=0x7f8300006eb0).ready entity=mon.0 client_cookie=6c9d4e0ce46aee6a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8308917700 1 -- 172.21.15.26:0/1237589891 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830000d570 con 0x7f83040b2aa0 2022-07-20T02:32:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8308917700 1 -- 172.21.15.26:0/1237589891 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8300000e90 con 0x7f83040b2aa0 2022-07-20T02:32:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.665+0000 7f8308917700 1 -- 172.21.15.26:0/1237589891 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830000d8c0 con 0x7f83040b2aa0 2022-07-20T02:32:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- 172.21.15.26:0/1237589891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040b2aa0 msgr2=0x7f83041438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040b2aa0 0x7f83041438c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f8300002f70 tx=0x7f8300006eb0).stop 2022-07-20T02:32:52.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- 172.21.15.26:0/1237589891 shutdown_connections 2022-07-20T02:32:52.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8304095800 0x7f8304143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- 172.21.15.26:0/1237589891 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040b2aa0 0x7f83041438c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- 172.21.15.26:0/1237589891 >> 172.21.15.26:0/1237589891 conn(0x7f8304063030 msgr2=0x7f8304063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:52.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- 172.21.15.26:0/1237589891 shutdown_connections 2022-07-20T02:32:52.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- 172.21.15.26:0/1237589891 wait complete. 2022-07-20T02:32:52.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 Processor -- start 2022-07-20T02:32:52.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- start start 2022-07-20T02:32:52.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8304095800 0x7f830413dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b2aa0 0x7f8304147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040634f0 con 0x7f83040a95a0 2022-07-20T02:32:52.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830413e970 con 0x7f8304095800 2022-07-20T02:32:52.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a91b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8304095160 con 0x7f83040b2aa0 2022-07-20T02:32:52.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f8309118700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f8309118700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40468/0 (socket says 172.21.15.26:40468) 2022-07-20T02:32:52.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f8309118700 1 -- 172.21.15.26:0/895195783 learned_addr learned my addr 172.21.15.26:0/895195783 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:52.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f8309919700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8304095800 0x7f830413dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.669+0000 7f830a11a700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b2aa0 0x7f8304147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 -- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b2aa0 msgr2=0x7f8304147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b2aa0 0x7f8304147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 -- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8304095800 msgr2=0x7f830413dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8304095800 0x7f830413dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 -- 172.21.15.26:0/895195783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830000b1e0 con 0x7f83040a95a0 2022-07-20T02:32:52.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f8309118700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f8304063630 tx=0x7f82f4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82f4010780 con 0x7f83040a95a0 2022-07-20T02:32:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f400b3b0 con 0x7f83040a95a0 2022-07-20T02:32:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830413f730 con 0x7f83040a95a0 2022-07-20T02:32:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82f4010ad0 con 0x7f83040a95a0 2022-07-20T02:32:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304097ed0 con 0x7f83040a95a0 2022-07-20T02:32:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8304146980 con 0x7f83040a95a0 2022-07-20T02:32:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.673+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f82f4010c30 con 0x7f83040a95a0 2022-07-20T02:32:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.677+0000 7f82f2ffd700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 0x7f82e4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.677+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f82f406c140 con 0x7f83040a95a0 2022-07-20T02:32:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.677+0000 7f8309919700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 0x7f82e4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:52.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.677+0000 7f8309919700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 0x7f82e4055fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f830001c000 tx=0x7f8300015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:52.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.685+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f82f40430d0 con 0x7f83040a95a0 2022-07-20T02:32:52.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.817+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f830413f8c0 con 0x7f83040a95a0 2022-07-20T02:32:52.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f82f2ffd700 1 -- 172.21.15.26:0/895195783 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f82f40466f0 con 0x7f83040a95a0 2022-07-20T02:32:52.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 msgr2=0x7f82e4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 0x7f82e4055fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f830001c000 tx=0x7f8300015000).stop 2022-07-20T02:32:52.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 msgr2=0x7f83041477d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:52.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f8304063630 tx=0x7f82f4006df0).stop 2022-07-20T02:32:52.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 shutdown_connections 2022-07-20T02:32:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83040a95a0 0x7f83041477d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f82e4053b00 0x7f82e4055fc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83040b2aa0 0x7f8304147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 --2- 172.21.15.26:0/895195783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8304095800 0x7f830413dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 >> 172.21.15.26:0/895195783 conn(0x7f8304063030 msgr2=0x7f83040b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 shutdown_connections 2022-07-20T02:32:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:52.825+0000 7f830a91b700 1 -- 172.21.15.26:0/895195783 wait complete. 2022-07-20T02:32:52.851 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:32:52.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:52.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:52.915 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:52.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:32:52.979 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:32:52.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:52.982 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:52.982 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:32:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:32:53.007 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:32:53.008 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:32:53.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:32:53.009 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:32:53.009 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:32:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:32:53.039 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:32:53.040 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:32:53.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:32:53.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.149+0000 7fa9dffff700 1 Processor -- start 2022-07-20T02:32:53.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.149+0000 7fa9dffff700 1 -- start start 2022-07-20T02:32:53.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9e0060f00 0x7fa9e013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e01474e0 con 0x7fa9e005cfd0 2022-07-20T02:32:53.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e00b4e40 con 0x7fa9e0060f00 2022-07-20T02:32:53.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e005d3b0 con 0x7fa9e0147000 2022-07-20T02:32:53.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9deffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9e005cfd0 0x7fa9e00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39756/0 (socket says 172.21.15.26:39756) 2022-07-20T02:32:53.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9deffd700 1 -- 172.21.15.26:0/1967201717 learned_addr learned my addr 172.21.15.26:0/1967201717 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:53.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9de7fc700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/1967201717 >> v2:172.21.15.120:3300/0 conn(0x7fa9e0060f00 0x7fa9e013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9ddffb700 1 -- 172.21.15.26:0/1967201717 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379058405 0 0) 0x7fa9e01474e0 con 0x7fa9e005cfd0 2022-07-20T02:32:53.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9ddffb700 1 -- 172.21.15.26:0/1967201717 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c4003260 con 0x7fa9e005cfd0 2022-07-20T02:32:53.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 -- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 msgr2=0x7fa9e0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:53.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 -- 172.21.15.26:0/1967201717 >> v1:172.21.15.26:6789/0 conn(0x7fa9e005cfd0 legacy=0x7fa9e00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 -- 172.21.15.26:0/1967201717 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e01431e0 con 0x7fa9e0060f00 2022-07-20T02:32:53.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e013c9b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fa9d80032f0 tx=0x7fa9d80091d0).ready entity=mon.1 client_cookie=539dfdae047cbaaa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:53.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9ddffb700 1 -- 172.21.15.26:0/1967201717 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d800b600 con 0x7fa9e0060f00 2022-07-20T02:32:53.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9ddffb700 1 -- 172.21.15.26:0/1967201717 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d800b760 con 0x7fa9e0060f00 2022-07-20T02:32:53.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9ddffb700 1 -- 172.21.15.26:0/1967201717 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d800b8d0 con 0x7fa9e0060f00 2022-07-20T02:32:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 msgr2=0x7fa9e013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e013c9b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fa9d80032f0 tx=0x7fa9d80091d0).stop 2022-07-20T02:32:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- 172.21.15.26:0/1967201717 shutdown_connections 2022-07-20T02:32:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e013c9b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 --2- 172.21.15.26:0/1967201717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- 172.21.15.26:0/1967201717 >> 172.21.15.26:0/1967201717 conn(0x7fa9e00620c0 msgr2=0x7fa9e005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- 172.21.15.26:0/1967201717 shutdown_connections 2022-07-20T02:32:53.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.153+0000 7fa9dffff700 1 -- 172.21.15.26:0/1967201717 wait complete. 2022-07-20T02:32:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.157+0000 7fa9dffff700 1 Processor -- start 2022-07-20T02:32:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.157+0000 7fa9dffff700 1 -- start start 2022-07-20T02:32:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.157+0000 7fa9dffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e0142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.157+0000 7fa9dffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e00ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00aeb50 con 0x7fa9e014a070 2022-07-20T02:32:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00af640 con 0x7fa9e0060f00 2022-07-20T02:32:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0152180 con 0x7fa9e0147000 2022-07-20T02:32:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9deffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e0142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40486/0 (socket says 172.21.15.26:40486) 2022-07-20T02:32:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9de7fc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e00ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 -- 172.21.15.26:0/4214449 learned_addr learned my addr 172.21.15.26:0/4214449 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 -- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 msgr2=0x7fa9e00ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e00ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 -- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 msgr2=0x7fa9e0142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e0142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 -- 172.21.15.26:0/4214449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d8009470 con 0x7fa9e014a070 2022-07-20T02:32:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9df7fe700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fa9d80032f0 tx=0x7fa9d8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d800b600 con 0x7fa9e014a070 2022-07-20T02:32:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d8011e00 con 0x7fa9e014a070 2022-07-20T02:32:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0149c90 con 0x7fa9e014a070 2022-07-20T02:32:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d8021630 con 0x7fa9e014a070 2022-07-20T02:32:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e01523e0 con 0x7fa9e014a070 2022-07-20T02:32:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.161+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e00ad830 con 0x7fa9e014a070 2022-07-20T02:32:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.165+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9d8027020 con 0x7fa9e014a070 2022-07-20T02:32:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.165+0000 7fa9cffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 0x7fa9bc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.165+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa9d806d350 con 0x7fa9e014a070 2022-07-20T02:32:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.165+0000 7fa9deffd700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 0x7fa9bc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.165+0000 7fa9deffd700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 0x7fa9bc055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa9d000d000 tx=0x7fa9d0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.173+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9d80442e0 con 0x7fa9e014a070 2022-07-20T02:32:53.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.301+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa9e0143480 con 0x7fa9e014a070 2022-07-20T02:32:53.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.301+0000 7fa9cffff700 1 -- 172.21.15.26:0/4214449 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v90) v1 ==== 81+0+29268 (secure 0 0 0) 0x7fa9d8015090 con 0x7fa9e014a070 2022-07-20T02:32:53.326 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:53.327 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:53.327 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 90, 2022-07-20T02:32:53.328 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:32:53.328 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:32:53.329 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:32:50.501278+0000", 2022-07-20T02:32:53.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:53.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:32:53.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:32:53.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:32:53.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:32:53.332 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:32:53.333 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:32:53.333 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:32:53.334 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:32:53.335 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:53.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:32:53.336 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:32:53.336 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:32:53.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:32:53.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:32:53.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 16, 2022-07-20T02:32:53.338 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:32:53.338 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:32:53.339 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:32:53.340 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:32:53.341 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:32:53.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.342 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:32:53.343 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:32:53.343 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:32:53.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:53.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:53.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:53.345 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:53.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:53.347 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:53.347 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:53.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:53.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:32:53.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:32:53.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:32:53.350 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:32:53.351 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:32:53.352 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:53.352 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:53.353 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:53.353 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:53.353 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:53.354 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:53.355 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:53.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.356 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:32:53.356 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:53.357 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:53.358 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:53.359 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:53.359 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:53.363 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:53.363 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:53.364 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:53.367 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:53.367 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:53.367 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:53.368 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:53.368 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:53.368 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:53.368 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:53.368 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:53.369 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:53.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:53.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:53.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:53.370 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:53.370 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:53.370 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:53.379 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:53.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:53.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:53.380 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:53.381 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:53.381 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:53.381 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:53.381 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:53.382 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:53.382 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:32:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:32:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.383 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:53.384 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:32:53.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.385 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:32:53.385 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:32:53.385 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:32:53.386 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:32:53.386 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:32:53.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:32:53.386 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:32:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:32:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:32:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:32:53.387 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:32:53.388 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:32:53.388 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:32:53.388 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:32:53.394 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:32:53.394 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:32:53.394 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:32:53.395 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:32:53.396 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:32:53.396 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:32:53.397 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:32:53.397 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:32:53.397 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:32:53.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.398 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:32:53.399 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:32:53.399 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:32:53.399 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:32:53.400 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:32:53.400 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:32:53.400 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:32:53.400 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:32:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:32:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:32:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:32:53.401 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:32:53.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:32:53.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:32:53.402 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:32:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:32:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:32:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:32:53.403 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:32:53.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:32:53.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:32:53.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:32:53.405 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:32:53.405 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:32:53.405 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:32:53.405 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:32:53.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:32:53.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.406 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:32:53.406 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:32:53.407 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:32:53.407 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:32:53.408 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:32:53.408 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:32:53.408 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:32:53.408 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:32:53.409 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:32:53.409 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:32:53.409 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:32:53.409 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:32:53.410 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:32:53.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:32:53.410 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.410 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.411 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:53.411 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:32:53.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.411 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:53.412 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:32:53.412 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.412 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.412 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.413 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.413 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.413 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.413 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.414 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.414 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.414 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.414 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:32:53.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:32:53.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.429 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:32:53.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:32:53.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.432 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:32:53.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:32:53.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.441 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:32:53.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:32:53.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:32:53.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.445 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:32:53.445 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:32:53.446 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:32:53.446 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:32:53.446 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.446 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.447 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:32:53.448 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 84, 2022-07-20T02:32:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.450 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:32:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.452 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:32:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.453 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:32:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:32:53.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.457 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:32:53.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:32:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:32:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:32:53.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:32:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:32:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:32:53.468 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:32:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:32:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.469 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.470 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:32:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.474 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.476 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.477 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.478 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:32:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:32:53.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.491 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:32:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:32:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:32:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:32:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:32:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:32:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:32:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:32:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:32:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:32:53.502 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:32:53.503 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.503 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:32:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.505 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.506 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.506 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.506 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.507 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.507 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:32:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:32:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:32:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:32:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:32:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:32:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.536 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:32:53.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:32:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:32:53.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:32:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:32:53.580 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:32:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:32:53.581 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.582 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.583 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:32:53.610 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.611 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.612 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.613 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.618 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.619 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.632 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:32:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:32:53.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:32:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:32:53.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.655 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:32:53.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:32:53.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.665 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:32:53.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:32:53.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:32:53.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:32:53.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:32:53.672 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:32:53.672 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:32:53.673 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.673 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.675 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.677 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:53.680 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:32:53.681 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.681 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.682 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.682 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.682 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.683 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.683 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.684 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.684 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.691 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.691 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:32:53.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:32:53.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.703 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:32:53.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:32:53.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.738 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:32:53.740 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:32:53.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.743 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:32:53.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.745 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:32:53.745 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:32:53.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.746 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:32:53.746 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:32:53.746 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:32:53.746 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:32:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.749 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:53.750 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:32:53.750 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.750 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.751 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.752 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:32:53.752 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:32:53.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:32:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:32:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:32:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.766 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:32:53.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:32:53.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.772 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:32:53.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:32:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:32:53.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:32:53.776 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:32:53.777 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:32:53.778 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:32:53.778 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.778 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.778 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.780 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:32:53.787 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:32:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:32:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:32:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:32:53.788 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:32:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:32:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:32:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 73, 2022-07-20T02:32:53.789 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:32:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:32:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:32:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:32:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:32:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:32:53.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:32:53.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:32:53.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.804 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:32:53.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:32:53.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:32:53.807 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.808 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:32:53.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:32:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:32:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:32:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:32:53.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:32:53.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:32:53.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:32:53.812 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:32:53.812 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:32:53.812 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:32:53.813 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:32:53.813 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:32:53.813 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:32:53.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:53.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.814 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:53.814 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:32:53.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.815 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:32:53.815 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.815 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.816 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.816 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.816 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.816 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:32:53.817 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.817 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:32:53.818 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.818 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.818 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.818 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.824 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.825 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:32:53.825 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:32:53.826 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.826 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.826 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.827 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.827 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.827 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:32:53.827 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.828 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:32:53.828 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.828 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.829 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.829 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.829 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.829 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:32:53.830 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:32:53.831 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.832 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.832 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.833 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.833 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:32:53.834 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:32:53.835 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:32:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:32:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:32:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:32:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:32:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:32:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:32:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:32:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:32:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:32:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:32:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:32:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:32:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:32:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:32:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:32:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:32:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/2208502036": "2022-07-21T02:32:46.524716+0000", 2022-07-20T02:32:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/2208502036": "2022-07-21T02:32:46.524716+0000" 2022-07-20T02:32:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:32:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:32:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:32:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:32:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:32:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:32:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:32:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:32:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:32:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:32:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:32:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:32:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:32:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:32:53.849 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:32:53.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:53.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:53.849 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:32:53.850 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:32:53.850 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:32:53.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:32:53.850 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:32:53.851 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 msgr2=0x7fa9bc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 0x7fa9bc055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa9d000d000 tx=0x7fa9d0009040).stop 2022-07-20T02:32:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 msgr2=0x7fa9e0152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fa9d80032f0 tx=0x7fa9d8016000).stop 2022-07-20T02:32:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 shutdown_connections 2022-07-20T02:32:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9e014a070 0x7fa9e0152880 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9bc053a70 0x7fa9bc055f30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9e0147000 0x7fa9e00ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 --2- 172.21.15.26:0/4214449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e0060f00 0x7fa9e0142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 >> 172.21.15.26:0/4214449 conn(0x7fa9e005cfd0 msgr2=0x7fa9e013afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 shutdown_connections 2022-07-20T02:32:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.305+0000 7fa9dffff700 1 -- 172.21.15.26:0/4214449 wait complete. 2022-07-20T02:32:53.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:53.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 Processor -- start 2022-07-20T02:32:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- start start 2022-07-20T02:32:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5fe00b2920 0x7f5fe0147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0067190 0x7f5fe0148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b1bc0 con 0x7f5fe0061050 2022-07-20T02:32:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe014b040 con 0x7f5fe00b2920 2022-07-20T02:32:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00a8860 con 0x7f5fe0067190 2022-07-20T02:32:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5fe00b2920 0x7f5fe0147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5fe0061050 0x7f5fe0066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39770/0 (socket says 172.21.15.26:39770) 2022-07-20T02:32:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdffff700 1 -- 172.21.15.26:0/511555146 learned_addr learned my addr 172.21.15.26:0/511555146 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe5167700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0067190 0x7f5fe0148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdf7fe700 1 -- 172.21.15.26:0/511555146 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743227297 0 0) 0x7f5fe00b1bc0 con 0x7f5fe0061050 2022-07-20T02:32:54.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 -- 172.21.15.26:0/511555146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0067190 msgr2=0x7f5fe0148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:54.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0067190 0x7f5fe0148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 -- 172.21.15.26:0/511555146 >> v1:172.21.15.26:6789/0 conn(0x7f5fe0061050 legacy=0x7f5fe0066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 -- 172.21.15.26:0/511555146 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0147700 con 0x7f5fe00b2920 2022-07-20T02:32:54.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe4966700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe00b2920 0x7f5fe0147d90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8006df0).ready entity=mon.0 client_cookie=ad21bfd350ed6c6b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdf7fe700 1 -- 172.21.15.26:0/511555146 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc800d570 con 0x7f5fe00b2920 2022-07-20T02:32:54.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdf7fe700 1 -- 172.21.15.26:0/511555146 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8000e90 con 0x7f5fe00b2920 2022-07-20T02:32:54.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fdf7fe700 1 -- 172.21.15.26:0/511555146 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc800d8c0 con 0x7f5fe00b2920 2022-07-20T02:32:54.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- 172.21.15.26:0/511555146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe00b2920 msgr2=0x7f5fe0147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe00b2920 0x7f5fe0147d90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8006df0).stop 2022-07-20T02:32:54.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- 172.21.15.26:0/511555146 shutdown_connections 2022-07-20T02:32:54.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0067190 0x7f5fe0148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/511555146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe00b2920 0x7f5fe0147d90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- 172.21.15.26:0/511555146 >> 172.21.15.26:0/511555146 conn(0x7f5fe00b60a0 msgr2=0x7f5fe00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:54.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- 172.21.15.26:0/511555146 shutdown_connections 2022-07-20T02:32:54.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.989+0000 7f5fe6bca700 1 -- 172.21.15.26:0/511555146 wait complete. 2022-07-20T02:32:54.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 Processor -- start 2022-07-20T02:32:54.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 -- start start 2022-07-20T02:32:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0061050 0x7f5fe014d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 0x7f5fe00aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe015ad40 con 0x7f5fe0067190 2022-07-20T02:32:54.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe0065110 con 0x7f5fe00b2920 2022-07-20T02:32:54.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe6bca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe0064bd0 con 0x7f5fe0061050 2022-07-20T02:32:54.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40500/0 (socket says 172.21.15.26:40500) 2022-07-20T02:32:54.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 -- 172.21.15.26:0/3392546213 learned_addr learned my addr 172.21.15.26:0/3392546213 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:54.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe4966700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0061050 0x7f5fe014d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe5167700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 0x7f5fe00aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 -- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0061050 msgr2=0x7f5fe014d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0061050 0x7f5fe014d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 -- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 msgr2=0x7f5fe00aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 0x7f5fe00aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 -- 172.21.15.26:0/3392546213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc800b180 con 0x7f5fe0067190 2022-07-20T02:32:54.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fe5167700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 0x7f5fe00aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:54.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.993+0000 7f5fdffff700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5fd0001000 tx=0x7f5fd0007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd00070d0 con 0x7f5fe0067190 2022-07-20T02:32:54.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd0007230 con 0x7f5fe0067190 2022-07-20T02:32:54.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe0064e70 con 0x7f5fe0067190 2022-07-20T02:32:54.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd00055d0 con 0x7f5fe0067190 2022-07-20T02:32:54.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe0065f00 con 0x7f5fe0067190 2022-07-20T02:32:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe014ada0 con 0x7f5fe0067190 2022-07-20T02:32:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fd001a060 con 0x7f5fe0067190 2022-07-20T02:32:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 0x7f5fb8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5fd006b3e0 con 0x7f5fe0067190 2022-07-20T02:32:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fe4966700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 0x7f5fb8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:53.997+0000 7f5fe4966700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 0x7f5fb8055ff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5fc801c000 tx=0x7f5fc8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.005+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fd0042480 con 0x7f5fe0067190 2022-07-20T02:32:54.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.165+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fe00574e0 con 0x7f5fe0067190 2022-07-20T02:32:54.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fddffb700 1 -- 172.21.15.26:0/3392546213 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f5fd0018070 con 0x7f5fe0067190 2022-07-20T02:32:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:54.193 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6957,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/961932243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":961932243},{"type":"v1","addr":"172.21.15.26:6837","nonce":961932243}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[]} 2022-07-20T02:32:54.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 msgr2=0x7f5fb8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 0x7f5fb8055ff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5fc801c000 tx=0x7f5fc8015000).stop 2022-07-20T02:32:54.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 msgr2=0x7f5fe00aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5fd0001000 tx=0x7f5fd0007c30).stop 2022-07-20T02:32:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 shutdown_connections 2022-07-20T02:32:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5fe0067190 0x7f5fe00aaef0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5fb8053b30 0x7f5fb8055ff0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fe00b2920 0x7f5fe00aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 --2- 172.21.15.26:0/3392546213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fe0061050 0x7f5fe014d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 >> 172.21.15.26:0/3392546213 conn(0x7f5fe00b60a0 msgr2=0x7f5fe0148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 shutdown_connections 2022-07-20T02:32:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.169+0000 7f5fe6bca700 1 -- 172.21.15.26:0/3392546213 wait complete. 2022-07-20T02:32:54.209 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 45 2022-07-20T02:32:54.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:32:54.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 Processor -- start 2022-07-20T02:32:54.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 -- start start 2022-07-20T02:32:54.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa0e80a7fc0 0x7fa0e81479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 0x7fa0e80aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e8147eb0 con 0x7fa0e80a7fc0 2022-07-20T02:32:54.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e80b62d0 con 0x7fa0e80a8c40 2022-07-20T02:32:54.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed8df700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e814aac0 con 0x7fa0e80b5ef0 2022-07-20T02:32:54.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa0e80a7fc0 0x7fa0e81479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa0e80a7fc0 0x7fa0e81479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40504/0 (socket says 172.21.15.26:40504) 2022-07-20T02:32:54.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 -- 172.21.15.26:0/2402021429 learned_addr learned my addr 172.21.15.26:0/2402021429 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:54.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed0de700 1 --2- 172.21.15.26:0/2402021429 >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 0x7fa0e80aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 -- 172.21.15.26:0/2402021429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 msgr2=0x7fa0e81474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:54.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 -- 172.21.15.26:0/2402021429 >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 msgr2=0x7fa0e80aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:54.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 --2- 172.21.15.26:0/2402021429 >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 0x7fa0e80aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 -- 172.21.15.26:0/2402021429 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e814ad60 con 0x7fa0e80a7fc0 2022-07-20T02:32:54.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ed0de700 1 --2- 172.21.15.26:0/2402021429 >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 0x7fa0e80aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:54.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:54.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e7fff700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e80a7fc0 0x7fa0e81479c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fa0d80032f0 tx=0x7fa0d8009290).ready entity=mon.0 client_cookie=75d7a70f6ad4fa1a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e77fe700 1 -- 172.21.15.26:0/2402021429 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0d800b870 con 0x7fa0e80a7fc0 2022-07-20T02:32:54.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.317+0000 7fa0e77fe700 1 -- 172.21.15.26:0/2402021429 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d800b9d0 con 0x7fa0e80a7fc0 2022-07-20T02:32:54.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0e77fe700 1 -- 172.21.15.26:0/2402021429 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0d800bb40 con 0x7fa0e80a7fc0 2022-07-20T02:32:54.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- 172.21.15.26:0/2402021429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e80a7fc0 msgr2=0x7fa0e81479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e80a7fc0 0x7fa0e81479c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fa0d80032f0 tx=0x7fa0d8009290).stop 2022-07-20T02:32:54.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- 172.21.15.26:0/2402021429 shutdown_connections 2022-07-20T02:32:54.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/2402021429 >> v2:172.21.15.120:3300/0 conn(0x7fa0e80a8c40 0x7fa0e80aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e80a7fc0 0x7fa0e81479c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/2402021429 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- 172.21.15.26:0/2402021429 >> 172.21.15.26:0/2402021429 conn(0x7fa0e8060310 msgr2=0x7fa0e80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:54.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- 172.21.15.26:0/2402021429 shutdown_connections 2022-07-20T02:32:54.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- 172.21.15.26:0/2402021429 wait complete. 2022-07-20T02:32:54.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 Processor -- start 2022-07-20T02:32:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- start start 2022-07-20T02:32:54.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 0x7fa0e80adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e80ab0c0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e80ac3e0 con 0x7fa0e80a8c40 2022-07-20T02:32:54.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed8df700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e80aa730 con 0x7fa0e80b5ef0 2022-07-20T02:32:54.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed0de700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed0de700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40514/0 (socket says 172.21.15.26:40514) 2022-07-20T02:32:54.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.321+0000 7fa0ed0de700 1 -- 172.21.15.26:0/156167944 learned_addr learned my addr 172.21.15.26:0/156167944 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:54.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 0x7fa0e80adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e7fff700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 -- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 msgr2=0x7fa0e81436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 -- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 msgr2=0x7fa0e80adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 0x7fa0e80adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 -- 172.21.15.26:0/156167944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8009530 con 0x7fa0e8143ba0 2022-07-20T02:32:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e7fff700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 0x7fa0e80adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:54.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed0de700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7fa0e000a000 tx=0x7fa0e0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e0011780 con 0x7fa0e8143ba0 2022-07-20T02:32:54.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e00118e0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e80ab2f0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e0011c40 con 0x7fa0e8143ba0 2022-07-20T02:32:54.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e80ab6e0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e814b000 con 0x7fa0e8143ba0 2022-07-20T02:32:54.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.325+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa0e001c460 con 0x7fa0e8143ba0 2022-07-20T02:32:54.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.329+0000 7fa0e5ffb700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 0x7fa0c8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.329+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa0e006c380 con 0x7fa0e8143ba0 2022-07-20T02:32:54.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.329+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 0x7fa0c8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.329+0000 7fa0ec8dd700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 0x7fa0c8055ea0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa0dc00c000 tx=0x7fa0dc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.333+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0e00432b0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0e80574e0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0e5ffb700 1 -- 172.21.15.26:0/156167944 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+1562 (secure 0 0 0) 0x7fa0e00468d0 con 0x7fa0e8143ba0 2022-07-20T02:32:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:54.509 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6559,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/1081130161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":1081130161},{"type":"v1","addr":"172.21.15.120:6833","nonce":1081130161}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2606429462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2606429462},{"type":"v1","addr":"172.21.15.26:6835","nonce":2606429462}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6957,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/961932243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":961932243},{"type":"v1","addr":"172.21.15.26:6837","nonce":961932243}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[]} 2022-07-20T02:32:54.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 msgr2=0x7fa0c8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 0x7fa0c8055ea0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa0dc00c000 tx=0x7fa0dc008040).stop 2022-07-20T02:32:54.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 msgr2=0x7fa0e8156920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7fa0e000a000 tx=0x7fa0e0008040).stop 2022-07-20T02:32:54.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 shutdown_connections 2022-07-20T02:32:54.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa0e8143ba0 0x7fa0e8156920 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa0c80539e0 0x7fa0c8055ea0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0e80b5ef0 0x7fa0e81436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 --2- 172.21.15.26:0/156167944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0e80a8c40 0x7fa0e80adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.493+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 >> 172.21.15.26:0/156167944 conn(0x7fa0e8060310 msgr2=0x7fa0e8148ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:54.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.497+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 shutdown_connections 2022-07-20T02:32:54.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.497+0000 7fa0ed8df700 1 -- 172.21.15.26:0/156167944 wait complete. 2022-07-20T02:32:54.574 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 45 2022-07-20T02:32:54.575 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:32:54.576 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:32:54.587 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:32:54.587 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:54.588 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:32:54.595 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:32:54.596 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:32:54.596 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:54.597 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:32:54.597 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:32:54.609 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:32:54.610 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:32:54.639 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:32:54.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:32:54.642 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:32:54.643 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:32:54.643 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:32:54.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:32:54.648 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:32:54.648 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:54.693 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:32:54.694 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:32:54.695 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:54.695 DEBUG:teuthology.parallel:result is None 2022-07-20T02:32:54.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:32:54.733 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:32:54.777 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:32:54.778 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:32:54.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 Processor -- start 2022-07-20T02:32:54.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 -- start start 2022-07-20T02:32:54.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f482c0b3260 0x7f482c067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c05ff70 con 0x7f482c0980d0 2022-07-20T02:32:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c0606e0 con 0x7f482c061860 2022-07-20T02:32:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f483096b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c0b3640 con 0x7f482c0b3260 2022-07-20T02:32:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f482b59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f482c0980d0 0x7f482c142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39826/0 (socket says 172.21.15.26:39826) 2022-07-20T02:32:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f482b59e700 1 -- 172.21.15.26:0/2708141933 learned_addr learned my addr 172.21.15.26:0/2708141933 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f482ad9d700 1 --2- 172.21.15.26:0/2708141933 >> v2:172.21.15.120:3300/0 conn(0x7f482c0b3260 0x7f482c067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395408377 0 0) 0x7f482c05ff70 con 0x7f482c0980d0 2022-07-20T02:32:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.817+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4810003260 con 0x7f482c0980d0 2022-07-20T02:32:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482ad9d700 1 -- 172.21.15.26:0/2708141933 >> v1:172.21.15.120:6789/0 conn(0x7f482c061860 legacy=0x7f482c0979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113904652 0 0) 0x7f482c0606e0 con 0x7f482c061860 2022-07-20T02:32:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482ad9d700 1 -- 172.21.15.26:0/2708141933 >> v1:172.21.15.26:6789/0 conn(0x7f482c0980d0 legacy=0x7f482c142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482ad9d700 1 -- 172.21.15.26:0/2708141933 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c0601a0 con 0x7f482c0b3260 2022-07-20T02:32:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482ad9d700 1 --2- 172.21.15.26:0/2708141933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c0b3260 0x7f482c067900 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c006e80).ready entity=mon.1 client_cookie=196f5fa1d821e692 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c014070 con 0x7f482c0b3260 2022-07-20T02:32:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c000e90 con 0x7f482c0b3260 2022-07-20T02:32:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f4829d9b700 1 -- 172.21.15.26:0/2708141933 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c0107f0 con 0x7f482c0b3260 2022-07-20T02:32:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- 172.21.15.26:0/2708141933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c0b3260 msgr2=0x7f482c067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 --2- 172.21.15.26:0/2708141933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c0b3260 0x7f482c067900 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c006e80).stop 2022-07-20T02:32:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- 172.21.15.26:0/2708141933 shutdown_connections 2022-07-20T02:32:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 --2- 172.21.15.26:0/2708141933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c0b3260 0x7f482c067900 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- 172.21.15.26:0/2708141933 >> 172.21.15.26:0/2708141933 conn(0x7f482c0b67f0 msgr2=0x7f482c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- 172.21.15.26:0/2708141933 shutdown_connections 2022-07-20T02:32:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- 172.21.15.26:0/2708141933 wait complete. 2022-07-20T02:32:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 Processor -- start 2022-07-20T02:32:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- start start 2022-07-20T02:32:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c061860 0x7f482c13d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f482c0b3260 0x7f482c0995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0679b0 con 0x7f482c0980d0 2022-07-20T02:32:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0949c0 con 0x7f482c061860 2022-07-20T02:32:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f483096b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0b05c0 con 0x7f482c0b3260 2022-07-20T02:32:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40550/0 (socket says 172.21.15.26:40550) 2022-07-20T02:32:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.821+0000 7f482a59c700 1 -- 172.21.15.26:0/3221795101 learned_addr learned my addr 172.21.15.26:0/3221795101 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482ad9d700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c061860 0x7f482c13d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482b59e700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f482c0b3260 0x7f482c0995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 -- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f482c0b3260 msgr2=0x7f482c0995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f482c0b3260 0x7f482c0995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 -- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c061860 msgr2=0x7f482c13d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c061860 0x7f482c13d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 -- 172.21.15.26:0/3221795101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c00c310 con 0x7f482c0980d0 2022-07-20T02:32:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482a59c700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f4820000fe0 tx=0x7f4820007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4820012400 con 0x7f482c0980d0 2022-07-20T02:32:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820012560 con 0x7f482c0980d0 2022-07-20T02:32:54.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c0952d0 con 0x7f482c0980d0 2022-07-20T02:32:54.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48200128c0 con 0x7f482c0980d0 2022-07-20T02:32:54.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c13dad0 con 0x7f482c0980d0 2022-07-20T02:32:54.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c094b30 con 0x7f482c0980d0 2022-07-20T02:32:54.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4820019030 con 0x7f482c0980d0 2022-07-20T02:32:54.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 0x7f4808055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:54.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f482006b020 con 0x7f482c0980d0 2022-07-20T02:32:54.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.825+0000 7f482ad9d700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 0x7f4808055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:54.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.829+0000 7f482ad9d700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 0x7f4808055fc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f481c01c000 tx=0x7f482c060330).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:54.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.833+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f482c094b30 con 0x7f482c0980d0 2022-07-20T02:32:54.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:54.969+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f482c094b30 con 0x7f482c0980d0 2022-07-20T02:32:55.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.813+0000 7f481bfff700 1 -- 172.21.15.26:0/3221795101 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7f482c094b30 con 0x7f482c0980d0 2022-07-20T02:32:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 msgr2=0x7f4808055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:55.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 0x7f4808055fc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f481c01c000 tx=0x7f482c060330).stop 2022-07-20T02:32:55.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 msgr2=0x7f482c0990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f4820000fe0 tx=0x7f4820007b50).stop 2022-07-20T02:32:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 shutdown_connections 2022-07-20T02:32:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f482c0980d0 0x7f482c0990b0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4808053b00 0x7f4808055fc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f482c0b3260 0x7f482c0995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 --2- 172.21.15.26:0/3221795101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f482c061860 0x7f482c13d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 >> 172.21.15.26:0/3221795101 conn(0x7f482c0b67f0 msgr2=0x7f482c0b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 shutdown_connections 2022-07-20T02:32:55.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:55.817+0000 7f483096b700 1 -- 172.21.15.26:0/3221795101 wait complete. 2022-07-20T02:32:55.871 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:32:55.879 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:32:55.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:32:56.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 Processor -- start 2022-07-20T02:32:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 -- start start 2022-07-20T02:32:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f65e80b2ab0 0x7f65e8061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80aa1e0 con 0x7f65e80af6d0 2022-07-20T02:32:56.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80ae110 con 0x7f65e80add30 2022-07-20T02:32:56.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65edbde700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e80a9f40 con 0x7f65e80b2ab0 2022-07-20T02:32:56.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65e7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f65e80b2ab0 0x7f65e8061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65ed3dd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f65e80af6d0 0x7f65e80afb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39844/0 (socket says 172.21.15.26:39844) 2022-07-20T02:32:56.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65ed3dd700 1 -- 172.21.15.26:0/554905112 learned_addr learned my addr 172.21.15.26:0/554905112 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:56.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65e7fff700 1 -- 172.21.15.26:0/554905112 >> v1:172.21.15.120:6789/0 conn(0x7f65e80add30 legacy=0x7f65e8060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65e77fe700 1 -- 172.21.15.26:0/554905112 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361508758 0 0) 0x7f65e80aa1e0 con 0x7f65e80af6d0 2022-07-20T02:32:56.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65e7fff700 1 -- 172.21.15.26:0/554905112 >> v1:172.21.15.26:6789/0 conn(0x7f65e80af6d0 legacy=0x7f65e80afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.016+0000 7f65e7fff700 1 -- 172.21.15.26:0/554905112 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e80b1110 con 0x7f65e80b2ab0 2022-07-20T02:32:56.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65e7fff700 1 --2- 172.21.15.26:0/554905112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e8061060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f65dc002f70 tx=0x7f65dc006df0).ready entity=mon.0 client_cookie=9fc88eee511fcbe2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65e77fe700 1 -- 172.21.15.26:0/554905112 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65dc00d610 con 0x7f65e80b2ab0 2022-07-20T02:32:56.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65e77fe700 1 -- 172.21.15.26:0/554905112 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc000e90 con 0x7f65e80b2ab0 2022-07-20T02:32:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65e77fe700 1 -- 172.21.15.26:0/554905112 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65dc00d960 con 0x7f65e80b2ab0 2022-07-20T02:32:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- 172.21.15.26:0/554905112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 msgr2=0x7f65e8061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 --2- 172.21.15.26:0/554905112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e8061060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f65dc002f70 tx=0x7f65dc006df0).stop 2022-07-20T02:32:56.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- 172.21.15.26:0/554905112 shutdown_connections 2022-07-20T02:32:56.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 --2- 172.21.15.26:0/554905112 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e8061060 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- 172.21.15.26:0/554905112 >> 172.21.15.26:0/554905112 conn(0x7f65e80b3050 msgr2=0x7f65e80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- 172.21.15.26:0/554905112 shutdown_connections 2022-07-20T02:32:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- 172.21.15.26:0/554905112 wait complete. 2022-07-20T02:32:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 Processor -- start 2022-07-20T02:32:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.020+0000 7f65edbde700 1 -- start start 2022-07-20T02:32:56.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65e80add30 0x7f65e8147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65e80af6d0 0x7f65e81454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8144e10 con 0x7f65e80b2ab0 2022-07-20T02:32:56.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b5670 con 0x7f65e80af6d0 2022-07-20T02:32:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b5910 con 0x7f65e80add30 2022-07-20T02:32:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40568/0 (socket says 172.21.15.26:40568) 2022-07-20T02:32:56.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 -- 172.21.15.26:0/2399945723 learned_addr learned my addr 172.21.15.26:0/2399945723 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ecbdc700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65e80add30 0x7f65e8147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65e7fff700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65e80af6d0 0x7f65e81454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 -- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65e80add30 msgr2=0x7f65e8147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65e80add30 0x7f65e8147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 -- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65e80af6d0 msgr2=0x7f65e81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65e80af6d0 0x7f65e81454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 -- 172.21.15.26:0/2399945723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc00b180 con 0x7f65e80b2ab0 2022-07-20T02:32:56.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65ed3dd700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f65e0001360 tx=0x7f65e0007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e0005ba0 con 0x7f65e80b2ab0 2022-07-20T02:32:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0005d00 con 0x7f65e80b2ab0 2022-07-20T02:32:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e8145000 con 0x7f65e80b2ab0 2022-07-20T02:32:56.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e00072c0 con 0x7f65e80b2ab0 2022-07-20T02:32:56.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.024+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e80b52f0 con 0x7f65e80b2ab0 2022-07-20T02:32:56.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f65e0010070 con 0x7f65e80b2ab0 2022-07-20T02:32:56.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e8145190 con 0x7f65e80b2ab0 2022-07-20T02:32:56.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65e5ffb700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 0x7f65c8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f65e006b630 con 0x7f65e80b2ab0 2022-07-20T02:32:56.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65ecbdc700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 0x7f65c8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.028+0000 7f65ecbdc700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 0x7f65c8055ff0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f65d800d000 tx=0x7f65d8007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.032+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65e0042730 con 0x7f65e80b2ab0 2022-07-20T02:32:56.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f65e80b13b0 con 0x7f65e80b2ab0 2022-07-20T02:32:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65e5ffb700 1 -- 172.21.15.26:0/2399945723 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f65e0045d50 con 0x7f65e80b2ab0 2022-07-20T02:32:56.210 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:56.211 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:32:56.211 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:32:56.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.212 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:32:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:32:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.235 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:32:56.237 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:32:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:56.238 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:32:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:32:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:32:56.248 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:32:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.256 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:32:56.284 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:32:56.289 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.294 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.296 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:32:56.297 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:32:56.301 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.302 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.304 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:32:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:32:56.305 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.306 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:32:56.309 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.310 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.311 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.314 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:32:56.322 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:32:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:32:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:32:56.326 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.327 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.328 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:32:56.336 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:32:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.337 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.345 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.347 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:32:56.348 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:32:56.348 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.349 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:32:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:32:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:32:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:32:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:32:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:32:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:32:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:32:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:32:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:32:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:32:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:32:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:32:56.375 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.376 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:56.376 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:32:56.377 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:56.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:56.379 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:32:56.379 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:32:56.384 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:32:56.385 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:32:56.386 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:32:56.387 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:32:56.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:56.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:32:56.392 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:32:56.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 msgr2=0x7f65c8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 0x7f65c8055ff0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f65d800d000 tx=0x7f65d8007b90).stop 2022-07-20T02:32:56.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 msgr2=0x7f65e814fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.200+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f65e0001360 tx=0x7f65e0007e50).stop 2022-07-20T02:32:56.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 shutdown_connections 2022-07-20T02:32:56.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65e80b2ab0 0x7f65e814fef0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f65c8053b30 0x7f65c8055ff0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65e80af6d0 0x7f65e81454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 --2- 172.21.15.26:0/2399945723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65e80add30 0x7f65e8147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 >> 172.21.15.26:0/2399945723 conn(0x7f65e80b3050 msgr2=0x7f65e8063800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 shutdown_connections 2022-07-20T02:32:56.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.204+0000 7f65edbde700 1 -- 172.21.15.26:0/2399945723 wait complete. 2022-07-20T02:32:56.404 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:32:56.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.516+0000 7faed359e700 1 Processor -- start 2022-07-20T02:32:56.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 -- start start 2022-07-20T02:32:56.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faed40b3260 0x7faed4060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faed40a95a0 0x7faed40676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed40b3640 con 0x7faed40b1720 2022-07-20T02:32:56.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40615c0 con 0x7faed40a95a0 2022-07-20T02:32:56.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed4061860 con 0x7faed40b3260 2022-07-20T02:32:56.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faed40a95a0 0x7faed40676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faed40a95a0 0x7faed40676b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40576/0 (socket says 172.21.15.26:40576) 2022-07-20T02:32:56.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 -- 172.21.15.26:0/2869828161 learned_addr learned my addr 172.21.15.26:0/2869828161 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:56.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed259c700 1 --2- 172.21.15.26:0/2869828161 >> v2:172.21.15.120:3300/0 conn(0x7faed40b3260 0x7faed4060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 -- 172.21.15.26:0/2869828161 >> v2:172.21.15.120:3300/0 conn(0x7faed40b3260 msgr2=0x7faed4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:56.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 --2- 172.21.15.26:0/2869828161 >> v2:172.21.15.120:3300/0 conn(0x7faed40b3260 0x7faed4060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 -- 172.21.15.26:0/2869828161 >> v1:172.21.15.120:6789/0 conn(0x7faed40b1720 legacy=0x7faed40b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 -- 172.21.15.26:0/2869828161 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4061b00 con 0x7faed40a95a0 2022-07-20T02:32:56.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faecbfff700 1 --2- 172.21.15.26:0/2869828161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed40676b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7faebc002f70 tx=0x7faebc006f40).ready entity=mon.0 client_cookie=ce74f17c0563a61b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed1d9b700 1 -- 172.21.15.26:0/2869828161 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faebc00da80 con 0x7faed40a95a0 2022-07-20T02:32:56.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed1d9b700 1 -- 172.21.15.26:0/2869828161 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc000e90 con 0x7faed40a95a0 2022-07-20T02:32:56.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.520+0000 7faed1d9b700 1 -- 172.21.15.26:0/2869828161 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faebc00de10 con 0x7faed40a95a0 2022-07-20T02:32:56.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- 172.21.15.26:0/2869828161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 msgr2=0x7faed40676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- 172.21.15.26:0/2869828161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed40676b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7faebc002f70 tx=0x7faebc006f40).stop 2022-07-20T02:32:56.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- 172.21.15.26:0/2869828161 shutdown_connections 2022-07-20T02:32:56.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- 172.21.15.26:0/2869828161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed40676b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- 172.21.15.26:0/2869828161 >> v2:172.21.15.120:3300/0 conn(0x7faed40b3260 0x7faed4060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- 172.21.15.26:0/2869828161 >> 172.21.15.26:0/2869828161 conn(0x7faed40b67f0 msgr2=0x7faed40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- 172.21.15.26:0/2869828161 shutdown_connections 2022-07-20T02:32:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- 172.21.15.26:0/2869828161 wait complete. 2022-07-20T02:32:56.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 Processor -- start 2022-07-20T02:32:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- start start 2022-07-20T02:32:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40b1720 0x7faed4155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40b3260 0x7faed4155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4063dd0 con 0x7faed40a95a0 2022-07-20T02:32:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4066010 con 0x7faed40b1720 2022-07-20T02:32:56.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed4064390 con 0x7faed40b3260 2022-07-20T02:32:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40580/0 (socket says 172.21.15.26:40580) 2022-07-20T02:32:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.524+0000 7faed259c700 1 -- 172.21.15.26:0/460341905 learned_addr learned my addr 172.21.15.26:0/460341905 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecbfff700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40b1720 0x7faed4155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed2d9d700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40b3260 0x7faed4155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 -- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40b3260 msgr2=0x7faed4155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40b3260 0x7faed4155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 -- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40b1720 msgr2=0x7faed4155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40b1720 0x7faed4155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 -- 172.21.15.26:0/460341905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc00b460 con 0x7faed40a95a0 2022-07-20T02:32:56.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed259c700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7faec400c000 tx=0x7faec4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec4011750 con 0x7faed40a95a0 2022-07-20T02:32:56.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec40118b0 con 0x7faed40a95a0 2022-07-20T02:32:56.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed4065cf0 con 0x7faed40a95a0 2022-07-20T02:32:56.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec4011c10 con 0x7faed40a95a0 2022-07-20T02:32:56.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed4060ba0 con 0x7faed40a95a0 2022-07-20T02:32:56.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed4067210 con 0x7faed40a95a0 2022-07-20T02:32:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faec4011d70 con 0x7faed40a95a0 2022-07-20T02:32:56.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.528+0000 7faecb7fe700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 0x7faeb8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.532+0000 7faecbfff700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 0x7faeb8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.532+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7faec406c0d0 con 0x7faed40a95a0 2022-07-20T02:32:56.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.532+0000 7faecbfff700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 0x7faeb8055ed0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7faebc01b000 tx=0x7faed40970e0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.536+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faec4043170 con 0x7faed40a95a0 2022-07-20T02:32:56.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.664+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faed40574e0 con 0x7faed40a95a0 2022-07-20T02:32:56.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.664+0000 7faecb7fe700 1 -- 172.21.15.26:0/460341905 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13463 (secure 0 0 0) 0x7faec4046790 con 0x7faed40a95a0 2022-07-20T02:32:56.672 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:56.673 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":94,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:55.802917+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:32:56.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 msgr2=0x7faeb8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 0x7faeb8055ed0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7faebc01b000 tx=0x7faed40970e0).stop 2022-07-20T02:32:56.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 msgr2=0x7faed4090870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7faec400c000 tx=0x7faec4006df0).stop 2022-07-20T02:32:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 shutdown_connections 2022-07-20T02:32:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faeb8053a10 0x7faeb8055ed0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faed40b3260 0x7faed4155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed40b1720 0x7faed4155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 --2- 172.21.15.26:0/460341905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faed40a95a0 0x7faed4090870 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 >> 172.21.15.26:0/460341905 conn(0x7faed40b67f0 msgr2=0x7faed409a150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 shutdown_connections 2022-07-20T02:32:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.668+0000 7faed359e700 1 -- 172.21.15.26:0/460341905 wait complete. 2022-07-20T02:32:56.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:32:56.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:32:56.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 Processor -- start 2022-07-20T02:32:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 -- start start 2022-07-20T02:32:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa63c0b1160 0x7fa63c140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c146e80 0x7fa63c1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63c05d3b0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63c1435b0 con 0x7fa63c0b1160 2022-07-20T02:32:56.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa642bc3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0b4bc0 con 0x7fa63c146e80 2022-07-20T02:32:56.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa64095f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa63c05cfd0 0x7fa63c1407c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39868/0 (socket says 172.21.15.26:39868) 2022-07-20T02:32:56.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa63c0b1160 0x7fa63c140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa64095f700 1 -- 172.21.15.26:0/844910496 learned_addr learned my addr 172.21.15.26:0/844910496 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:56.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa641160700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c146e80 0x7fa63c1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa6337fe700 1 -- 172.21.15.26:0/844910496 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512487599 0 0) 0x7fa63c05d3b0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa6337fe700 1 -- 172.21.15.26:0/844910496 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa628003260 con 0x7fa63c05cfd0 2022-07-20T02:32:56.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 -- 172.21.15.26:0/844910496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c146e80 msgr2=0x7fa63c1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:56.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c146e80 0x7fa63c1472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 -- 172.21.15.26:0/844910496 >> v1:172.21.15.26:6789/0 conn(0x7fa63c05cfd0 legacy=0x7fa63c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 -- 172.21.15.26:0/844910496 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c0b4e60 con 0x7fa63c0b1160 2022-07-20T02:32:56.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa633fff700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c0b1160 0x7fa63c140ed0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fa62c002f70 tx=0x7fa62c006df0).ready entity=mon.0 client_cookie=f1c12d303bfaaba3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa6337fe700 1 -- 172.21.15.26:0/844910496 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa62c00d4a0 con 0x7fa63c0b1160 2022-07-20T02:32:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa6337fe700 1 -- 172.21.15.26:0/844910496 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c000e90 con 0x7fa63c0b1160 2022-07-20T02:32:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.816+0000 7fa6337fe700 1 -- 172.21.15.26:0/844910496 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa62c00d7f0 con 0x7fa63c0b1160 2022-07-20T02:32:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- 172.21.15.26:0/844910496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c0b1160 msgr2=0x7fa63c140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c0b1160 0x7fa63c140ed0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fa62c002f70 tx=0x7fa62c006df0).stop 2022-07-20T02:32:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- 172.21.15.26:0/844910496 shutdown_connections 2022-07-20T02:32:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c146e80 0x7fa63c1472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 --2- 172.21.15.26:0/844910496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c0b1160 0x7fa63c140ed0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- 172.21.15.26:0/844910496 >> 172.21.15.26:0/844910496 conn(0x7fa63c066f70 msgr2=0x7fa63c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- 172.21.15.26:0/844910496 shutdown_connections 2022-07-20T02:32:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- 172.21.15.26:0/844910496 wait complete. 2022-07-20T02:32:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 Processor -- start 2022-07-20T02:32:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.820+0000 7fa642bc3700 1 -- start start 2022-07-20T02:32:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c0b1160 0x7fa63c1560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c146e80 0x7fa63c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0adc20 con 0x7fa63c05cfd0 2022-07-20T02:32:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0ad6e0 con 0x7fa63c146e80 2022-07-20T02:32:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0ad980 con 0x7fa63c0b1160 2022-07-20T02:32:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40594/0 (socket says 172.21.15.26:40594) 2022-07-20T02:32:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 -- 172.21.15.26:0/763912185 learned_addr learned my addr 172.21.15.26:0/763912185 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa641160700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c146e80 0x7fa63c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa633fff700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c0b1160 0x7fa63c1560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 -- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c0b1160 msgr2=0x7fa63c1560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c0b1160 0x7fa63c1560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 -- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c146e80 msgr2=0x7fa63c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c146e80 0x7fa63c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 -- 172.21.15.26:0/763912185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c00b180 con 0x7fa63c05cfd0 2022-07-20T02:32:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa64095f700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fa638001110 tx=0x7fa638007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6380070d0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c155c10 con 0x7fa63c05cfd0 2022-07-20T02:32:56.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa638007230 con 0x7fa63c05cfd0 2022-07-20T02:32:56.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa638005610 con 0x7fa63c05cfd0 2022-07-20T02:32:56.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c0ae6e0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c0574e0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.824+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa638010070 con 0x7fa63c05cfd0 2022-07-20T02:32:56.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.828+0000 7fa631ffb700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 0x7fa61c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:56.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.828+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa63806b530 con 0x7fa63c05cfd0 2022-07-20T02:32:56.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.828+0000 7fa633fff700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 0x7fa61c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:56.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.828+0000 7fa633fff700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 0x7fa61c055ff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa62c01a000 tx=0x7fa62c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:56.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.832+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa6380424a0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.952+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa63c05db80 con 0x7fa63c05cfd0 2022-07-20T02:32:56.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa631ffb700 1 -- 172.21.15.26:0/763912185 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v94) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa638045ac0 con 0x7fa63c05cfd0 2022-07-20T02:32:56.962 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:32:56.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 msgr2=0x7fa61c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 0x7fa61c055ff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa62c01a000 tx=0x7fa62c014000).stop 2022-07-20T02:32:56.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 msgr2=0x7fa63c13f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:56.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fa638001110 tx=0x7fa638007b80).stop 2022-07-20T02:32:56.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 shutdown_connections 2022-07-20T02:32:56.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa61c053b30 0x7fa61c055ff0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c146e80 0x7fa63c1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa63c0b1160 0x7fa63c1560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 --2- 172.21.15.26:0/763912185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa63c05cfd0 0x7fa63c13f610 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:56.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 >> 172.21.15.26:0/763912185 conn(0x7fa63c066f70 msgr2=0x7fa63c0b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:56.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 shutdown_connections 2022-07-20T02:32:56.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:56.956+0000 7fa642bc3700 1 -- 172.21.15.26:0/763912185 wait complete. 2022-07-20T02:32:56.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:32:57.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 Processor -- start 2022-07-20T02:32:57.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 -- start start 2022-07-20T02:32:57.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ee40b7920 0x7f1ee40b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee40614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee40608d0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee40b7d00 con 0x7f1ee40b7920 2022-07-20T02:32:57.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee9bd6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40b4120 con 0x7f1ee40b31b0 2022-07-20T02:32:57.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee40614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee8bd4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ee40b7920 0x7f1ee40b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee93d5700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1ee40b2c70 0x7f1ee40b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:54098/0 (socket says 172.21.15.26:54098) 2022-07-20T02:32:57.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee93d5700 1 -- 172.21.15.26:0/404453653 learned_addr learned my addr 172.21.15.26:0/404453653 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:57.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee37fe700 1 -- 172.21.15.26:0/404453653 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624154116 0 0) 0x7f1ee40608d0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee37fe700 1 -- 172.21.15.26:0/404453653 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed0003260 con 0x7f1ee40b2c70 2022-07-20T02:32:57.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 -- 172.21.15.26:0/404453653 >> v2:172.21.15.120:3300/0 conn(0x7f1ee40b7920 msgr2=0x7f1ee40b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/404453653 >> v2:172.21.15.120:3300/0 conn(0x7f1ee40b7920 0x7f1ee40b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 -- 172.21.15.26:0/404453653 >> v1:172.21.15.120:6789/0 conn(0x7f1ee40b2c70 legacy=0x7f1ee40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 -- 172.21.15.26:0/404453653 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee4060b00 con 0x7f1ee40b31b0 2022-07-20T02:32:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.084+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/404453653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee40614e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1ed4002f70 tx=0x7f1ed4006df0).ready entity=mon.2 client_cookie=dad3d4fea73d7faa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee37fe700 1 -- 172.21.15.26:0/404453653 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed400d4a0 con 0x7f1ee40b31b0 2022-07-20T02:32:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee37fe700 1 -- 172.21.15.26:0/404453653 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed4000e90 con 0x7f1ee40b31b0 2022-07-20T02:32:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee37fe700 1 -- 172.21.15.26:0/404453653 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed400d7f0 con 0x7f1ee40b31b0 2022-07-20T02:32:57.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/404453653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 msgr2=0x7f1ee40614e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/404453653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee40614e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1ed4002f70 tx=0x7f1ed4006df0).stop 2022-07-20T02:32:57.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/404453653 shutdown_connections 2022-07-20T02:32:57.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/404453653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee40614e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/404453653 >> v2:172.21.15.120:3300/0 conn(0x7f1ee40b7920 0x7f1ee40b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/404453653 >> 172.21.15.26:0/404453653 conn(0x7f1ee405f840 msgr2=0x7f1ee405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:57.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/404453653 shutdown_connections 2022-07-20T02:32:57.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/404453653 wait complete. 2022-07-20T02:32:57.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 Processor -- start 2022-07-20T02:32:57.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.088+0000 7f1ee9bd6700 1 -- start start 2022-07-20T02:32:57.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee4152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 0x7f1ee40ad740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40adde0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41524f0 con 0x7f1ee40b7920 2022-07-20T02:32:57.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4152790 con 0x7f1ee40b31b0 2022-07-20T02:32:57.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40608/0 (socket says 172.21.15.26:40608) 2022-07-20T02:32:57.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 -- 172.21.15.26:0/4220136049 learned_addr learned my addr 172.21.15.26:0/4220136049 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:57.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee4152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee93d5700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 0x7f1ee40ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 -- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 msgr2=0x7f1ee4152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee4152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 -- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 msgr2=0x7f1ee40ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 0x7f1ee40ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 -- 172.21.15.26:0/4220136049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed400b180 con 0x7f1ee40b2c70 2022-07-20T02:32:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee93d5700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 0x7f1ee40ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee4152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee8bd4700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f1ed800c000 tx=0x7f1ed8006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed8011750 con 0x7f1ee40b2c70 2022-07-20T02:32:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed80118b0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee4152a30 con 0x7f1ee40b2c70 2022-07-20T02:32:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed8011c10 con 0x7f1ee40b2c70 2022-07-20T02:32:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.092+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee4146f40 con 0x7f1ee40b2c70 2022-07-20T02:32:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee4060da0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ed8011d70 con 0x7f1ee40b2c70 2022-07-20T02:32:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee1ffb700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 0x7f1ec4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f1ed806bc60 con 0x7f1ee40b2c70 2022-07-20T02:32:57.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 0x7f1ec4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.096+0000 7f1ee3fff700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 0x7f1ec4055fc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f1ed401a000 tx=0x7f1ed4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.104+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ed8042d00 con 0x7f1ee40b2c70 2022-07-20T02:32:57.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1ee40574e0 con 0x7f1ee40b2c70 2022-07-20T02:32:57.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee1ffb700 1 -- 172.21.15.26:0/4220136049 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v94) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1ed8046320 con 0x7f1ee40b2c70 2022-07-20T02:32:57.256 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:32:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 msgr2=0x7f1ec4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 0x7f1ec4055fc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f1ed401a000 tx=0x7f1ed4014000).stop 2022-07-20T02:32:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 msgr2=0x7f1ee41465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f1ed800c000 tx=0x7f1ed8006d90).stop 2022-07-20T02:32:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 shutdown_connections 2022-07-20T02:32:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ec4053b00 0x7f1ec4055fc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee40b7920 0x7f1ee40ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee40b31b0 0x7f1ee4152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 --2- 172.21.15.26:0/4220136049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ee40b2c70 0x7f1ee41465c0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 >> 172.21.15.26:0/4220136049 conn(0x7f1ee405f840 msgr2=0x7f1ee405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 shutdown_connections 2022-07-20T02:32:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.224+0000 7f1ee9bd6700 1 -- 172.21.15.26:0/4220136049 wait complete. 2022-07-20T02:32:57.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:32:57.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 Processor -- start 2022-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- start start 2022-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040b1ae0 con 0x7f59040672a0 2022-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040b2e20 con 0x7f59040b24c0 2022-07-20T02:32:57.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b1d80 con 0x7f59041465b0 2022-07-20T02:32:57.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5902ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:33900/0 (socket says 172.21.15.26:33900) 2022-07-20T02:32:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5902ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40620/0 (socket says 172.21.15.26:40620) 2022-07-20T02:32:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 -- 172.21.15.26:0/2091613273 learned_addr learned my addr 172.21.15.26:0/2091613273 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 -- 172.21.15.26:0/2091613273 >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 msgr2=0x7f59040604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 --2- 172.21.15.26:0/2091613273 >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 -- 172.21.15.26:0/2091613273 >> v1:172.21.15.120:6789/0 conn(0x7f59040672a0 legacy=0x7f5904146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 -- 172.21.15.26:0/2091613273 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040b2020 con 0x7f59041465b0 2022-07-20T02:32:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5902ffd700 1 --2- 172.21.15.26:0/2091613273 >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:32:57.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59037fe700 1 --2- 172.21.15.26:0/2091613273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f58f40032f0 tx=0x7f58f4009290).ready entity=mon.2 client_cookie=b2ee591b30815788 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59027fc700 1 -- 172.21.15.26:0/2091613273 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f400b8a0 con 0x7f59041465b0 2022-07-20T02:32:57.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59027fc700 1 -- 172.21.15.26:0/2091613273 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f400ba00 con 0x7f59041465b0 2022-07-20T02:32:57.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f59027fc700 1 -- 172.21.15.26:0/2091613273 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f400bb70 con 0x7f59041465b0 2022-07-20T02:32:57.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- 172.21.15.26:0/2091613273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 msgr2=0x7f5904066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 --2- 172.21.15.26:0/2091613273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f58f40032f0 tx=0x7f58f4009290).stop 2022-07-20T02:32:57.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- 172.21.15.26:0/2091613273 shutdown_connections 2022-07-20T02:32:57.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 --2- 172.21.15.26:0/2091613273 >> v2:172.21.15.26:3300/0 conn(0x7f59040b24c0 0x7f59040604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 --2- 172.21.15.26:0/2091613273 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59041465b0 0x7f5904066db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- 172.21.15.26:0/2091613273 >> 172.21.15.26:0/2091613273 conn(0x7f59040b3050 msgr2=0x7f59040b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:57.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.368+0000 7f5908a8d700 1 -- 172.21.15.26:0/2091613273 shutdown_connections 2022-07-20T02:32:57.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- 172.21.15.26:0/2091613273 wait complete. 2022-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 Processor -- start 2022-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- start start 2022-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59040672a0 0x7f5904152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 0x7f590413f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5904145bb0 con 0x7f59041465b0 2022-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5904141870 con 0x7f59040672a0 2022-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5904145d30 con 0x7f5904064980 2022-07-20T02:32:57.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f59037fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59040672a0 0x7f5904152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40624/0 (socket says 172.21.15.26:40624) 2022-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 -- 172.21.15.26:0/3261901098 learned_addr learned my addr 172.21.15.26:0/3261901098 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5903fff700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 0x7f590413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 -- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 msgr2=0x7f590413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 0x7f590413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 -- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59040672a0 msgr2=0x7f5904152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59040672a0 0x7f5904152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 -- 172.21.15.26:0/3261901098 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec00b180 con 0x7f59041465b0 2022-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5903fff700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 0x7f590413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:32:57.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5902ffd700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f58ec00c000 tx=0x7f58ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58ec00c890 con 0x7f59041465b0 2022-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec00c9f0 con 0x7f59041465b0 2022-07-20T02:32:57.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f4009530 con 0x7f59041465b0 2022-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58ec00cd50 con 0x7f59041465b0 2022-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.372+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59041461f0 con 0x7f59041465b0 2022-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f58ec01c460 con 0x7f59041465b0 2022-07-20T02:32:57.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5904141a30 con 0x7f59041465b0 2022-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f5900ff9700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 0x7f58dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f58ec06c3f0 con 0x7f59041465b0 2022-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f59037fe700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 0x7f58dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.376+0000 7f59037fe700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 0x7f58dc055fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f58f401b000 tx=0x7f58f4016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.380+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58ec043380 con 0x7f59041465b0 2022-07-20T02:32:57.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5904141c40 con 0x7f59041465b0 2022-07-20T02:32:57.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5900ff9700 1 -- 172.21.15.26:0/3261901098 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+143 (secure 0 0 0) 0x7f58ec0469a0 con 0x7f59041465b0 2022-07-20T02:32:57.519 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:32:57.519 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:32:57.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:57.519 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:32:57.520 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:32:57.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:32:57.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:32:57.525 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:32:57.525 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:32:57.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:32:57.526 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:32:57.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 msgr2=0x7f58dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 0x7f58dc055fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f58f401b000 tx=0x7f58f4016000).stop 2022-07-20T02:32:57.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 msgr2=0x7f5904145370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f58ec00c000 tx=0x7f58ec006df0).stop 2022-07-20T02:32:57.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 shutdown_connections 2022-07-20T02:32:57.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59041465b0 0x7f5904145370 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f58dc053b00 0x7f58dc055fc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5904064980 0x7f590413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 --2- 172.21.15.26:0/3261901098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59040672a0 0x7f5904152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 >> 172.21.15.26:0/3261901098 conn(0x7f59040b3050 msgr2=0x7f59040b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 shutdown_connections 2022-07-20T02:32:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.512+0000 7f5908a8d700 1 -- 172.21.15.26:0/3261901098 wait complete. 2022-07-20T02:32:57.547 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:32:57.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:32:57.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.656+0000 7f850bfff700 1 Processor -- start 2022-07-20T02:32:57.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.656+0000 7f850bfff700 1 -- start start 2022-07-20T02:32:57.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.656+0000 7f850bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f850c05cfd0 0x7f850c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.656+0000 7f850bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f850c060f00 0x7f850c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c1474e0 con 0x7f850c147000 2022-07-20T02:32:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0b4e40 con 0x7f850c060f00 2022-07-20T02:32:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c05d3b0 con 0x7f850c05cfd0 2022-07-20T02:32:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f850c060f00 0x7f850c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f850c060f00 0x7f850c13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40634/0 (socket says 172.21.15.26:40634) 2022-07-20T02:32:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 -- 172.21.15.26:0/3976817871 learned_addr learned my addr 172.21.15.26:0/3976817871 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850affd700 1 --2- 172.21.15.26:0/3976817871 >> v2:172.21.15.120:3300/0 conn(0x7f850c05cfd0 0x7f850c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 -- 172.21.15.26:0/3976817871 >> v2:172.21.15.120:3300/0 conn(0x7f850c05cfd0 msgr2=0x7f850c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 --2- 172.21.15.26:0/3976817871 >> v2:172.21.15.120:3300/0 conn(0x7f850c05cfd0 0x7f850c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 -- 172.21.15.26:0/3976817871 >> v1:172.21.15.120:6789/0 conn(0x7f850c147000 legacy=0x7f850c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 -- 172.21.15.26:0/3976817871 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c1431e0 con 0x7f850c060f00 2022-07-20T02:32:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850b7fe700 1 --2- 172.21.15.26:0/3976817871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c13c9b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f85040032f0 tx=0x7f8504009290).ready entity=mon.0 client_cookie=ac83edaf60c64161 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850a7fc700 1 -- 172.21.15.26:0/3976817871 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850400b870 con 0x7f850c060f00 2022-07-20T02:32:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850a7fc700 1 -- 172.21.15.26:0/3976817871 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850400b9d0 con 0x7f850c060f00 2022-07-20T02:32:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850a7fc700 1 -- 172.21.15.26:0/3976817871 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850400bb40 con 0x7f850c060f00 2022-07-20T02:32:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- 172.21.15.26:0/3976817871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 msgr2=0x7f850c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 --2- 172.21.15.26:0/3976817871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c13c9b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f85040032f0 tx=0x7f8504009290).stop 2022-07-20T02:32:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- 172.21.15.26:0/3976817871 shutdown_connections 2022-07-20T02:32:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 --2- 172.21.15.26:0/3976817871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c13c9b0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 --2- 172.21.15.26:0/3976817871 >> v2:172.21.15.120:3300/0 conn(0x7f850c05cfd0 0x7f850c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- 172.21.15.26:0/3976817871 >> 172.21.15.26:0/3976817871 conn(0x7f850c0620c0 msgr2=0x7f850c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- 172.21.15.26:0/3976817871 shutdown_connections 2022-07-20T02:32:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.660+0000 7f850bfff700 1 -- 172.21.15.26:0/3976817871 wait complete. 2022-07-20T02:32:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 Processor -- start 2022-07-20T02:32:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- start start 2022-07-20T02:32:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f850c147000 0x7f850c155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f850c0aee40 0x7f850c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c152320 con 0x7f850c060f00 2022-07-20T02:32:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c152510 con 0x7f850c0aee40 2022-07-20T02:32:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1527b0 con 0x7f850c147000 2022-07-20T02:32:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40638/0 (socket says 172.21.15.26:40638) 2022-07-20T02:32:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 -- 172.21.15.26:0/4216744647 learned_addr learned my addr 172.21.15.26:0/4216744647 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850b7fe700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f850c0aee40 0x7f850c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f85027fc700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f850c147000 0x7f850c155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 -- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f850c147000 msgr2=0x7f850c155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f850c147000 0x7f850c155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 -- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f850c0aee40 msgr2=0x7f850c155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f850c0aee40 0x7f850c155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 -- 172.21.15.26:0/4216744647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504009530 con 0x7f850c060f00 2022-07-20T02:32:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850affd700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f84fc0030b0 tx=0x7f84fc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc011780 con 0x7f850c060f00 2022-07-20T02:32:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc0118e0 con 0x7f850c060f00 2022-07-20T02:32:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c152a50 con 0x7f850c060f00 2022-07-20T02:32:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc011c40 con 0x7f850c060f00 2022-07-20T02:32:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.664+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c153190 con 0x7f850c060f00 2022-07-20T02:32:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c152ef0 con 0x7f850c060f00 2022-07-20T02:32:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84fc01c460 con 0x7f850c060f00 2022-07-20T02:32:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f8508ff9700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f84fc06bf50 con 0x7f850c060f00 2022-07-20T02:32:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f85027fc700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.668+0000 7f85027fc700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 0x7f84f0055f30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f84f400d000 tx=0x7f84f4000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.676+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f850c152ef0 con 0x7f850c060f00 2022-07-20T02:32:57.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:57.800+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f850c152ef0 con 0x7f850c060f00 2022-07-20T02:32:58.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.560+0000 7f8508ff9700 1 -- 172.21.15.26:0/4216744647 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v95) v1 ==== 133+0+0 (secure 0 0 0) 0x7f850c152ef0 con 0x7f850c060f00 2022-07-20T02:32:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 msgr2=0x7f84f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 0x7f84f0055f30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f84f400d000 tx=0x7f84f4000bc0).stop 2022-07-20T02:32:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 msgr2=0x7f850c1421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f84fc0030b0 tx=0x7f84fc008040).stop 2022-07-20T02:32:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 shutdown_connections 2022-07-20T02:32:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f850c0aee40 0x7f850c155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f850c147000 0x7f850c155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 --2- 172.21.15.26:0/4216744647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f850c060f00 0x7f850c1421d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 >> 172.21.15.26:0/4216744647 conn(0x7f850c05cfd0 msgr2=0x7f850c0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 shutdown_connections 2022-07-20T02:32:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.564+0000 7f850bfff700 1 -- 172.21.15.26:0/4216744647 wait complete. 2022-07-20T02:32:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:32:58.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:32:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.692+0000 7f0aa9b9b700 1 Processor -- start 2022-07-20T02:32:58.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.692+0000 7f0aa9b9b700 1 -- start start 2022-07-20T02:32:58.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa9b9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0aa41431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa9b9b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0aa4062710 0x7f0aa40b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa9b9b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40ba060 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa9b9b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa412bee0 con 0x7f0aa4062710 2022-07-20T02:32:58.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa9b9b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4061f80 con 0x7f0aa40b9c80 2022-07-20T02:32:58.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0aa40b6fe0 0x7f0aa41403e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39924/0 (socket says 172.21.15.26:39924) 2022-07-20T02:32:58.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 -- 172.21.15.26:0/3948104955 learned_addr learned my addr 172.21.15.26:0/3948104955 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:32:58.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa2ffd700 1 --2- 172.21.15.26:0/3948104955 >> v2:172.21.15.120:3300/0 conn(0x7f0aa4062710 0x7f0aa40b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa37fe700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0aa41431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618478841 0 0) 0x7f0aa40ba060 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a84003260 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2196585955 0 0) 0x7f0a84003260 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 msgr2=0x7f0aa41431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:58.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0aa41431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 >> v2:172.21.15.120:3300/0 conn(0x7f0aa4062710 msgr2=0x7f0aa40b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:58.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 --2- 172.21.15.26:0/3948104955 >> v2:172.21.15.120:3300/0 conn(0x7f0aa4062710 0x7f0aa40b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4062220 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f0a9c0019e0 con 0x7f0aa40b6fe0 2022-07-20T02:32:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 >> v1:172.21.15.26:6789/0 conn(0x7f0aa40b6fe0 legacy=0x7f0aa41403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0a84003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a84003c80 0x7f0a84004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0a84004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa412bee0 con 0x7f0a840049f0 2022-07-20T02:32:58.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4061f80 con 0x7f0a84003c80 2022-07-20T02:32:58.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8400b750 con 0x7f0aa40b9c80 2022-07-20T02:32:58.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0a84004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa37fe700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0a84003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa2ffd700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a84003c80 0x7f0a84004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 -- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 msgr2=0x7f0a84003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:58.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa40b9c80 0x7f0a84003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 -- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a84003c80 msgr2=0x7f0a84004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a84003c80 0x7f0a84004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 -- 172.21.15.26:0/3948104955 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84003260 con 0x7f0a840049f0 2022-07-20T02:32:58.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.696+0000 7f0aa3fff700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0a84004e70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f0a840054d0 tx=0x7f0aa40ae720).ready entity=mon.0 client_cookie=d1474895f8c32484 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:58.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9c00cc40 con 0x7f0a840049f0 2022-07-20T02:32:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c00cda0 con 0x7f0a840049f0 2022-07-20T02:32:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa27fc700 1 -- 172.21.15.26:0/3948104955 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9c005b80 con 0x7f0a840049f0 2022-07-20T02:32:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/3948104955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 msgr2=0x7f0a84004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/3948104955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0a84004e70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f0a840054d0 tx=0x7f0aa40ae720).stop 2022-07-20T02:32:58.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa3fff700 1 -- 172.21.15.26:0/3948104955 reap_dead start 2022-07-20T02:32:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/3948104955 shutdown_connections 2022-07-20T02:32:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/3948104955 >> 172.21.15.26:0/3948104955 conn(0x7f0aa4066f70 msgr2=0x7f0aa405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/3948104955 shutdown_connections 2022-07-20T02:32:58.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/3948104955 wait complete. 2022-07-20T02:32:58.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 Processor -- start 2022-07-20T02:32:58.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.700+0000 7f0aa9b9b700 1 -- start start 2022-07-20T02:32:58.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa4152170 0x7f0aa412d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa412d500 0x7f0aa413fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40b14a0 con 0x7f0a840049f0 2022-07-20T02:32:58.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40b16d0 con 0x7f0aa412d500 2022-07-20T02:32:58.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40b3630 con 0x7f0aa4152170 2022-07-20T02:32:58.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40656/0 (socket says 172.21.15.26:40656) 2022-07-20T02:32:58.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 -- 172.21.15.26:0/629953396 learned_addr learned my addr 172.21.15.26:0/629953396 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:58.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa3fff700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa412d500 0x7f0aa413fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa2ffd700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa4152170 0x7f0aa412d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 -- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa4152170 msgr2=0x7f0aa412d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa4152170 0x7f0aa412d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 -- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa412d500 msgr2=0x7f0aa413fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:58.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa412d500 0x7f0aa413fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:58.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 -- 172.21.15.26:0/629953396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40b38d0 con 0x7f0a840049f0 2022-07-20T02:32:58.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa37fe700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f0a9400b000 tx=0x7f0a94006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:58.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9400b540 con 0x7f0a840049f0 2022-07-20T02:32:58.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9400f760 con 0x7f0a840049f0 2022-07-20T02:32:58.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa40b2960 con 0x7f0a840049f0 2022-07-20T02:32:58.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9400fac0 con 0x7f0a840049f0 2022-07-20T02:32:58.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.704+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa40b1020 con 0x7f0a840049f0 2022-07-20T02:32:58.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa4152930 con 0x7f0a840049f0 2022-07-20T02:32:58.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0a9400fc20 con 0x7f0a840049f0 2022-07-20T02:32:58.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa0ff9700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 0x7f0a88055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:58.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f0a9406c830 con 0x7f0a840049f0 2022-07-20T02:32:58.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa2ffd700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 0x7f0a88055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:58.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.708+0000 7f0aa2ffd700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 0x7f0a88055ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0a8c00b000 tx=0x7f0a8c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:58.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.716+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0aa4152930 con 0x7f0a840049f0 2022-07-20T02:32:58.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:58.848+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f0aa4152930 con 0x7f0a840049f0 2022-07-20T02:32:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.564+0000 7f0aa0ff9700 1 -- 172.21.15.26:0/629953396 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v96) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0aa4152930 con 0x7f0a840049f0 2022-07-20T02:32:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 msgr2=0x7f0a88055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 0x7f0a88055ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0a8c00b000 tx=0x7f0a8c006df0).stop 2022-07-20T02:32:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 msgr2=0x7f0aa40b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f0a9400b000 tx=0x7f0a94006df0).stop 2022-07-20T02:32:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 shutdown_connections 2022-07-20T02:32:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0a88053a10 0x7f0a88055ed0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0aa412d500 0x7f0aa413fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0aa4152170 0x7f0aa412d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 --2- 172.21.15.26:0/629953396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0a840049f0 0x7f0aa40b1de0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 >> 172.21.15.26:0/629953396 conn(0x7f0aa4066f70 msgr2=0x7f0aa405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 shutdown_connections 2022-07-20T02:32:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.576+0000 7f0aa9b9b700 1 -- 172.21.15.26:0/629953396 wait complete. 2022-07-20T02:32:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:32:59.596 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:32:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.708+0000 7f6e7d2d5700 1 Processor -- start 2022-07-20T02:32:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.708+0000 7f6e7d2d5700 1 -- start start 2022-07-20T02:32:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.708+0000 7f6e7d2d5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e78143090 0x7f6e78098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.708+0000 7f6e7d2d5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.708+0000 7f6e7d2d5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e78143470 con 0x7f6e780b0c50 2022-07-20T02:32:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7805fd90 con 0x7f6e78143090 2022-07-20T02:32:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78097d50 con 0x7f6e780b0810 2022-07-20T02:32:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e78143090 0x7f6e78098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e78143090 0x7f6e78098490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40668/0 (socket says 172.21.15.26:40668) 2022-07-20T02:32:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e777fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e780b0c50 0x7f6e780955d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39944/0 (socket says 172.21.15.26:39944) 2022-07-20T02:32:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 -- 172.21.15.26:0/1077086290 learned_addr learned my addr 172.21.15.26:0/1077086290 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e767fc700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 -- 172.21.15.26:0/1077086290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 msgr2=0x7f6e780950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:32:59.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e75ffb700 1 -- 172.21.15.26:0/1077086290 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373007123 0 0) 0x7f6e78143470 con 0x7f6e780b0c50 2022-07-20T02:32:59.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 -- 172.21.15.26:0/1077086290 >> v1:172.21.15.26:6789/0 conn(0x7f6e780b0c50 legacy=0x7f6e780955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 -- 172.21.15.26:0/1077086290 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78097ff0 con 0x7f6e78143090 2022-07-20T02:32:59.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e76ffd700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78098490 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f6e78098c00 tx=0x7f6e6c006eb0).ready entity=mon.0 client_cookie=8a881d12323af6d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:59.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e75ffb700 1 -- 172.21.15.26:0/1077086290 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e6c00d5c0 con 0x7f6e78143090 2022-07-20T02:32:59.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e75ffb700 1 -- 172.21.15.26:0/1077086290 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c000e90 con 0x7f6e78143090 2022-07-20T02:32:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e75ffb700 1 -- 172.21.15.26:0/1077086290 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e6c00d910 con 0x7f6e78143090 2022-07-20T02:32:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/1077086290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 msgr2=0x7f6e78098490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78098490 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f6e78098c00 tx=0x7f6e6c006eb0).stop 2022-07-20T02:32:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/1077086290 shutdown_connections 2022-07-20T02:32:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/1077086290 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78098490 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/1077086290 >> 172.21.15.26:0/1077086290 conn(0x7f6e780b5e70 msgr2=0x7f6e780b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:59.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/1077086290 shutdown_connections 2022-07-20T02:32:59.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.712+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/1077086290 wait complete. 2022-07-20T02:32:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 Processor -- start 2022-07-20T02:32:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 -- start start 2022-07-20T02:32:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e780b0c50 0x7f6e780ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780aebe0 con 0x7f6e78143090 2022-07-20T02:32:59.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780abe30 con 0x7f6e780b0c50 2022-07-20T02:32:59.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e7d2d5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780ac0d0 con 0x7f6e780b0810 2022-07-20T02:32:59.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40674/0 (socket says 172.21.15.26:40674) 2022-07-20T02:32:59.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e76ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 -- 172.21.15.26:0/2930099303 learned_addr learned my addr 172.21.15.26:0/2930099303 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:32:59.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e767fc700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e780b0c50 0x7f6e780ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 -- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 msgr2=0x7f6e780aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 -- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e780b0c50 msgr2=0x7f6e780ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e780b0c50 0x7f6e780ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 -- 172.21.15.26:0/2930099303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c00b1e0 con 0x7f6e78143090 2022-07-20T02:32:59.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.716+0000 7f6e777fe700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f6e70001360 tx=0x7f6e70007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:59.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e70005ae0 con 0x7f6e78143090 2022-07-20T02:32:59.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70005c40 con 0x7f6e78143090 2022-07-20T02:32:59.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780ac370 con 0x7f6e78143090 2022-07-20T02:32:59.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e700070d0 con 0x7f6e78143090 2022-07-20T02:32:59.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7813fac0 con 0x7f6e78143090 2022-07-20T02:32:59.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e780ac500 con 0x7f6e78143090 2022-07-20T02:32:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.720+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e70010070 con 0x7f6e78143090 2022-07-20T02:32:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.724+0000 7f6e67fff700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 0x7f6e54055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:32:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.724+0000 7f6e76ffd700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 0x7f6e54055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:32:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.724+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f6e7006b8d0 con 0x7f6e78143090 2022-07-20T02:32:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.724+0000 7f6e76ffd700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 0x7f6e54055ea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f6e6c01c000 tx=0x7f6e6c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:32:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.728+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e70042710 con 0x7f6e78143090 2022-07-20T02:32:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.896+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f6e78095ce0 con 0x7f6e78143090 2022-07-20T02:32:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.912+0000 7f6e67fff700 1 -- 172.21.15.26:0/2930099303 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v47) v1 ==== 189+0+0 (secure 0 0 0) 0x7f6e70018070 con 0x7f6e78143090 2022-07-20T02:32:59.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 msgr2=0x7f6e54055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 0x7f6e54055ea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f6e6c01c000 tx=0x7f6e6c015000).stop 2022-07-20T02:32:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 msgr2=0x7f6e78061180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:32:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f6e70001360 tx=0x7f6e70007e20).stop 2022-07-20T02:32:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 shutdown_connections 2022-07-20T02:32:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e78143090 0x7f6e78061180 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e540539e0 0x7f6e54055ea0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e780b0c50 0x7f6e780ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 --2- 172.21.15.26:0/2930099303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e780b0810 0x7f6e780aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:32:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 >> 172.21.15.26:0/2930099303 conn(0x7f6e780b5e70 msgr2=0x7f6e78140b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:32:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 shutdown_connections 2022-07-20T02:32:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:32:59.916+0000 7f6e7d2d5700 1 -- 172.21.15.26:0/2930099303 wait complete. 2022-07-20T02:32:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 17 and data pool 18 2022-07-20T02:32:59.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:00.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.032+0000 7f484d76c700 1 Processor -- start 2022-07-20T02:33:00.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.032+0000 7f484d76c700 1 -- start start 2022-07-20T02:33:00.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4848147000 0x7f4848143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48481474e0 con 0x7f484805cfd0 2022-07-20T02:33:00.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b4e40 con 0x7f4848060f00 2022-07-20T02:33:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484805d3b0 con 0x7f4848147000 2022-07-20T02:33:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4847fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f484805cfd0 0x7f48480b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39966/0 (socket says 172.21.15.26:39966) 2022-07-20T02:33:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f48477fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4848147000 0x7f4848143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4847fff700 1 -- 172.21.15.26:0/3272668349 learned_addr learned my addr 172.21.15.26:0/3272668349 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:00.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f48477fe700 1 -- 172.21.15.26:0/3272668349 >> v1:172.21.15.120:6789/0 conn(0x7f4848060f00 legacy=0x7f484813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4846ffd700 1 -- 172.21.15.26:0/3272668349 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838010714 0 0) 0x7f48481474e0 con 0x7f484805cfd0 2022-07-20T02:33:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f48477fe700 1 -- 172.21.15.26:0/3272668349 >> v1:172.21.15.26:6789/0 conn(0x7f484805cfd0 legacy=0x7f48480b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f48477fe700 1 -- 172.21.15.26:0/3272668349 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48481431e0 con 0x7f4848147000 2022-07-20T02:33:00.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f48477fe700 1 --2- 172.21.15.26:0/3272668349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848147000 0x7f4848143bc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f483c002f70 tx=0x7f483c006df0).ready entity=mon.0 client_cookie=c700ebb07ee2cf23 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4846ffd700 1 -- 172.21.15.26:0/3272668349 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f483c00d610 con 0x7f4848147000 2022-07-20T02:33:00.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4846ffd700 1 -- 172.21.15.26:0/3272668349 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c000e90 con 0x7f4848147000 2022-07-20T02:33:00.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f4846ffd700 1 -- 172.21.15.26:0/3272668349 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f483c00d960 con 0x7f4848147000 2022-07-20T02:33:00.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- 172.21.15.26:0/3272668349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848147000 msgr2=0x7f4848143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 --2- 172.21.15.26:0/3272668349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848147000 0x7f4848143bc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f483c002f70 tx=0x7f483c006df0).stop 2022-07-20T02:33:00.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- 172.21.15.26:0/3272668349 shutdown_connections 2022-07-20T02:33:00.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 --2- 172.21.15.26:0/3272668349 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848147000 0x7f4848143bc0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- 172.21.15.26:0/3272668349 >> 172.21.15.26:0/3272668349 conn(0x7f48480620c0 msgr2=0x7f484805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- 172.21.15.26:0/3272668349 shutdown_connections 2022-07-20T02:33:00.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.036+0000 7f484d76c700 1 -- 172.21.15.26:0/3272668349 wait complete. 2022-07-20T02:33:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 Processor -- start 2022-07-20T02:33:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- start start 2022-07-20T02:33:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 0x7f4848152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4848146010 0x7f48480b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848146540 con 0x7f4848060f00 2022-07-20T02:33:00.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848156230 con 0x7f4848146010 2022-07-20T02:33:00.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48481566a0 con 0x7f4848147000 2022-07-20T02:33:00.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40692/0 (socket says 172.21.15.26:40692) 2022-07-20T02:33:00.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 -- 172.21.15.26:0/3011921551 learned_addr learned my addr 172.21.15.26:0/3011921551 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:00.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484cf6b700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4848146010 0x7f48480b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f48477fe700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 0x7f4848152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 -- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 msgr2=0x7f4848152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 0x7f4848152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 -- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4848146010 msgr2=0x7f48480b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4848146010 0x7f48480b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 -- 172.21.15.26:0/3011921551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c00b180 con 0x7f4848060f00 2022-07-20T02:33:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f48477fe700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 0x7f4848152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f4847fff700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f4840000f40 tx=0x7f4840007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4840005ba0 con 0x7f4848060f00 2022-07-20T02:33:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48481467a0 con 0x7f4848060f00 2022-07-20T02:33:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4840005d00 con 0x7f4848060f00 2022-07-20T02:33:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48400072c0 con 0x7f4848060f00 2022-07-20T02:33:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.040+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48480b7720 con 0x7f4848060f00 2022-07-20T02:33:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848156400 con 0x7f4848060f00 2022-07-20T02:33:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4840010070 con 0x7f4848060f00 2022-07-20T02:33:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f48457fa700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 0x7f4828055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f484006bb80 con 0x7f4848060f00 2022-07-20T02:33:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f48477fe700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 0x7f4828055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.044+0000 7f48477fe700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 0x7f4828055f30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f483c01a000 tx=0x7f483c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.052+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4848156400 con 0x7f4848060f00 2022-07-20T02:33:00.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.180+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4848156400 con 0x7f4848060f00 2022-07-20T02:33:00.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.180+0000 7f48457fa700 1 -- 172.21.15.26:0/3011921551 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16335 (secure 0 0 0) 0x7f4848156400 con 0x7f4848060f00 2022-07-20T02:33:00.187 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:00.188 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":97,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:59.904119+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:57.859395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:58.906424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 msgr2=0x7f4828055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 0x7f4828055f30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f483c01a000 tx=0x7f483c015000).stop 2022-07-20T02:33:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 msgr2=0x7f48480b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f4840000f40 tx=0x7f4840007f70).stop 2022-07-20T02:33:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 shutdown_connections 2022-07-20T02:33:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4828053a70 0x7f4828055f30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4848146010 0x7f48480b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4848147000 0x7f4848152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 --2- 172.21.15.26:0/3011921551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4848060f00 0x7f48480b2ee0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 >> 172.21.15.26:0/3011921551 conn(0x7f484805cfd0 msgr2=0x7f484813a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 shutdown_connections 2022-07-20T02:33:00.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.184+0000 7f484d76c700 1 -- 172.21.15.26:0/3011921551 wait complete. 2022-07-20T02:33:00.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 Processor -- start 2022-07-20T02:33:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- start start 2022-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0a7fc0 0x7fbe9c1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0b62d0 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c147eb0 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c14aac0 con 0x7fbe9c0a7fc0 2022-07-20T02:33:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40700/0 (socket says 172.21.15.26:40700) 2022-07-20T02:33:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 -- 172.21.15.26:0/2317371600 learned_addr learned my addr 172.21.15.26:0/2317371600 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe99d9b700 1 --2- 172.21.15.26:0/2317371600 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0a7fc0 0x7fbe9c1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 -- 172.21.15.26:0/2317371600 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0a7fc0 msgr2=0x7fbe9c1479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 --2- 172.21.15.26:0/2317371600 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0a7fc0 0x7fbe9c1479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 -- 172.21.15.26:0/2317371600 >> v1:172.21.15.120:6789/0 conn(0x7fbe9c0a8c40 legacy=0x7fbe9c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 -- 172.21.15.26:0/2317371600 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c14ad60 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9a59c700 1 --2- 172.21.15.26:0/2317371600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fbe8c0032f0 tx=0x7fbe8c009290).ready entity=mon.0 client_cookie=5f8c898e890558f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9959a700 1 -- 172.21.15.26:0/2317371600 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe8c00b870 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9959a700 1 -- 172.21.15.26:0/2317371600 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00b9d0 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9959a700 1 -- 172.21.15.26:0/2317371600 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe8c00bb40 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- 172.21.15.26:0/2317371600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0b5ef0 msgr2=0x7fbe9c1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/2317371600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fbe8c0032f0 tx=0x7fbe8c009290).stop 2022-07-20T02:33:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- 172.21.15.26:0/2317371600 shutdown_connections 2022-07-20T02:33:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/2317371600 >> v2:172.21.15.120:3300/0 conn(0x7fbe9c0a7fc0 0x7fbe9c1479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/2317371600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0b5ef0 0x7fbe9c1474d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- 172.21.15.26:0/2317371600 >> 172.21.15.26:0/2317371600 conn(0x7fbe9c060310 msgr2=0x7fbe9c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- 172.21.15.26:0/2317371600 shutdown_connections 2022-07-20T02:33:00.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.336+0000 7fbe9b59e700 1 -- 172.21.15.26:0/2317371600 wait complete. 2022-07-20T02:33:00.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 Processor -- start 2022-07-20T02:33:00.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- start start 2022-07-20T02:33:00.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 0x7fbe9c066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 0x7fbe9c066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0b5ef0 0x7fbe9c11dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b5230 con 0x7fbe9c0a7fc0 2022-07-20T02:33:00.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c157600 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c1578a0 con 0x7fbe9c0b5ef0 2022-07-20T02:33:00.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 0x7fbe9c066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 0x7fbe9c066160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40708/0 (socket says 172.21.15.26:40708) 2022-07-20T02:33:00.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 0x7fbe9c066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9a59c700 1 -- 172.21.15.26:0/3846393900 learned_addr learned my addr 172.21.15.26:0/3846393900 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:00.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9ad9d700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0b5ef0 0x7fbe9c11dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 -- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0b5ef0 msgr2=0x7fbe9c11dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0b5ef0 0x7fbe9c11dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 -- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 msgr2=0x7fbe9c066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 0x7fbe9c066160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 -- 172.21.15.26:0/3846393900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c009530 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe99d9b700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 0x7fbe9c066650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbe8400a000 tx=0x7fbe84008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe84012610 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c064ca0 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84012770 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe84012ad0 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.340+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c064900 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c14b000 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbe84012c30 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe937fe700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 0x7fbe80055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbe8406bfe0 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe9a59c700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 0x7fbe80055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.344+0000 7fbe9a59c700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 0x7fbe80055ff0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fbe8c01b000 tx=0x7fbe8c016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.352+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbe84042c70 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.480+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe9c0574e0 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.480+0000 7fbe937fe700 1 -- 172.21.15.26:0/3846393900 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16335 (secure 0 0 0) 0x7fbe84046290 con 0x7fbe9c0a8c40 2022-07-20T02:33:00.489 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:00.489 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":97,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:32:59.904119+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:57.859395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:58.906424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 msgr2=0x7fbe80055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 0x7fbe80055ff0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fbe8c01b000 tx=0x7fbe8c016000).stop 2022-07-20T02:33:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 msgr2=0x7fbe9c066650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 0x7fbe9c066650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbe8400a000 tx=0x7fbe84008040).stop 2022-07-20T02:33:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 shutdown_connections 2022-07-20T02:33:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbe80053b30 0x7fbe80055ff0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe9c0b5ef0 0x7fbe9c11dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe9c0a8c40 0x7fbe9c066650 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 --2- 172.21.15.26:0/3846393900 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe9c0a7fc0 0x7fbe9c066160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 >> 172.21.15.26:0/3846393900 conn(0x7fbe9c060310 msgr2=0x7fbe9c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 shutdown_connections 2022-07-20T02:33:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.484+0000 7fbe9b59e700 1 -- 172.21.15.26:0/3846393900 wait complete. 2022-07-20T02:33:00.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:33:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.644+0000 7f10e87a3700 1 Processor -- start 2022-07-20T02:33:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.644+0000 7f10e87a3700 1 -- start start 2022-07-20T02:33:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10e00b0050 0x7f10e00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e005d050 0x7f10e0143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e00a9e10 con 0x7f10e0143e00 2022-07-20T02:33:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00617a0 con 0x7f10e00b0050 2022-07-20T02:33:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00aa040 con 0x7f10e005d050 2022-07-20T02:33:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10e00b0050 0x7f10e00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f10e00b0050 0x7f10e00b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40714/0 (socket says 172.21.15.26:40714) 2022-07-20T02:33:00.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 -- 172.21.15.26:0/736596207 learned_addr learned my addr 172.21.15.26:0/736596207 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e5d3e700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e005d050 0x7f10e0143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 -- 172.21.15.26:0/736596207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e005d050 msgr2=0x7f10e0143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e005d050 0x7f10e0143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 -- 172.21.15.26:0/736596207 >> v1:172.21.15.120:6789/0 conn(0x7f10e0143e00 legacy=0x7f10e00b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 -- 172.21.15.26:0/736596207 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00aa2e0 con 0x7f10e00b0050 2022-07-20T02:33:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e653f700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b0430 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f10dc002f70 tx=0x7f10dc006eb0).ready entity=mon.0 client_cookie=b8949f9de8a6c832 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e553d700 1 -- 172.21.15.26:0/736596207 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10dc00d570 con 0x7f10e00b0050 2022-07-20T02:33:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e553d700 1 -- 172.21.15.26:0/736596207 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc000e90 con 0x7f10e00b0050 2022-07-20T02:33:00.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e553d700 1 -- 172.21.15.26:0/736596207 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10dc00d900 con 0x7f10e00b0050 2022-07-20T02:33:00.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- 172.21.15.26:0/736596207 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 msgr2=0x7f10e00b0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b0430 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f10dc002f70 tx=0x7f10dc006eb0).stop 2022-07-20T02:33:00.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- 172.21.15.26:0/736596207 shutdown_connections 2022-07-20T02:33:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e005d050 0x7f10e0143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 --2- 172.21.15.26:0/736596207 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b0430 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- 172.21.15.26:0/736596207 >> 172.21.15.26:0/736596207 conn(0x7f10e0060950 msgr2=0x7f10e0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- 172.21.15.26:0/736596207 shutdown_connections 2022-07-20T02:33:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.648+0000 7f10e87a3700 1 -- 172.21.15.26:0/736596207 wait complete. 2022-07-20T02:33:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 Processor -- start 2022-07-20T02:33:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 -- start start 2022-07-20T02:33:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 0x7f10e00abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 0x7f10e00b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0146590 con 0x7f10e00b0050 2022-07-20T02:33:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e01467c0 con 0x7f10e005d050 2022-07-20T02:33:00.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0146b30 con 0x7f10e0143e00 2022-07-20T02:33:00.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40722/0 (socket says 172.21.15.26:40722) 2022-07-20T02:33:00.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 -- 172.21.15.26:0/1684147856 learned_addr learned my addr 172.21.15.26:0/1684147856 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:00.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e653f700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 0x7f10e00abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e6d40700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 0x7f10e00b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 -- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 msgr2=0x7f10e00b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 0x7f10e00b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 -- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 msgr2=0x7f10e00abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 0x7f10e00abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 -- 172.21.15.26:0/1684147856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc00b1e0 con 0x7f10e00b0050 2022-07-20T02:33:00.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e653f700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 0x7f10e00abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:00.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e6d40700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 0x7f10e00b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:00.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e5d3e700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f10d000c000 tx=0x7f10d0006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10d00104c0 con 0x7f10e00b0050 2022-07-20T02:33:00.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d0010620 con 0x7f10e00b0050 2022-07-20T02:33:00.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10d0010980 con 0x7f10e00b0050 2022-07-20T02:33:00.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.652+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e0146dd0 con 0x7f10e00b0050 2022-07-20T02:33:00.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e0146220 con 0x7f10e00b0050 2022-07-20T02:33:00.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e0146f60 con 0x7f10e00b0050 2022-07-20T02:33:00.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f10d001a030 con 0x7f10e00b0050 2022-07-20T02:33:00.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10cf7fe700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 0x7f10c0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:00.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f10d006beb0 con 0x7f10e00b0050 2022-07-20T02:33:00.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.656+0000 7f10e653f700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 0x7f10c0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:00.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.660+0000 7f10e653f700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 0x7f10c0055f60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f10dc017000 tx=0x7f10dc015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:00.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.660+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10d0042b60 con 0x7f10e00b0050 2022-07-20T02:33:00.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.824+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f10e00ae520 con 0x7f10e00b0050 2022-07-20T02:33:00.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.908+0000 7f10cf7fe700 1 -- 172.21.15.26:0/1684147856 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f10d0046180 con 0x7f10e00b0050 2022-07-20T02:33:00.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 msgr2=0x7f10c0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 0x7f10c0055f60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f10dc017000 tx=0x7f10dc015000).stop 2022-07-20T02:33:00.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 msgr2=0x7f10e00b45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:00.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f10d000c000 tx=0x7f10d0006d90).stop 2022-07-20T02:33:00.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 shutdown_connections 2022-07-20T02:33:00.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10e00b0050 0x7f10e00b45f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f10c0053aa0 0x7f10c0055f60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10e0143e00 0x7f10e00b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 --2- 172.21.15.26:0/1684147856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10e005d050 0x7f10e00abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:00.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 >> 172.21.15.26:0/1684147856 conn(0x7f10e0060950 msgr2=0x7f10e0142700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:00.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 shutdown_connections 2022-07-20T02:33:00.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:00.912+0000 7f10e87a3700 1 -- 172.21.15.26:0/1684147856 wait complete. 2022-07-20T02:33:00.945 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:01.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.056+0000 7f21e00bc700 1 Processor -- start 2022-07-20T02:33:01.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.056+0000 7f21e00bc700 1 -- start start 2022-07-20T02:33:01.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f21d80aff70 0x7f21d80af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80b2ea0 con 0x7f21d80b2ac0 2022-07-20T02:33:01.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80ae810 con 0x7f21d80aff70 2022-07-20T02:33:01.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80610a0 con 0x7f21d8141460 2022-07-20T02:33:01.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f21d80aff70 0x7f21d80af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21de659700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f21d80aff70 0x7f21d80af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:60420/0 (socket says 172.21.15.26:60420) 2022-07-20T02:33:01.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 -- 172.21.15.26:0/3321070741 learned_addr learned my addr 172.21.15.26:0/3321070741 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21de659700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34016/0 (socket says 172.21.15.26:34016) 2022-07-20T02:33:01.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 -- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 msgr2=0x7f21d81418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:01.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 -- 172.21.15.26:0/3321070741 >> v1:172.21.15.120:6789/0 conn(0x7f21d80b2ac0 legacy=0x7f21d80aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 -- 172.21.15.26:0/3321070741 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d8060b60 con 0x7f21d80aff70 2022-07-20T02:33:01.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21de659700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:01.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dd657700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80aff70 0x7f21d80af380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f21d40032f0 tx=0x7f21d4009290).ready entity=mon.1 client_cookie=e2ef1d3af5175ef8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dce56700 1 -- 172.21.15.26:0/3321070741 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d400b8a0 con 0x7f21d80aff70 2022-07-20T02:33:01.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dce56700 1 -- 172.21.15.26:0/3321070741 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d400ba00 con 0x7f21d80aff70 2022-07-20T02:33:01.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21dce56700 1 -- 172.21.15.26:0/3321070741 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d400bb70 con 0x7f21d80aff70 2022-07-20T02:33:01.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80aff70 msgr2=0x7f21d80af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80aff70 0x7f21d80af380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f21d40032f0 tx=0x7f21d4009290).stop 2022-07-20T02:33:01.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- 172.21.15.26:0/3321070741 shutdown_connections 2022-07-20T02:33:01.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d8141460 0x7f21d81418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 --2- 172.21.15.26:0/3321070741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d80aff70 0x7f21d80af380 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- 172.21.15.26:0/3321070741 >> 172.21.15.26:0/3321070741 conn(0x7f21d80b60a0 msgr2=0x7f21d80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- 172.21.15.26:0/3321070741 shutdown_connections 2022-07-20T02:33:01.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.060+0000 7f21e00bc700 1 -- 172.21.15.26:0/3321070741 wait complete. 2022-07-20T02:33:01.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 Processor -- start 2022-07-20T02:33:01.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- start start 2022-07-20T02:33:01.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d80b2ac0 0x7f21d8143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 0x7f21d80ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8142770 con 0x7f21d80aff70 2022-07-20T02:33:01.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80ac9a0 con 0x7f21d8141460 2022-07-20T02:33:01.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d805aec0 con 0x7f21d80b2ac0 2022-07-20T02:33:01.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40736/0 (socket says 172.21.15.26:40736) 2022-07-20T02:33:01.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 -- 172.21.15.26:0/4287591746 learned_addr learned my addr 172.21.15.26:0/4287591746 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dd657700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d80b2ac0 0x7f21d8143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21de659700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 0x7f21d80ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 -- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d80b2ac0 msgr2=0x7f21d8143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d80b2ac0 0x7f21d8143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 -- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 msgr2=0x7f21d80ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 0x7f21d80ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 -- 172.21.15.26:0/4287591746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d000b180 con 0x7f21d80aff70 2022-07-20T02:33:01.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21de659700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 0x7f21d80ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:01.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21dde58700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f21d8061cf0 tx=0x7f21c8007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8007030 con 0x7f21d80aff70 2022-07-20T02:33:01.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c8007190 con 0x7f21d80aff70 2022-07-20T02:33:01.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d4009530 con 0x7f21d80aff70 2022-07-20T02:33:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8006510 con 0x7f21d80aff70 2022-07-20T02:33:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.064+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80aabd0 con 0x7f21d80aff70 2022-07-20T02:33:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d8060e00 con 0x7f21d80aff70 2022-07-20T02:33:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f21c8006670 con 0x7f21d80aff70 2022-07-20T02:33:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21c6ffd700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 0x7f21b8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f21c806ba60 con 0x7f21d80aff70 2022-07-20T02:33:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21dd657700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 0x7f21b8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.068+0000 7f21dd657700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 0x7f21b8055e10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f21d401b000 tx=0x7f21d4016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.076+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21c80426b0 con 0x7f21d80aff70 2022-07-20T02:33:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.236+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21d80574e0 con 0x7f21d80aff70 2022-07-20T02:33:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.236+0000 7f21c6ffd700 1 -- 172.21.15.26:0/4287591746 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f21c8045cd0 con 0x7f21d80aff70 2022-07-20T02:33:01.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:01.245 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:01.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 msgr2=0x7f21b8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 0x7f21b8055e10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f21d401b000 tx=0x7f21d4016000).stop 2022-07-20T02:33:01.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 msgr2=0x7f21d8141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f21d8061cf0 tx=0x7f21c8007c00).stop 2022-07-20T02:33:01.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 shutdown_connections 2022-07-20T02:33:01.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f21b8053950 0x7f21b8055e10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d8141460 0x7f21d80ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f21d80b2ac0 0x7f21d8143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 --2- 172.21.15.26:0/4287591746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f21d80aff70 0x7f21d8141f20 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 >> 172.21.15.26:0/4287591746 conn(0x7f21d80b60a0 msgr2=0x7f21d80b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 shutdown_connections 2022-07-20T02:33:01.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.240+0000 7f21e00bc700 1 -- 172.21.15.26:0/4287591746 wait complete. 2022-07-20T02:33:01.258 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:01.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:01.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 Processor -- start 2022-07-20T02:33:01.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 -- start start 2022-07-20T02:33:01.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35ac0b5ef0 0x7f35ac1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a7fc0 0x7f35ac1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 0x7f35ac0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac147eb0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0b62d0 con 0x7f35ac0a8c40 2022-07-20T02:33:01.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b1416700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac14aac0 con 0x7f35ac0a7fc0 2022-07-20T02:33:01.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35ac0b5ef0 0x7f35ac1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f35ac0b5ef0 0x7f35ac1474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40742/0 (socket says 172.21.15.26:40742) 2022-07-20T02:33:01.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 -- 172.21.15.26:0/2371514878 learned_addr learned my addr 172.21.15.26:0/2371514878 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b0c15700 1 --2- 172.21.15.26:0/2371514878 >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 0x7f35ac0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35ab7fe700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a7fc0 0x7f35ac1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 -- 172.21.15.26:0/2371514878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a7fc0 msgr2=0x7f35ac1479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a7fc0 0x7f35ac1479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 -- 172.21.15.26:0/2371514878 >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 msgr2=0x7f35ac0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:01.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 --2- 172.21.15.26:0/2371514878 >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 0x7f35ac0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 -- 172.21.15.26:0/2371514878 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac14ad60 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35b0c15700 1 --2- 172.21.15.26:0/2371514878 >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 0x7f35ac0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:01.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35abfff700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac1474d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f3594002f70 tx=0x7f3594006eb0).ready entity=mon.0 client_cookie=3aa0f293cea89a38 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35aaffd700 1 -- 172.21.15.26:0/2371514878 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f359400d570 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35aaffd700 1 -- 172.21.15.26:0/2371514878 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594000e90 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.376+0000 7f35aaffd700 1 -- 172.21.15.26:0/2371514878 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f359400d900 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2371514878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 msgr2=0x7f35ac1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac1474d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f3594002f70 tx=0x7f3594006eb0).stop 2022-07-20T02:33:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2371514878 shutdown_connections 2022-07-20T02:33:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- 172.21.15.26:0/2371514878 >> v2:172.21.15.120:3300/0 conn(0x7f35ac0a8c40 0x7f35ac0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a7fc0 0x7f35ac1479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- 172.21.15.26:0/2371514878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac1474d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2371514878 >> 172.21.15.26:0/2371514878 conn(0x7f35ac060310 msgr2=0x7f35ac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2371514878 shutdown_connections 2022-07-20T02:33:01.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2371514878 wait complete. 2022-07-20T02:33:01.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 Processor -- start 2022-07-20T02:33:01.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- start start 2022-07-20T02:33:01.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 0x7f35ac156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a8c40 0x7f35ac1569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac147800 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0ac010 con 0x7f35ac0a7fc0 2022-07-20T02:33:01.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac156ee0 con 0x7f35ac0a8c40 2022-07-20T02:33:01.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40750/0 (socket says 172.21.15.26:40750) 2022-07-20T02:33:01.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35ab7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a8c40 0x7f35ac1569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 -- 172.21.15.26:0/2079586765 learned_addr learned my addr 172.21.15.26:0/2079586765 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35abfff700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 0x7f35ac156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 -- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a8c40 msgr2=0x7f35ac1569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a8c40 0x7f35ac1569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 -- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 msgr2=0x7f35ac156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 0x7f35ac156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 -- 172.21.15.26:0/2079586765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a400b180 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35abfff700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 0x7f35ac156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:01.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b0c15700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f35ac157020 tx=0x7f35a4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a400c7e0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a400c940 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359400b1e0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a400cca0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.380+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac0ab8d0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac0abd10 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35a4026430 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35a97fa700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 0x7f3590055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f35a406d0d0 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35abfff700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 0x7f3590055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.384+0000 7f35abfff700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 0x7f3590055f30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f359401a000 tx=0x7f3594015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.392+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35a4043e30 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.516+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35ac14b000 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35a97fa700 1 -- 172.21.15.26:0/2079586765 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16322 (secure 0 0 0) 0x7f35a4047450 con 0x7f35ac0b5ef0 2022-07-20T02:33:01.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:01.531 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":98,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:00.907866+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:57.859395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:58.906424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:01.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 msgr2=0x7f3590055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 0x7f3590055f30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f359401a000 tx=0x7f3594015000).stop 2022-07-20T02:33:01.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 msgr2=0x7f35ac159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f35ac157020 tx=0x7f35a4006df0).stop 2022-07-20T02:33:01.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 shutdown_connections 2022-07-20T02:33:01.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35ac0b5ef0 0x7f35ac159a90 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3590053a70 0x7f3590055f30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ac0a8c40 0x7f35ac1569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 --2- 172.21.15.26:0/2079586765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ac0a7fc0 0x7f35ac156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 >> 172.21.15.26:0/2079586765 conn(0x7f35ac060310 msgr2=0x7f35ac14c370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 shutdown_connections 2022-07-20T02:33:01.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.520+0000 7f35b1416700 1 -- 172.21.15.26:0/2079586765 wait complete. 2022-07-20T02:33:01.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:01.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.680+0000 7fcc6ecb0700 1 Processor -- start 2022-07-20T02:33:01.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- start start 2022-07-20T02:33:01.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 0x7fcc680b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 0x7fcc68064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcc680b3380 0x7fcc68065440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc680b85f0 con 0x7fcc680b3380 2022-07-20T02:33:01.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc68064880 con 0x7fcc68064450 2022-07-20T02:33:01.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680b80b0 con 0x7fcc680670c0 2022-07-20T02:33:01.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcc680b3380 0x7fcc68065440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcc680b3380 0x7fcc68065440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40756/0 (socket says 172.21.15.26:40756) 2022-07-20T02:33:01.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 -- 172.21.15.26:0/804610037 learned_addr learned my addr 172.21.15.26:0/804610037 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6dcae700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 0x7fcc680b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6d4ad700 1 --2- 172.21.15.26:0/804610037 >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 0x7fcc68064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 -- 172.21.15.26:0/804610037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 msgr2=0x7fcc680b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:01.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 0x7fcc680b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 -- 172.21.15.26:0/804610037 >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 msgr2=0x7fcc68064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:01.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 --2- 172.21.15.26:0/804610037 >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 0x7fcc68064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 -- 172.21.15.26:0/804610037 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680b8350 con 0x7fcc680b3380 2022-07-20T02:33:01.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6d4ad700 1 --2- 172.21.15.26:0/804610037 >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 0x7fcc68064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6dcae700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 0x7fcc680b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6e4af700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680b3380 0x7fcc68065440 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fcc600032f0 tx=0x7fcc60009290).ready entity=mon.0 client_cookie=d99076e0d09aefb2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ccac700 1 -- 172.21.15.26:0/804610037 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6000b870 con 0x7fcc680b3380 2022-07-20T02:33:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ccac700 1 -- 172.21.15.26:0/804610037 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6000b9d0 con 0x7fcc680b3380 2022-07-20T02:33:01.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ccac700 1 -- 172.21.15.26:0/804610037 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6000bb40 con 0x7fcc680b3380 2022-07-20T02:33:01.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/804610037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680b3380 msgr2=0x7fcc68065440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680b3380 0x7fcc68065440 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fcc600032f0 tx=0x7fcc60009290).stop 2022-07-20T02:33:01.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/804610037 shutdown_connections 2022-07-20T02:33:01.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680b3380 0x7fcc68065440 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/804610037 >> v2:172.21.15.120:3300/0 conn(0x7fcc68064450 0x7fcc68064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/804610037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680670c0 0x7fcc680b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/804610037 >> 172.21.15.26:0/804610037 conn(0x7fcc6805f840 msgr2=0x7fcc6805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/804610037 shutdown_connections 2022-07-20T02:33:01.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.684+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/804610037 wait complete. 2022-07-20T02:33:01.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 Processor -- start 2022-07-20T02:33:01.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- start start 2022-07-20T02:33:01.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc68141a00 0x7fcc68140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 0x7fcc68155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681408d0 con 0x7fcc680670c0 2022-07-20T02:33:01.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68147010 con 0x7fcc68141a00 2022-07-20T02:33:01.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681472b0 con 0x7fcc68062480 2022-07-20T02:33:01.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40762/0 (socket says 172.21.15.26:40762) 2022-07-20T02:33:01.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 -- 172.21.15.26:0/3731321490 learned_addr learned my addr 172.21.15.26:0/3731321490 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:01.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6d4ad700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc68141a00 0x7fcc68140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6e4af700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 0x7fcc68155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 -- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 msgr2=0x7fcc68155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 0x7fcc68155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 -- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc68141a00 msgr2=0x7fcc68140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc68141a00 0x7fcc68140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 -- 172.21.15.26:0/3731321490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc60009530 con 0x7fcc680670c0 2022-07-20T02:33:01.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6e4af700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 0x7fcc68155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:01.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6dcae700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fcc58007000 tx=0x7fcc58008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc58010580 con 0x7fcc680670c0 2022-07-20T02:33:01.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc580106e0 con 0x7fcc680670c0 2022-07-20T02:33:01.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68140b00 con 0x7fcc680670c0 2022-07-20T02:33:01.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc58010a40 con 0x7fcc680670c0 2022-07-20T02:33:01.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.688+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc68155b70 con 0x7fcc680670c0 2022-07-20T02:33:01.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc68140c90 con 0x7fcc680670c0 2022-07-20T02:33:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcc58010ba0 con 0x7fcc680670c0 2022-07-20T02:33:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc56ffd700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 0x7fcc48055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc6d4ad700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 0x7fcc48055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcc5806c050 con 0x7fcc680670c0 2022-07-20T02:33:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.692+0000 7fcc6d4ad700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 0x7fcc48055e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fcc6400d000 tx=0x7fcc64008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.700+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcc58042e10 con 0x7fcc680670c0 2022-07-20T02:33:01.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.864+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcc68064ab0 con 0x7fcc680670c0 2022-07-20T02:33:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.872+0000 7fcc56ffd700 1 -- 172.21.15.26:0/3731321490 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcc58046430 con 0x7fcc680670c0 2022-07-20T02:33:01.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 msgr2=0x7fcc48055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 0x7fcc48055e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fcc6400d000 tx=0x7fcc64008040).stop 2022-07-20T02:33:01.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 msgr2=0x7fcc68060c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:01.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fcc58007000 tx=0x7fcc58008040).stop 2022-07-20T02:33:01.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 shutdown_connections 2022-07-20T02:33:01.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcc48053980 0x7fcc48055e40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc68062480 0x7fcc68155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc68141a00 0x7fcc68140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 --2- 172.21.15.26:0/3731321490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcc680670c0 0x7fcc68060c10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:01.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 >> 172.21.15.26:0/3731321490 conn(0x7fcc6805f840 msgr2=0x7fcc6805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:01.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 shutdown_connections 2022-07-20T02:33:01.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:01.876+0000 7fcc6ecb0700 1 -- 172.21.15.26:0/3731321490 wait complete. 2022-07-20T02:33:01.900 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:33:01.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:02.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 Processor -- start 2022-07-20T02:33:02.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 -- start start 2022-07-20T02:33:02.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2468147000 0x7f2468143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2468060f00 0x7f246813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24681474e0 con 0x7f246805cfd0 2022-07-20T02:33:02.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24680b4e40 con 0x7f2468147000 2022-07-20T02:33:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246de52700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246805d3b0 con 0x7f2468060f00 2022-07-20T02:33:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2468147000 0x7f2468143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2468147000 0x7f2468143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40772/0 (socket says 172.21.15.26:40772) 2022-07-20T02:33:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 -- 172.21.15.26:0/2071941776 learned_addr learned my addr 172.21.15.26:0/2071941776 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f246d651700 1 --2- 172.21.15.26:0/2071941776 >> v2:172.21.15.120:3300/0 conn(0x7f2468060f00 0x7f246813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 -- 172.21.15.26:0/2071941776 >> v2:172.21.15.120:3300/0 conn(0x7f2468060f00 msgr2=0x7f246813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:02.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 --2- 172.21.15.26:0/2071941776 >> v2:172.21.15.120:3300/0 conn(0x7f2468060f00 0x7f246813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 -- 172.21.15.26:0/2071941776 >> v1:172.21.15.120:6789/0 conn(0x7f246805cfd0 legacy=0x7f24680b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.036+0000 7f2467fff700 1 -- 172.21.15.26:0/2071941776 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24681431e0 con 0x7f2468147000 2022-07-20T02:33:02.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f2467fff700 1 --2- 172.21.15.26:0/2071941776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468143bc0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f2458002f70 tx=0x7f2458006eb0).ready entity=mon.0 client_cookie=78e685bc6913e23f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f24677fe700 1 -- 172.21.15.26:0/2071941776 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800d5c0 con 0x7f2468147000 2022-07-20T02:33:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f24677fe700 1 -- 172.21.15.26:0/2071941776 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458000e90 con 0x7f2468147000 2022-07-20T02:33:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f24677fe700 1 -- 172.21.15.26:0/2071941776 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800d950 con 0x7f2468147000 2022-07-20T02:33:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 -- 172.21.15.26:0/2071941776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 msgr2=0x7f2468143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 --2- 172.21.15.26:0/2071941776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468143bc0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f2458002f70 tx=0x7f2458006eb0).stop 2022-07-20T02:33:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 -- 172.21.15.26:0/2071941776 shutdown_connections 2022-07-20T02:33:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 --2- 172.21.15.26:0/2071941776 >> v2:172.21.15.120:3300/0 conn(0x7f2468060f00 0x7f246813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 --2- 172.21.15.26:0/2071941776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468143bc0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 -- 172.21.15.26:0/2071941776 >> 172.21.15.26:0/2071941776 conn(0x7f24680620c0 msgr2=0x7f246805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 -- 172.21.15.26:0/2071941776 shutdown_connections 2022-07-20T02:33:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.040+0000 7f246de52700 1 -- 172.21.15.26:0/2071941776 wait complete. 2022-07-20T02:33:02.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 Processor -- start 2022-07-20T02:33:02.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- start start 2022-07-20T02:33:02.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2468060f00 0x7f24680b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24680b7fc0 0x7f2468145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24680b01f0 con 0x7f2468147000 2022-07-20T02:33:02.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2468152f30 con 0x7f24680b7fc0 2022-07-20T02:33:02.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2468146860 con 0x7f2468060f00 2022-07-20T02:33:02.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40778/0 (socket says 172.21.15.26:40778) 2022-07-20T02:33:02.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 -- 172.21.15.26:0/468031026 learned_addr learned my addr 172.21.15.26:0/468031026 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:02.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246ce50700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2468060f00 0x7f24680b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246d651700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24680b7fc0 0x7f2468145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 -- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2468060f00 msgr2=0x7f24680b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2468060f00 0x7f24680b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 -- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24680b7fc0 msgr2=0x7f2468145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24680b7fc0 0x7f2468145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 -- 172.21.15.26:0/468031026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245800b1e0 con 0x7f2468147000 2022-07-20T02:33:02.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2467fff700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f2458002f70 tx=0x7f2458015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24580116e0 con 0x7f2468147000 2022-07-20T02:33:02.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458011840 con 0x7f2468147000 2022-07-20T02:33:02.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2468153160 con 0x7f2468147000 2022-07-20T02:33:02.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2458011ba0 con 0x7f2468147000 2022-07-20T02:33:02.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.044+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2468145e10 con 0x7f2468147000 2022-07-20T02:33:02.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2468143480 con 0x7f2468147000 2022-07-20T02:33:02.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2458025030 con 0x7f2468147000 2022-07-20T02:33:02.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f2465ffb700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 0x7f244c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f245806d0e0 con 0x7f2468147000 2022-07-20T02:33:02.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f246ce50700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 0x7f244c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.048+0000 7f246ce50700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 0x7f244c055fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f245000d000 tx=0x7f2450007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.056+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2458043d90 con 0x7f2468147000 2022-07-20T02:33:02.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.220+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f24680574e0 con 0x7f2468147000 2022-07-20T02:33:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f2465ffb700 1 -- 172.21.15.26:0/468031026 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f24580230e0 con 0x7f2468147000 2022-07-20T02:33:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 msgr2=0x7f244c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 0x7f244c055fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f245000d000 tx=0x7f2450007b90).stop 2022-07-20T02:33:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 msgr2=0x7f2468145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f2458002f70 tx=0x7f2458015000).stop 2022-07-20T02:33:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 shutdown_connections 2022-07-20T02:33:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2468147000 0x7f2468145430 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f244c053b00 0x7f244c055fc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24680b7fc0 0x7f2468145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 --2- 172.21.15.26:0/468031026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2468060f00 0x7f24680b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 >> 172.21.15.26:0/468031026 conn(0x7f246805cfd0 msgr2=0x7f24680b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 shutdown_connections 2022-07-20T02:33:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.232+0000 7f246de52700 1 -- 172.21.15.26:0/468031026 wait complete. 2022-07-20T02:33:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:33:02.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:02.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.356+0000 7f43ecdcb700 1 Processor -- start 2022-07-20T02:33:02.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.356+0000 7f43ecdcb700 1 -- start start 2022-07-20T02:33:02.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 0x7f43e8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43e8061050 0x7f43e8148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80b1bc0 con 0x7f43e8066a80 2022-07-20T02:33:02.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e814b040 con 0x7f43e8061050 2022-07-20T02:33:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e80a8860 con 0x7f43e80b2920 2022-07-20T02:33:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43e8061050 0x7f43e8148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e6d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f43e8066a80 0x7f43e8066ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40068/0 (socket says 172.21.15.26:40068) 2022-07-20T02:33:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43e8061050 0x7f43e8148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40784/0 (socket says 172.21.15.26:40784) 2022-07-20T02:33:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e6d9d700 1 -- 172.21.15.26:0/2232820170 learned_addr learned my addr 172.21.15.26:0/2232820170 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e659c700 1 --2- 172.21.15.26:0/2232820170 >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 0x7f43e8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e559a700 1 -- 172.21.15.26:0/2232820170 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679366185 0 0) 0x7f43e80b1bc0 con 0x7f43e8066a80 2022-07-20T02:33:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e559a700 1 -- 172.21.15.26:0/2232820170 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c8003260 con 0x7f43e8066a80 2022-07-20T02:33:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 -- 172.21.15.26:0/2232820170 >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 msgr2=0x7f43e8147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 --2- 172.21.15.26:0/2232820170 >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 0x7f43e8147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 -- 172.21.15.26:0/2232820170 >> v1:172.21.15.26:6789/0 conn(0x7f43e8066a80 legacy=0x7f43e8066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 -- 172.21.15.26:0/2232820170 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8147700 con 0x7f43e8061050 2022-07-20T02:33:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e659c700 1 --2- 172.21.15.26:0/2232820170 >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 0x7f43e8147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e5d9b700 1 --2- 172.21.15.26:0/2232820170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e8061050 0x7f43e8148280 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f43d8002f70 tx=0x7f43d8006df0).ready entity=mon.0 client_cookie=6655d42a8062c500 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e559a700 1 -- 172.21.15.26:0/2232820170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d800d570 con 0x7f43e8061050 2022-07-20T02:33:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e559a700 1 -- 172.21.15.26:0/2232820170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8000e90 con 0x7f43e8061050 2022-07-20T02:33:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43e559a700 1 -- 172.21.15.26:0/2232820170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d800d8c0 con 0x7f43e8061050 2022-07-20T02:33:02.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- 172.21.15.26:0/2232820170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e8061050 msgr2=0x7f43e8148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/2232820170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e8061050 0x7f43e8148280 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f43d8002f70 tx=0x7f43d8006df0).stop 2022-07-20T02:33:02.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- 172.21.15.26:0/2232820170 shutdown_connections 2022-07-20T02:33:02.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/2232820170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e8061050 0x7f43e8148280 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/2232820170 >> v2:172.21.15.120:3300/0 conn(0x7f43e80b2920 0x7f43e8147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- 172.21.15.26:0/2232820170 >> 172.21.15.26:0/2232820170 conn(0x7f43e80b60a0 msgr2=0x7f43e80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- 172.21.15.26:0/2232820170 shutdown_connections 2022-07-20T02:33:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.360+0000 7f43ecdcb700 1 -- 172.21.15.26:0/2232820170 wait complete. 2022-07-20T02:33:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 Processor -- start 2022-07-20T02:33:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- start start 2022-07-20T02:33:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e8061050 0x7f43e8066390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43e8066a80 0x7f43e8143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e8159e40 con 0x7f43e80b2920 2022-07-20T02:33:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e81443a0 con 0x7f43e8061050 2022-07-20T02:33:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e8143e60 con 0x7f43e8066a80 2022-07-20T02:33:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40792/0 (socket says 172.21.15.26:40792) 2022-07-20T02:33:02.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 -- 172.21.15.26:0/956077376 learned_addr learned my addr 172.21.15.26:0/956077376 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:02.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e5d9b700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43e8066a80 0x7f43e8143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e659c700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e8061050 0x7f43e8066390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 -- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43e8066a80 msgr2=0x7f43e8143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43e8066a80 0x7f43e8143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 -- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e8061050 msgr2=0x7f43e8066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e8061050 0x7f43e8066390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 -- 172.21.15.26:0/956077376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d800b180 con 0x7f43e80b2920 2022-07-20T02:33:02.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43e6d9d700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f43e0001490 tx=0x7f43e0007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e0005ba0 con 0x7f43e80b2920 2022-07-20T02:33:02.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e0005d00 con 0x7f43e80b2920 2022-07-20T02:33:02.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e8144100 con 0x7f43e80b2920 2022-07-20T02:33:02.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e00072c0 con 0x7f43e80b2920 2022-07-20T02:33:02.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.364+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e8156a00 con 0x7f43e80b2920 2022-07-20T02:33:02.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e814ada0 con 0x7f43e80b2920 2022-07-20T02:33:02.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43e0010070 con 0x7f43e80b2920 2022-07-20T02:33:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43df7fe700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 0x7f43cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f43e006b870 con 0x7f43e80b2920 2022-07-20T02:33:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43e659c700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 0x7f43cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.368+0000 7f43e659c700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 0x7f43cc055f60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f43d000c000 tx=0x7f43d0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.376+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43e0042630 con 0x7f43e80b2920 2022-07-20T02:33:02.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.536+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e80574e0 con 0x7f43e80b2920 2022-07-20T02:33:02.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.536+0000 7f43df7fe700 1 -- 172.21.15.26:0/956077376 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f43e0018070 con 0x7f43e80b2920 2022-07-20T02:33:02.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:02.548 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 msgr2=0x7f43cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 0x7f43cc055f60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f43d000c000 tx=0x7f43d0008040).stop 2022-07-20T02:33:02.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 msgr2=0x7f43e8156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:02.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f43e0001490 tx=0x7f43e0007e40).stop 2022-07-20T02:33:02.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 shutdown_connections 2022-07-20T02:33:02.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43e80b2920 0x7f43e8156f40 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43cc053aa0 0x7f43cc055f60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43e8066a80 0x7f43e8143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 --2- 172.21.15.26:0/956077376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e8061050 0x7f43e8066390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:02.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 >> 172.21.15.26:0/956077376 conn(0x7f43e80b60a0 msgr2=0x7f43e814a220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:02.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 shutdown_connections 2022-07-20T02:33:02.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:02.540+0000 7f43ecdcb700 1 -- 172.21.15.26:0/956077376 wait complete. 2022-07-20T02:33:02.559 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:02.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:59.904474+0000', 'modified': '2022-07-20T02:33:00.907175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7053}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7053': {'gid': 7053, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2339237946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2339237946}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2339237946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:02.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:02.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:02.995 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7053,"want_state":"up:active","state":"up:active","rank_uptime":3.0777655570000002,"mdsmap_epoch":48,"osdmap_epoch":98,"osdmap_epoch_barrier":97,"uptime":8.2056233580000004} 2022-07-20T02:33:02.998 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7053, "mdsmap_epoch": 48, "osdmap_epoch": 98, "osdmap_epoch_barrier": 97, "rank_uptime": 3.077765557, "state": "up:active", "uptime": 8.205623358, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:03.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:03.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.120+0000 7fc4370b3700 1 Processor -- start 2022-07-20T02:33:03.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.120+0000 7fc4370b3700 1 -- start start 2022-07-20T02:33:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc43005cfd0 0x7fc430143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc430147000 0x7fc4300b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4301474e0 con 0x7fc430060f00 2022-07-20T02:33:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43005d3b0 con 0x7fc43005cfd0 2022-07-20T02:33:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4301431e0 con 0x7fc430147000 2022-07-20T02:33:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc43005cfd0 0x7fc430143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc43005cfd0 0x7fc430143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40798/0 (socket says 172.21.15.26:40798) 2022-07-20T02:33:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 -- 172.21.15.26:0/461098479 learned_addr learned my addr 172.21.15.26:0/461098479 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4368b2700 1 --1- 172.21.15.26:0/461098479 >> v1:172.21.15.26:6789/0 conn(0x7fc430060f00 0x7fc43013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40080/0 (socket says 172.21.15.26:40080) 2022-07-20T02:33:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4358b0700 1 --2- 172.21.15.26:0/461098479 >> v2:172.21.15.120:3300/0 conn(0x7fc430147000 0x7fc4300b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 -- 172.21.15.26:0/461098479 >> v2:172.21.15.120:3300/0 conn(0x7fc430147000 msgr2=0x7fc4300b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4350af700 1 -- 172.21.15.26:0/461098479 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897515261 0 0) 0x7fc4301474e0 con 0x7fc430060f00 2022-07-20T02:33:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 --2- 172.21.15.26:0/461098479 >> v2:172.21.15.120:3300/0 conn(0x7fc430147000 0x7fc4300b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 -- 172.21.15.26:0/461098479 >> v1:172.21.15.26:6789/0 conn(0x7fc430060f00 legacy=0x7fc43013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 -- 172.21.15.26:0/461098479 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430143480 con 0x7fc43005cfd0 2022-07-20T02:33:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4360b1700 1 --2- 172.21.15.26:0/461098479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc43005cfd0 0x7fc430143bc0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7fc4280032f0 tx=0x7fc428009290).ready entity=mon.0 client_cookie=b0147173dbf9e956 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4350af700 1 -- 172.21.15.26:0/461098479 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42800b870 con 0x7fc43005cfd0 2022-07-20T02:33:03.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4350af700 1 -- 172.21.15.26:0/461098479 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42800b9d0 con 0x7fc43005cfd0 2022-07-20T02:33:03.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4350af700 1 -- 172.21.15.26:0/461098479 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42800bb40 con 0x7fc43005cfd0 2022-07-20T02:33:03.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- 172.21.15.26:0/461098479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc43005cfd0 msgr2=0x7fc430143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 --2- 172.21.15.26:0/461098479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc43005cfd0 0x7fc430143bc0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7fc4280032f0 tx=0x7fc428009290).stop 2022-07-20T02:33:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- 172.21.15.26:0/461098479 shutdown_connections 2022-07-20T02:33:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 --2- 172.21.15.26:0/461098479 >> v2:172.21.15.120:3300/0 conn(0x7fc430147000 0x7fc4300b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 --2- 172.21.15.26:0/461098479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc43005cfd0 0x7fc430143bc0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- 172.21.15.26:0/461098479 >> 172.21.15.26:0/461098479 conn(0x7fc4300620c0 msgr2=0x7fc43005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- 172.21.15.26:0/461098479 shutdown_connections 2022-07-20T02:33:03.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.124+0000 7fc4370b3700 1 -- 172.21.15.26:0/461098479 wait complete. 2022-07-20T02:33:03.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 Processor -- start 2022-07-20T02:33:03.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 -- start start 2022-07-20T02:33:03.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc430147000 0x7fc4301520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4300b1e80 0x7fc4301525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4300b2950 con 0x7fc430060f00 2022-07-20T02:33:03.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430144420 con 0x7fc4300b1e80 2022-07-20T02:33:03.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4370b3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4301446c0 con 0x7fc430147000 2022-07-20T02:33:03.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40804/0 (socket says 172.21.15.26:40804) 2022-07-20T02:33:03.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 -- 172.21.15.26:0/2598369861 learned_addr learned my addr 172.21.15.26:0/2598369861 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:03.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4358b0700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc430147000 0x7fc4301520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4368b2700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4300b1e80 0x7fc4301525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 -- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc430147000 msgr2=0x7fc4301520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc430147000 0x7fc4301520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 -- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4300b1e80 msgr2=0x7fc4301525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4300b1e80 0x7fc4301525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 -- 172.21.15.26:0/2598369861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428009530 con 0x7fc430060f00 2022-07-20T02:33:03.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.128+0000 7fc4360b1700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7fc4280032f0 tx=0x7fc428016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc428021630 con 0x7fc430060f00 2022-07-20T02:33:03.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc428021790 con 0x7fc430060f00 2022-07-20T02:33:03.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430144960 con 0x7fc430060f00 2022-07-20T02:33:03.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc428021af0 con 0x7fc430060f00 2022-07-20T02:33:03.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc430144e00 con 0x7fc430060f00 2022-07-20T02:33:03.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430143720 con 0x7fc430060f00 2022-07-20T02:33:03.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc428027020 con 0x7fc430060f00 2022-07-20T02:33:03.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 0x7fc410055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc42806d860 con 0x7fc430060f00 2022-07-20T02:33:03.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4358b0700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 0x7fc410055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.132+0000 7fc4358b0700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 0x7fc410055fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fc42c00c000 tx=0x7fc42c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.140+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc428044620 con 0x7fc430060f00 2022-07-20T02:33:03.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.268+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4300574e0 con 0x7fc430060f00 2022-07-20T02:33:03.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.268+0000 7fc4237fe700 1 -- 172.21.15.26:0/2598369861 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16322 (secure 0 0 0) 0x7fc428015020 con 0x7fc430060f00 2022-07-20T02:33:03.286 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:03.287 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":98,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:00.907866+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:57.859395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:58.906424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 msgr2=0x7fc410055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 0x7fc410055fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fc42c00c000 tx=0x7fc42c008040).stop 2022-07-20T02:33:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 msgr2=0x7fc4300b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7fc4280032f0 tx=0x7fc428016000).stop 2022-07-20T02:33:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 shutdown_connections 2022-07-20T02:33:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc410053b00 0x7fc410055fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4300b1e80 0x7fc4301525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc430147000 0x7fc4301520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 --2- 172.21.15.26:0/2598369861 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc430060f00 0x7fc4300b3470 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 >> 172.21.15.26:0/2598369861 conn(0x7fc43005cfd0 msgr2=0x7fc430139370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 shutdown_connections 2022-07-20T02:33:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.272+0000 7fc4370b3700 1 -- 172.21.15.26:0/2598369861 wait complete. 2022-07-20T02:33:03.308 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:32:57.859395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:32:58.906424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:03.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:03.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 Processor -- start 2022-07-20T02:33:03.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- start start 2022-07-20T02:33:03.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f104405cfd0 0x7f1044143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1044147000 0x7f10440b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10441474e0 con 0x7f1044060f00 2022-07-20T02:33:03.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104405d3b0 con 0x7f104405cfd0 2022-07-20T02:33:03.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10441431e0 con 0x7f1044147000 2022-07-20T02:33:03.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f104405cfd0 0x7f1044143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f10437fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1044060f00 0x7f104413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40094/0 (socket says 172.21.15.26:40094) 2022-07-20T02:33:03.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f104405cfd0 0x7f1044143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40812/0 (socket says 172.21.15.26:40812) 2022-07-20T02:33:03.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f10437fe700 1 -- 172.21.15.26:0/3105166049 learned_addr learned my addr 172.21.15.26:0/3105166049 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:03.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f10427fc700 1 --2- 172.21.15.26:0/3105166049 >> v2:172.21.15.120:3300/0 conn(0x7f1044147000 0x7f10440b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 -- 172.21.15.26:0/3105166049 >> v2:172.21.15.120:3300/0 conn(0x7f1044147000 msgr2=0x7f10440b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 --2- 172.21.15.26:0/3105166049 >> v2:172.21.15.120:3300/0 conn(0x7f1044147000 0x7f10440b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 -- 172.21.15.26:0/3105166049 >> v1:172.21.15.26:6789/0 conn(0x7f1044060f00 legacy=0x7f104413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 -- 172.21.15.26:0/3105166049 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044143480 con 0x7f104405cfd0 2022-07-20T02:33:03.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1042ffd700 1 --2- 172.21.15.26:0/3105166049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f104405cfd0 0x7f1044143bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f1034002f70 tx=0x7f1034006df0).ready entity=mon.0 client_cookie=3f9f00904140dfd4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1041ffb700 1 -- 172.21.15.26:0/3105166049 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f103400d570 con 0x7f104405cfd0 2022-07-20T02:33:03.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1041ffb700 1 -- 172.21.15.26:0/3105166049 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034000e90 con 0x7f104405cfd0 2022-07-20T02:33:03.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1041ffb700 1 -- 172.21.15.26:0/3105166049 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f103400d8c0 con 0x7f104405cfd0 2022-07-20T02:33:03.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- 172.21.15.26:0/3105166049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f104405cfd0 msgr2=0x7f1044143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 --2- 172.21.15.26:0/3105166049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f104405cfd0 0x7f1044143bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f1034002f70 tx=0x7f1034006df0).stop 2022-07-20T02:33:03.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- 172.21.15.26:0/3105166049 shutdown_connections 2022-07-20T02:33:03.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 --2- 172.21.15.26:0/3105166049 >> v2:172.21.15.120:3300/0 conn(0x7f1044147000 0x7f10440b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 --2- 172.21.15.26:0/3105166049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f104405cfd0 0x7f1044143bc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- 172.21.15.26:0/3105166049 >> 172.21.15.26:0/3105166049 conn(0x7f10440620c0 msgr2=0x7f104405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- 172.21.15.26:0/3105166049 shutdown_connections 2022-07-20T02:33:03.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.424+0000 7f1043fff700 1 -- 172.21.15.26:0/3105166049 wait complete. 2022-07-20T02:33:03.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 Processor -- start 2022-07-20T02:33:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 -- start start 2022-07-20T02:33:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1044060f00 0x7f1044145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1044142d30 0x7f10440b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10440b1c10 con 0x7f1044147000 2022-07-20T02:33:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10440b2a80 con 0x7f1044142d30 2022-07-20T02:33:03.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.428+0000 7f1043fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10440b1e00 con 0x7f1044060f00 2022-07-20T02:33:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40820/0 (socket says 172.21.15.26:40820) 2022-07-20T02:33:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 -- 172.21.15.26:0/3413733437 learned_addr learned my addr 172.21.15.26:0/3413733437 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1042ffd700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1044060f00 0x7f1044145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10437fe700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1044142d30 0x7f10440b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 -- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1044060f00 msgr2=0x7f1044145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1044060f00 0x7f1044145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 -- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1044142d30 msgr2=0x7f10440b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1044142d30 0x7f10440b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 -- 172.21.15.26:0/3413733437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103400b180 con 0x7f1044147000 2022-07-20T02:33:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f10427fc700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f10440b81d0 tx=0x7f1038006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1038011750 con 0x7f1044147000 2022-07-20T02:33:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10380118b0 con 0x7f1044147000 2022-07-20T02:33:03.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10440b2cb0 con 0x7f1044147000 2022-07-20T02:33:03.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1038011c10 con 0x7f1044147000 2022-07-20T02:33:03.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10440b9400 con 0x7f1044147000 2022-07-20T02:33:03.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1038011d70 con 0x7f1044147000 2022-07-20T02:33:03.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10440b3220 con 0x7f1044147000 2022-07-20T02:33:03.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.432+0000 7f1033fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 0x7f1020055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.436+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f103806c370 con 0x7f1044147000 2022-07-20T02:33:03.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.436+0000 7f1042ffd700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 0x7f1020055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.436+0000 7f1042ffd700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 0x7f1020055f60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f103401a000 tx=0x7f1034014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.440+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1038043030 con 0x7f1044147000 2022-07-20T02:33:03.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.564+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1044155a00 con 0x7f1044147000 2022-07-20T02:33:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1033fff700 1 -- 172.21.15.26:0/3413733437 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v98) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1038046650 con 0x7f1044147000 2022-07-20T02:33:03.573 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 msgr2=0x7f1020055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 0x7f1020055f60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f103401a000 tx=0x7f1034014000).stop 2022-07-20T02:33:03.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 msgr2=0x7f10440b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f10440b81d0 tx=0x7f1038006df0).stop 2022-07-20T02:33:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 shutdown_connections 2022-07-20T02:33:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1044147000 0x7f10440b3420 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1020053aa0 0x7f1020055f60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1044142d30 0x7f10440b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 --2- 172.21.15.26:0/3413733437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1044060f00 0x7f1044145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 >> 172.21.15.26:0/3413733437 conn(0x7f104405cfd0 msgr2=0x7f10440b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 shutdown_connections 2022-07-20T02:33:03.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.568+0000 7f1043fff700 1 -- 172.21.15.26:0/3413733437 wait complete. 2022-07-20T02:33:03.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 Processor -- start 2022-07-20T02:33:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- start start 2022-07-20T02:33:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fad680618a0 0x7fad68147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad68061c80 con 0x7fad680b22d0 2022-07-20T02:33:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad680b2810 con 0x7fad68143f80 2022-07-20T02:33:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad680b2ab0 con 0x7fad680618a0 2022-07-20T02:33:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6cd4d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fad680618a0 0x7fad68147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fad680b22d0 0x7fad68143870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40106/0 (socket says 172.21.15.26:40106) 2022-07-20T02:33:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5ffff700 1 -- 172.21.15.26:0/2898369273 learned_addr learned my addr 172.21.15.26:0/2898369273 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5f7fe700 1 -- 172.21.15.26:0/2898369273 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273193059 0 0) 0x7fad68061c80 con 0x7fad680b22d0 2022-07-20T02:33:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2898369273 >> v1:172.21.15.120:6789/0 conn(0x7fad68143f80 legacy=0x7fad680b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2898369273 >> v1:172.21.15.26:6789/0 conn(0x7fad680b22d0 legacy=0x7fad68143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2898369273 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680b2d50 con 0x7fad680618a0 2022-07-20T02:33:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6cd4d700 1 --2- 172.21.15.26:0/2898369273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad680618a0 0x7fad68147250 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fad64002f70 tx=0x7fad64006df0).ready entity=mon.0 client_cookie=35bbb1e6144ad8e6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5f7fe700 1 -- 172.21.15.26:0/2898369273 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6400d610 con 0x7fad680618a0 2022-07-20T02:33:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5f7fe700 1 -- 172.21.15.26:0/2898369273 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64000e90 con 0x7fad680618a0 2022-07-20T02:33:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad5f7fe700 1 -- 172.21.15.26:0/2898369273 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6400d960 con 0x7fad680618a0 2022-07-20T02:33:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2898369273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad680618a0 msgr2=0x7fad68147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2898369273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad680618a0 0x7fad68147250 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fad64002f70 tx=0x7fad64006df0).stop 2022-07-20T02:33:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2898369273 shutdown_connections 2022-07-20T02:33:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2898369273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad680618a0 0x7fad68147250 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2898369273 >> 172.21.15.26:0/2898369273 conn(0x7fad68066f70 msgr2=0x7fad68067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2898369273 shutdown_connections 2022-07-20T02:33:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.704+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2898369273 wait complete. 2022-07-20T02:33:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 Processor -- start 2022-07-20T02:33:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 -- start start 2022-07-20T02:33:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 0x7fad6812d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad680b22d0 0x7fad68145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 0x7fad681530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68141e90 con 0x7fad68143f80 2022-07-20T02:33:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6813fc80 con 0x7fad680618a0 2022-07-20T02:33:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6dd4f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6813ff20 con 0x7fad680b22d0 2022-07-20T02:33:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6d54e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 0x7fad681530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6d54e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 0x7fad681530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40836/0 (socket says 172.21.15.26:40836) 2022-07-20T02:33:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6d54e700 1 -- 172.21.15.26:0/2359082202 learned_addr learned my addr 172.21.15.26:0/2359082202 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:03.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 0x7fad6812d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad5ffff700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad680b22d0 0x7fad68145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad680b22d0 msgr2=0x7fad68145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad680b22d0 0x7fad68145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 msgr2=0x7fad681530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 0x7fad681530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 -- 172.21.15.26:0/2359082202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6400b180 con 0x7fad680618a0 2022-07-20T02:33:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.708+0000 7fad6cd4d700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 0x7fad6812d2a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fad68141fd0 tx=0x7fad64015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6400d990 con 0x7fad680618a0 2022-07-20T02:33:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad681401c0 con 0x7fad680618a0 2022-07-20T02:33:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64011b40 con 0x7fad680618a0 2022-07-20T02:33:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad64011ea0 con 0x7fad680618a0 2022-07-20T02:33:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad681535e0 con 0x7fad680618a0 2022-07-20T02:33:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad680574e0 con 0x7fad680618a0 2022-07-20T02:33:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad6401a040 con 0x7fad680618a0 2022-07-20T02:33:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad6406d010 con 0x7fad680618a0 2022-07-20T02:33:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5ffff700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.712+0000 7fad5ffff700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 0x7fad48055ea0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fad5800d000 tx=0x7fad58009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.720+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad64043d30 con 0x7fad680618a0 2022-07-20T02:33:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fad680b9550 con 0x7fad680618a0 2022-07-20T02:33:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad5dffb700 1 -- 172.21.15.26:0/2359082202 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7fad64023020 con 0x7fad680618a0 2022-07-20T02:33:03.860 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:03.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 msgr2=0x7fad48055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 0x7fad48055ea0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fad5800d000 tx=0x7fad58009040).stop 2022-07-20T02:33:03.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 msgr2=0x7fad6812d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:03.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 0x7fad6812d2a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fad68141fd0 tx=0x7fad64015000).stop 2022-07-20T02:33:03.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 shutdown_connections 2022-07-20T02:33:03.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad68143f80 0x7fad681530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad680b22d0 0x7fad68145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 --2- 172.21.15.26:0/2359082202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad680618a0 0x7fad6812d2a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:03.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 >> 172.21.15.26:0/2359082202 conn(0x7fad68066f70 msgr2=0x7fad680b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:03.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 shutdown_connections 2022-07-20T02:33:03.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:03.852+0000 7fad6dd4f700 1 -- 172.21.15.26:0/2359082202 wait complete. 2022-07-20T02:33:03.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:04.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 Processor -- start 2022-07-20T02:33:04.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 -- start start 2022-07-20T02:33:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62840b5ef0 0x7f62841474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62840a8c40 0x7f62840af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840b62d0 con 0x7f62840a7fc0 2022-07-20T02:33:04.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628414aac0 con 0x7f62840b5ef0 2022-07-20T02:33:04.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6289288700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628414ad60 con 0x7f62840a8c40 2022-07-20T02:33:04.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62840b5ef0 0x7f62841474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f62840b5ef0 0x7f62841474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40840/0 (socket says 172.21.15.26:40840) 2022-07-20T02:33:04.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 -- 172.21.15.26:0/3722358115 learned_addr learned my addr 172.21.15.26:0/3722358115 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6288a87700 1 --2- 172.21.15.26:0/3722358115 >> v2:172.21.15.120:3300/0 conn(0x7f62840a8c40 0x7f62840af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 -- 172.21.15.26:0/3722358115 >> v2:172.21.15.120:3300/0 conn(0x7f62840a8c40 msgr2=0x7f62840af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:04.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 --2- 172.21.15.26:0/3722358115 >> v2:172.21.15.120:3300/0 conn(0x7f62840a8c40 0x7f62840af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 -- 172.21.15.26:0/3722358115 >> v1:172.21.15.120:6789/0 conn(0x7f62840a7fc0 legacy=0x7f62840aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.004+0000 7f6283fff700 1 -- 172.21.15.26:0/3722358115 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628414b000 con 0x7f62840b5ef0 2022-07-20T02:33:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6283fff700 1 --2- 172.21.15.26:0/3722358115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841474d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f6278002f70 tx=0x7f6278006eb0).ready entity=mon.0 client_cookie=210a8d23f3469bf3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6282ffd700 1 -- 172.21.15.26:0/3722358115 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f627800d5c0 con 0x7f62840b5ef0 2022-07-20T02:33:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6282ffd700 1 -- 172.21.15.26:0/3722358115 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278000e90 con 0x7f62840b5ef0 2022-07-20T02:33:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6282ffd700 1 -- 172.21.15.26:0/3722358115 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f627800d950 con 0x7f62840b5ef0 2022-07-20T02:33:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- 172.21.15.26:0/3722358115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 msgr2=0x7f62841474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- 172.21.15.26:0/3722358115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841474d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f6278002f70 tx=0x7f6278006eb0).stop 2022-07-20T02:33:04.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- 172.21.15.26:0/3722358115 shutdown_connections 2022-07-20T02:33:04.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- 172.21.15.26:0/3722358115 >> v2:172.21.15.120:3300/0 conn(0x7f62840a8c40 0x7f62840af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- 172.21.15.26:0/3722358115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841474d0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- 172.21.15.26:0/3722358115 >> 172.21.15.26:0/3722358115 conn(0x7f6284060310 msgr2=0x7f62840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- 172.21.15.26:0/3722358115 shutdown_connections 2022-07-20T02:33:04.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- 172.21.15.26:0/3722358115 wait complete. 2022-07-20T02:33:04.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 Processor -- start 2022-07-20T02:33:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- start start 2022-07-20T02:33:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62840a7fc0 0x7f62840635c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62840a8c40 0x7f62840a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62841571d0 con 0x7f62840b5ef0 2022-07-20T02:33:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840abeb0 con 0x7f62840a7fc0 2022-07-20T02:33:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.008+0000 7f6289288700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6284157400 con 0x7f62840a8c40 2022-07-20T02:33:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40850/0 (socket says 172.21.15.26:40850) 2022-07-20T02:33:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 -- 172.21.15.26:0/1118053627 learned_addr learned my addr 172.21.15.26:0/1118053627 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6283fff700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62840a7fc0 0x7f62840635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f62837fe700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62840a8c40 0x7f62840a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 -- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62840a8c40 msgr2=0x7f62840a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62840a8c40 0x7f62840a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 -- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62840a7fc0 msgr2=0x7f62840635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62840a7fc0 0x7f62840635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 -- 172.21.15.26:0/1118053627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627800b1e0 con 0x7f62840b5ef0 2022-07-20T02:33:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6288a87700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f627c00c000 tx=0x7f627c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f627c011750 con 0x7f62840b5ef0 2022-07-20T02:33:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c0118b0 con 0x7f62840b5ef0 2022-07-20T02:33:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6284063e00 con 0x7f62840b5ef0 2022-07-20T02:33:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6284062f60 con 0x7f62840b5ef0 2022-07-20T02:33:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f627c011c10 con 0x7f62840b5ef0 2022-07-20T02:33:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62841575c0 con 0x7f62840b5ef0 2022-07-20T02:33:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.012+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f627c011d70 con 0x7f62840b5ef0 2022-07-20T02:33:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.016+0000 7f62817fa700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 0x7f6264055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.016+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f627c06c4b0 con 0x7f62840b5ef0 2022-07-20T02:33:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.016+0000 7f6283fff700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 0x7f6264055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.016+0000 7f6283fff700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 0x7f6264055f30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f627801c000 tx=0x7f6278015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.020+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f627c043270 con 0x7f62840b5ef0 2022-07-20T02:33:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.144+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f628414b2a0 con 0x7f62840b5ef0 2022-07-20T02:33:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.144+0000 7f62817fa700 1 -- 172.21.15.26:0/1118053627 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+10 (secure 0 0 0) 0x7f627c046890 con 0x7f62840b5ef0 2022-07-20T02:33:04.152 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 msgr2=0x7f6264055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 0x7f6264055f30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f627801c000 tx=0x7f6278015000).stop 2022-07-20T02:33:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 msgr2=0x7f62841478c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f627c00c000 tx=0x7f627c006df0).stop 2022-07-20T02:33:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 shutdown_connections 2022-07-20T02:33:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f62840b5ef0 0x7f62841478c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6264053a70 0x7f6264055f30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62840a8c40 0x7f62840a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 --2- 172.21.15.26:0/1118053627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62840a7fc0 0x7f62840635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 >> 172.21.15.26:0/1118053627 conn(0x7f6284060310 msgr2=0x7f6284148ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 shutdown_connections 2022-07-20T02:33:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.148+0000 7f6289288700 1 -- 172.21.15.26:0/1118053627 wait complete. 2022-07-20T02:33:04.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 Processor -- start 2022-07-20T02:33:04.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 -- start start 2022-07-20T02:33:04.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b413fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff6b40a9ea0 0x7ff6b40aa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b40a8cc0 con 0x7ff6b40ac940 2022-07-20T02:33:04.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b40aba20 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b40ab4e0 con 0x7ff6b413f660 2022-07-20T02:33:04.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b2d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff6b40a9ea0 0x7ff6b40aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b2d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff6b40a9ea0 0x7ff6b40aa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40858/0 (socket says 172.21.15.26:40858) 2022-07-20T02:33:04.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b2d9d700 1 -- 172.21.15.26:0/4065673470 learned_addr learned my addr 172.21.15.26:0/4065673470 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b259c700 1 --1- 172.21.15.26:0/4065673470 >> v1:172.21.15.26:6789/0 conn(0x7ff6b40ac940 0x7ff6b40acd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40134/0 (socket says 172.21.15.26:40134) 2022-07-20T02:33:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.276+0000 7ff6b1d9b700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b413fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b2d9d700 1 -- 172.21.15.26:0/4065673470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 msgr2=0x7ff6b413fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b2d9d700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b413fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b2d9d700 1 -- 172.21.15.26:0/4065673470 >> v1:172.21.15.26:6789/0 conn(0x7ff6b40ac940 legacy=0x7ff6b40acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b2d9d700 1 -- 172.21.15.26:0/4065673470 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b40ab240 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b2d9d700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b40aa330 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7ff6ac0032f0 tx=0x7ff6ac00b530).ready entity=mon.0 client_cookie=2ce9ef08b12deb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b159a700 1 -- 172.21.15.26:0/4065673470 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6ac010660 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b159a700 1 -- 172.21.15.26:0/4065673470 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac0107c0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b159a700 1 -- 172.21.15.26:0/4065673470 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6ac010b20 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- 172.21.15.26:0/4065673470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 msgr2=0x7ff6b40aa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b40aa330 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7ff6ac0032f0 tx=0x7ff6ac00b530).stop 2022-07-20T02:33:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- 172.21.15.26:0/4065673470 shutdown_connections 2022-07-20T02:33:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b40aa330 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 --2- 172.21.15.26:0/4065673470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b413fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- 172.21.15.26:0/4065673470 >> 172.21.15.26:0/4065673470 conn(0x7ff6b4060310 msgr2=0x7ff6b40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- 172.21.15.26:0/4065673470 shutdown_connections 2022-07-20T02:33:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- 172.21.15.26:0/4065673470 wait complete. 2022-07-20T02:33:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 Processor -- start 2022-07-20T02:33:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.280+0000 7ff6b359e700 1 -- start start 2022-07-20T02:33:04.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 0x7ff6b40b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b41409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4140ed0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4141ef0 con 0x7ff6b40ac940 2022-07-20T02:33:04.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4141960 con 0x7ff6b413f660 2022-07-20T02:33:04.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40860/0 (socket says 172.21.15.26:40860) 2022-07-20T02:33:04.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 -- 172.21.15.26:0/2886000761 learned_addr learned my addr 172.21.15.26:0/2886000761 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b1d9b700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 0x7ff6b40b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b2d9d700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b41409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 -- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 msgr2=0x7ff6b41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b41409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 -- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 msgr2=0x7ff6b40b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 0x7ff6b40b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 -- 172.21.15.26:0/2886000761 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac00b7d0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b2d9d700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b41409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:04.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b1d9b700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 0x7ff6b40b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:04.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b259c700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7ff6b40afd30 tx=0x7ff6a4007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6a400ce30 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a40072e0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b40adbb0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6a4005740 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b40af9b0 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.284+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b4141d50 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.288+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff6a4010070 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.288+0000 7ff6ab7fe700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 0x7ff698055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.288+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff6a406b770 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.288+0000 7ff6b1d9b700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 0x7ff698055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.288+0000 7ff6b1d9b700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 0x7ff698055ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff69c008000 tx=0x7ff69c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.296+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff6b4141d50 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6b4141d50 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6ab7fe700 1 -- 172.21.15.26:0/2886000761 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff6b4141d50 con 0x7ff6b40a9ea0 2022-07-20T02:33:04.430 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 msgr2=0x7ff698055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 0x7ff698055ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff69c008000 tx=0x7ff69c008040).stop 2022-07-20T02:33:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 msgr2=0x7ff6b4142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7ff6b40afd30 tx=0x7ff6a4007c00).stop 2022-07-20T02:33:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 shutdown_connections 2022-07-20T02:33:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff698053a10 0x7ff698055ed0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6b413f660 0x7ff6b41409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6b40ac940 0x7ff6b40b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 --2- 172.21.15.26:0/2886000761 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6b40a9ea0 0x7ff6b4142500 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 >> 172.21.15.26:0/2886000761 conn(0x7ff6b4060310 msgr2=0x7ff6b413d500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 shutdown_connections 2022-07-20T02:33:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.424+0000 7ff6b359e700 1 -- 172.21.15.26:0/2886000761 wait complete. 2022-07-20T02:33:04.444 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:04.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 Processor -- start 2022-07-20T02:33:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- start start 2022-07-20T02:33:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f44940b2920 0x7f4494147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 0x7f4494148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44940b1bc0 con 0x7f4494061050 2022-07-20T02:33:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449414b040 con 0x7f4494067190 2022-07-20T02:33:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44940a8860 con 0x7f44940b2920 2022-07-20T02:33:04.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449a271700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 0x7f4494148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449a271700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 0x7f4494148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40870/0 (socket says 172.21.15.26:40870) 2022-07-20T02:33:04.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449a271700 1 -- 172.21.15.26:0/577120426 learned_addr learned my addr 172.21.15.26:0/577120426 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 --2- 172.21.15.26:0/577120426 >> v2:172.21.15.120:3300/0 conn(0x7f44940b2920 0x7f4494147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 -- 172.21.15.26:0/577120426 >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 msgr2=0x7f4494148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 --2- 172.21.15.26:0/577120426 >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 0x7f4494148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 -- 172.21.15.26:0/577120426 >> v1:172.21.15.120:6789/0 conn(0x7f4494061050 legacy=0x7f4494066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 -- 172.21.15.26:0/577120426 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494147700 con 0x7f44940b2920 2022-07-20T02:33:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4499a70700 1 --2- 172.21.15.26:0/577120426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494147d90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f44840032f0 tx=0x7f44840091d0).ready entity=mon.1 client_cookie=44748a316c6851fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4498a6e700 1 -- 172.21.15.26:0/577120426 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448400b680 con 0x7f44940b2920 2022-07-20T02:33:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4498a6e700 1 -- 172.21.15.26:0/577120426 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448400b7e0 con 0x7f44940b2920 2022-07-20T02:33:04.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f4498a6e700 1 -- 172.21.15.26:0/577120426 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448400b950 con 0x7f44940b2920 2022-07-20T02:33:04.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- 172.21.15.26:0/577120426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 msgr2=0x7f4494147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 --2- 172.21.15.26:0/577120426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494147d90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f44840032f0 tx=0x7f44840091d0).stop 2022-07-20T02:33:04.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- 172.21.15.26:0/577120426 shutdown_connections 2022-07-20T02:33:04.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 --2- 172.21.15.26:0/577120426 >> v2:172.21.15.26:3300/0 conn(0x7f4494067190 0x7f4494148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 --2- 172.21.15.26:0/577120426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494147d90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- 172.21.15.26:0/577120426 >> 172.21.15.26:0/577120426 conn(0x7f44940b60a0 msgr2=0x7f44940b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- 172.21.15.26:0/577120426 shutdown_connections 2022-07-20T02:33:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 -- 172.21.15.26:0/577120426 wait complete. 2022-07-20T02:33:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.564+0000 7f449bcd4700 1 Processor -- start 2022-07-20T02:33:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- start start 2022-07-20T02:33:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4494061050 0x7f44940b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 0x7f44941458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494065870 con 0x7f4494067190 2022-07-20T02:33:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44941462c0 con 0x7f44940b2920 2022-07-20T02:33:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4494065330 con 0x7f4494061050 2022-07-20T02:33:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449926f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 0x7f44941458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f4499a70700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4494061050 0x7f44940b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:60570/0 (socket says 172.21.15.26:60570) 2022-07-20T02:33:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 -- 172.21.15.26:0/4127150453 learned_addr learned my addr 172.21.15.26:0/4127150453 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:04.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449926f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 0x7f44941458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40876/0 (socket says 172.21.15.26:40876) 2022-07-20T02:33:04.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 -- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4494061050 msgr2=0x7f44940b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4494061050 0x7f44940b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 -- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 msgr2=0x7f44941458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 0x7f44941458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 -- 172.21.15.26:0/4127150453 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484009470 con 0x7f44940b2920 2022-07-20T02:33:04.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449a271700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f448c00c000 tx=0x7f448c006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448c012600 con 0x7f44940b2920 2022-07-20T02:33:04.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c012760 con 0x7f44940b2920 2022-07-20T02:33:04.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44940655d0 con 0x7f44940b2920 2022-07-20T02:33:04.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448c012ac0 con 0x7f44940b2920 2022-07-20T02:33:04.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4494066400 con 0x7f44940b2920 2022-07-20T02:33:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.568+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44940665c0 con 0x7f44940b2920 2022-07-20T02:33:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.572+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f448c012c20 con 0x7f44940b2920 2022-07-20T02:33:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.572+0000 7f4492ffd700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 0x7f4474055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.572+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f448c06c1e0 con 0x7f44940b2920 2022-07-20T02:33:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.572+0000 7f4499a70700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 0x7f4474055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.572+0000 7f4499a70700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 0x7f4474055ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f448401b000 tx=0x7f4484016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.580+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f448c042f40 con 0x7f44940b2920 2022-07-20T02:33:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.744+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44940649b0 con 0x7f44940b2920 2022-07-20T02:33:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.744+0000 7f4492ffd700 1 -- 172.21.15.26:0/4127150453 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f448c016090 con 0x7f44940b2920 2022-07-20T02:33:04.765 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:04.765 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 msgr2=0x7f4474055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 0x7f4474055ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f448401b000 tx=0x7f4484016000).stop 2022-07-20T02:33:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 msgr2=0x7f4494145dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f448c00c000 tx=0x7f448c006eb0).stop 2022-07-20T02:33:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 shutdown_connections 2022-07-20T02:33:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4494067190 0x7f44941458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4474053b30 0x7f4474055ff0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44940b2920 0x7f4494145dd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 --2- 172.21.15.26:0/4127150453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4494061050 0x7f44940b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 >> 172.21.15.26:0/4127150453 conn(0x7f44940b60a0 msgr2=0x7f4494147cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 shutdown_connections 2022-07-20T02:33:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:04.748+0000 7f449bcd4700 1 -- 172.21.15.26:0/4127150453 wait complete. 2022-07-20T02:33:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:05.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:59.904474+0000', 'modified': '2022-07-20T02:33:00.907175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7053}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7053': {'gid': 7053, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2339237946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2339237946}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2339237946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:05.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:05.088 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:05.207 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7053,"want_state":"up:active","state":"up:active","rank_uptime":5.2903907549999998,"mdsmap_epoch":48,"osdmap_epoch":98,"osdmap_epoch_barrier":97,"uptime":10.418249811000001} 2022-07-20T02:33:05.209 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7053, "mdsmap_epoch": 48, "osdmap_epoch": 98, "osdmap_epoch_barrier": 97, "rank_uptime": 5.290390755, "state": "up:active", "uptime": 10.418249811, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:05.216 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:33:05.217 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:33:05.217 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:33:05.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.219 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:05.235 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:05.266 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:05.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:05.324 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:05.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.328 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:05.340 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:33:05.341 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:05.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:05.407 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:05.408 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:05.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:33:05.419+0000 7fe715b9d080 -1 init, newargv = 0x55aefd19e750 newargc=9ceph-fuse[ 2022-07-20T02:33:05.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:23068]: starting ceph client 2022-07-20T02:33:05.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23068]: starting fuse 2022-07-20T02:33:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:05.439 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:05.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.440 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:05.495 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:33:05.495 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:33:05.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.499 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:33:05.500 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:33:05.501 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:33:05.502 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:33:05.503 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:33:05.504 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:05.504 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:33:05.505 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:33:05.506 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:33:05.507 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:05.508 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:33:05.508 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:33:05.509 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:33:05.512 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:33:05.513 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:05.514 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:33:05.515 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:33:05.516 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:33:05.517 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:33:05.518 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:33:05.520 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:05.521 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:33:05.521 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:33:05.571 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23068.asok 2022-07-20T02:33:05.580 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23068.asok 2022-07-20T02:33:05.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23068.asok status 2022-07-20T02:33:05.725 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:33:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:33:05.725 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:33:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:33:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:33:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:33:05.726 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:33:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23068", 2022-07-20T02:33:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:33:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:05.727 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:33:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:33:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 6275, 2022-07-20T02:33:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:33:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:33:05.728 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:33:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 6275 2022-07-20T02:33:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:05.729 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:05.732 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:05.732 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 1850400162 2022-07-20T02:33:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:33:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:05.733 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:05.734 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:05.734 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:05.734 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 1850400162 2022-07-20T02:33:05.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:05.737 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.6275 172.21.15.159:0/1850400162", 2022-07-20T02:33:05.738 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/1850400162", 2022-07-20T02:33:05.738 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:33:05.738 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 48, 2022-07-20T02:33:05.738 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 98, 2022-07-20T02:33:05.738 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:33:05.739 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:33:05.739 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:33:05.739 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:33:05.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.741 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:05.749 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:05.761 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:33:05.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:05.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:05.767 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:05.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 Processor -- start 2022-07-20T02:33:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- start start 2022-07-20T02:33:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff4a4061050 0x7ff4a4067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a40b1bc0 con 0x7ff4a41484a0 2022-07-20T02:33:05.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a4147700 con 0x7ff4a40b2920 2022-07-20T02:33:05.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a414b040 con 0x7ff4a4061050 2022-07-20T02:33:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff49b7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff4a41484a0 0x7ff4a4066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40162/0 (socket says 172.21.15.26:40162) 2022-07-20T02:33:05.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff49b7fe700 1 -- 172.21.15.26:0/856885065 learned_addr learned my addr 172.21.15.26:0/856885065 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:05.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a8928700 1 --2- 172.21.15.26:0/856885065 >> v2:172.21.15.120:3300/0 conn(0x7ff4a4061050 0x7ff4a4067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:05.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519712134 0 0) 0x7ff4a40b1bc0 con 0x7ff4a41484a0 2022-07-20T02:33:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff484003260 con 0x7ff4a41484a0 2022-07-20T02:33:05.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004450265 0 0) 0x7ff4a4147700 con 0x7ff4a40b2920 2022-07-20T02:33:05.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a40b1bc0 con 0x7ff4a40b2920 2022-07-20T02:33:05.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a8928700 1 -- 172.21.15.26:0/856885065 >> v1:172.21.15.120:6789/0 conn(0x7ff4a40b2920 legacy=0x7ff4a4147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:05.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858821018 0 0) 0x7ff484003260 con 0x7ff4a41484a0 2022-07-20T02:33:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a8928700 1 -- 172.21.15.26:0/856885065 >> v1:172.21.15.26:6789/0 conn(0x7ff4a41484a0 legacy=0x7ff4a4066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a8928700 1 -- 172.21.15.26:0/856885065 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a4147930 con 0x7ff4a4061050 2022-07-20T02:33:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a8928700 1 --2- 172.21.15.26:0/856885065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a4061050 0x7ff4a4067190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff48c001110 tx=0x7ff49c0091d0).ready entity=mon.1 client_cookie=e6ca8bb75c0ec6c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48c001660 con 0x7ff4a4061050 2022-07-20T02:33:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00f510 con 0x7ff4a4061050 2022-07-20T02:33:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4a37fe700 1 -- 172.21.15.26:0/856885065 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff49c00f680 con 0x7ff4a4061050 2022-07-20T02:33:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- 172.21.15.26:0/856885065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a4061050 msgr2=0x7ff4a4067190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/856885065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a4061050 0x7ff4a4067190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff48c001110 tx=0x7ff49c0091d0).stop 2022-07-20T02:33:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- 172.21.15.26:0/856885065 shutdown_connections 2022-07-20T02:33:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/856885065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a4061050 0x7ff4a4067190 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- 172.21.15.26:0/856885065 >> 172.21.15.26:0/856885065 conn(0x7ff4a40b60a0 msgr2=0x7ff4a40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- 172.21.15.26:0/856885065 shutdown_connections 2022-07-20T02:33:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.940+0000 7ff4aa38b700 1 -- 172.21.15.26:0/856885065 wait complete. 2022-07-20T02:33:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 Processor -- start 2022-07-20T02:33:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 -- start start 2022-07-20T02:33:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4a4061050 0x7ff4a414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a41484a0 0x7ff4a40ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40645e0 con 0x7ff4a40b2920 2022-07-20T02:33:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40665b0 con 0x7ff4a41484a0 2022-07-20T02:33:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4aa38b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4064810 con 0x7ff4a4061050 2022-07-20T02:33:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40888/0 (socket says 172.21.15.26:40888) 2022-07-20T02:33:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 -- 172.21.15.26:0/3866689916 learned_addr learned my addr 172.21.15.26:0/3866689916 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4a8928700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a41484a0 0x7ff4a40ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4a3fff700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4a4061050 0x7ff4a414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 -- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4a4061050 msgr2=0x7ff4a414e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4a4061050 0x7ff4a414e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 -- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a41484a0 msgr2=0x7ff4a40ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a41484a0 0x7ff4a40ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 -- 172.21.15.26:0/3866689916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009470 con 0x7ff4a40b2920 2022-07-20T02:33:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff49b7fe700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7ff48c000fe0 tx=0x7ff48c001a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48c0072e0 con 0x7ff4a40b2920 2022-07-20T02:33:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c007440 con 0x7ff4a40b2920 2022-07-20T02:33:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.944+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48c005800 con 0x7ff4a40b2920 2022-07-20T02:33:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a4066310 con 0x7ff4a40b2920 2022-07-20T02:33:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a40641e0 con 0x7ff4a40b2920 2022-07-20T02:33:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a4066130 con 0x7ff4a40b2920 2022-07-20T02:33:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff48c010070 con 0x7ff4a40b2920 2022-07-20T02:33:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4a1ffb700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 0x7ff488055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff48c06ba70 con 0x7ff4a40b2920 2022-07-20T02:33:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.948+0000 7ff4a3fff700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 0x7ff488055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.952+0000 7ff4a3fff700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 0x7ff488055fc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff49400b000 tx=0x7ff494009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:05.956+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff48c0426a0 con 0x7ff4a40b2920 2022-07-20T02:33:06.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.084+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4a414ada0 con 0x7ff4a40b2920 2022-07-20T02:33:06.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.088+0000 7ff4a1ffb700 1 -- 172.21.15.26:0/3866689916 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16322 (secure 0 0 0) 0x7ff48c018030 con 0x7ff4a40b2920 2022-07-20T02:33:06.098 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:06.099 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":98,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:00.907866+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:32:57.859395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T02:32:58.906424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:06.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 msgr2=0x7ff488055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 0x7ff488055fc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff49400b000 tx=0x7ff494009040).stop 2022-07-20T02:33:06.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 msgr2=0x7ff4a40aafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7ff48c000fe0 tx=0x7ff48c001a10).stop 2022-07-20T02:33:06.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 shutdown_connections 2022-07-20T02:33:06.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff4a40b2920 0x7ff4a40aafc0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff488053b00 0x7ff488055fc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4a41484a0 0x7ff4a40ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 --2- 172.21.15.26:0/3866689916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff4a4061050 0x7ff4a414e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 >> 172.21.15.26:0/3866689916 conn(0x7ff4a40b60a0 msgr2=0x7ff4a41469c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 shutdown_connections 2022-07-20T02:33:06.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.092+0000 7ff4aa38b700 1 -- 172.21.15.26:0/3866689916 wait complete. 2022-07-20T02:33:06.117 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:32:57.859395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:32:58.906424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:06.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:06.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 Processor -- start 2022-07-20T02:33:06.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 -- start start 2022-07-20T02:33:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c64143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c641474e0 con 0x7f4c64060f00 2022-07-20T02:33:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640b4e40 con 0x7f4c64147000 2022-07-20T02:33:06.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c69639700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6405d3b0 con 0x7f4c6405cfd0 2022-07-20T02:33:06.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c68e38700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c64060f00 0x7f4c6413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40178/0 (socket says 172.21.15.26:40178) 2022-07-20T02:33:06.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c68e38700 1 -- 172.21.15.26:0/1120930033 learned_addr learned my addr 172.21.15.26:0/1120930033 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:06.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c64143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290453487 0 0) 0x7f4c641474e0 con 0x7f4c64060f00 2022-07-20T02:33:06.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50003260 con 0x7f4c64060f00 2022-07-20T02:33:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960707600 0 0) 0x7f4c640b4e40 con 0x7f4c64147000 2022-07-20T02:33:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.240+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c641474e0 con 0x7f4c64147000 2022-07-20T02:33:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169952703 0 0) 0x7f4c50003260 con 0x7f4c64060f00 2022-07-20T02:33:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 msgr2=0x7f4c64143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c64143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 >> v1:172.21.15.120:6789/0 conn(0x7f4c64147000 legacy=0x7f4c640b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c641431e0 con 0x7f4c64060f00 2022-07-20T02:33:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f4c5c0019e0 con 0x7f4c64060f00 2022-07-20T02:33:06.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c64143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:06.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 >> v1:172.21.15.26:6789/0 conn(0x7f4c64060f00 legacy=0x7f4c6413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c50003910 0x7f4c50003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c50004500 0x7f4c50004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c50004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6405d3b0 con 0x7f4c50003910 2022-07-20T02:33:06.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5000b750 con 0x7f4c50004500 2022-07-20T02:33:06.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5000b910 con 0x7f4c6405cfd0 2022-07-20T02:33:06.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c50003910 0x7f4c50003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c637fe700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c50004500 0x7f4c50004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c68e38700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c50004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 -- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 msgr2=0x7f4c50004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c6405cfd0 0x7f4c50004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 -- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c50004500 msgr2=0x7f4c50004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c50004500 0x7f4c50004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 -- 172.21.15.26:0/1120930033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5800b180 con 0x7f4c50003910 2022-07-20T02:33:06.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c63fff700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c50003910 0x7f4c50003d10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f4c5800c000 tx=0x7f4c58006df0).ready entity=mon.0 client_cookie=2d4295ffa801cc14 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c5800e830 con 0x7f4c50003910 2022-07-20T02:33:06.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5800e990 con 0x7f4c50003910 2022-07-20T02:33:06.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c62ffd700 1 -- 172.21.15.26:0/1120930033 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c5800ecf0 con 0x7f4c50003910 2022-07-20T02:33:06.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- 172.21.15.26:0/1120930033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c50003910 msgr2=0x7f4c50003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 --2- 172.21.15.26:0/1120930033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c50003910 0x7f4c50003d10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f4c5800c000 tx=0x7f4c58006df0).stop 2022-07-20T02:33:06.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c68e38700 1 -- 172.21.15.26:0/1120930033 reap_dead start 2022-07-20T02:33:06.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- 172.21.15.26:0/1120930033 shutdown_connections 2022-07-20T02:33:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- 172.21.15.26:0/1120930033 >> 172.21.15.26:0/1120930033 conn(0x7f4c640620c0 msgr2=0x7f4c6405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- 172.21.15.26:0/1120930033 shutdown_connections 2022-07-20T02:33:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- 172.21.15.26:0/1120930033 wait complete. 2022-07-20T02:33:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 Processor -- start 2022-07-20T02:33:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.244+0000 7f4c69639700 1 -- start start 2022-07-20T02:33:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c64137220 0x7f4c641535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 0x7f4c641564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64138670 con 0x7f4c640b7460 2022-07-20T02:33:06.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64137600 con 0x7f4c640b78c0 2022-07-20T02:33:06.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64152c00 con 0x7f4c64137220 2022-07-20T02:33:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40908/0 (socket says 172.21.15.26:40908) 2022-07-20T02:33:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 -- 172.21.15.26:0/2645363953 learned_addr learned my addr 172.21.15.26:0/2645363953 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:06.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c63fff700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c64137220 0x7f4c641535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c68e38700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 0x7f4c641564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 -- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c64137220 msgr2=0x7f4c641535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c64137220 0x7f4c641535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 -- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 msgr2=0x7f4c641564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 0x7f4c641564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 -- 172.21.15.26:0/2645363953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64152ea0 con 0x7f4c640b7460 2022-07-20T02:33:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c68e38700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 0x7f4c641564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:06.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c637fe700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f4c64137790 tx=0x7f4c54007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c5400fac0 con 0x7f4c640b7460 2022-07-20T02:33:06.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5400fc20 con 0x7f4c640b7460 2022-07-20T02:33:06.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c64153140 con 0x7f4c640b7460 2022-07-20T02:33:06.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c54014450 con 0x7f4c640b7460 2022-07-20T02:33:06.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c641382f0 con 0x7f4c640b7460 2022-07-20T02:33:06.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4c5401b030 con 0x7f4c640b7460 2022-07-20T02:33:06.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c641379f0 con 0x7f4c640b7460 2022-07-20T02:33:06.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 0x7f4c44055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4c5406c1b0 con 0x7f4c640b7460 2022-07-20T02:33:06.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.248+0000 7f4c63fff700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 0x7f4c44055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.252+0000 7f4c63fff700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 0x7f4c44055f60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f4c5801e000 tx=0x7f4c58019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.256+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c54046e60 con 0x7f4c640b7460 2022-07-20T02:33:06.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.384+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4c641532d0 con 0x7f4c640b7460 2022-07-20T02:33:06.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.384+0000 7f4c617fa700 1 -- 172.21.15.26:0/2645363953 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v98) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4c54042f70 con 0x7f4c640b7460 2022-07-20T02:33:06.397 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:06.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 msgr2=0x7f4c44055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 0x7f4c44055f60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f4c5801e000 tx=0x7f4c58019000).stop 2022-07-20T02:33:06.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 msgr2=0x7f4c64155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f4c64137790 tx=0x7f4c54007a00).stop 2022-07-20T02:33:06.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 shutdown_connections 2022-07-20T02:33:06.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c640b7460 0x7f4c64155fd0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c44053aa0 0x7f4c44055f60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c640b78c0 0x7f4c641564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 --2- 172.21.15.26:0/2645363953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c64137220 0x7f4c641535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 >> 172.21.15.26:0/2645363953 conn(0x7f4c6405c4e0 msgr2=0x7f4c640b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 shutdown_connections 2022-07-20T02:33:06.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.388+0000 7f4c69639700 1 -- 172.21.15.26:0/2645363953 wait complete. 2022-07-20T02:33:06.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:06.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.516+0000 7fcdddb5d700 1 Processor -- start 2022-07-20T02:33:06.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.516+0000 7fcdddb5d700 1 -- start start 2022-07-20T02:33:06.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80b2920 0x7fcdd80a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcdd80a8f60 0x7fcdd80a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 0x7fcdd8143b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80aad20 con 0x7fcdd80a8f60 2022-07-20T02:33:06.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80aaa40 con 0x7fcdd8143700 2022-07-20T02:33:06.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b1b30 con 0x7fcdd80b2920 2022-07-20T02:33:06.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcdd80a8f60 0x7fcdd80a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcdd80a8f60 0x7fcdd80a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40918/0 (socket says 172.21.15.26:40918) 2022-07-20T02:33:06.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 -- 172.21.15.26:0/825332066 learned_addr learned my addr 172.21.15.26:0/825332066 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd7fff700 1 --2- 172.21.15.26:0/825332066 >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 0x7fcdd8143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd77fe700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80b2920 0x7fcdd80a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 -- 172.21.15.26:0/825332066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80b2920 msgr2=0x7fcdd80a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80b2920 0x7fcdd80a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 -- 172.21.15.26:0/825332066 >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 msgr2=0x7fcdd8143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 --2- 172.21.15.26:0/825332066 >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 0x7fcdd8143b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 -- 172.21.15.26:0/825332066 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80b1dd0 con 0x7fcdd80a8f60 2022-07-20T02:33:06.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd7fff700 1 --2- 172.21.15.26:0/825332066 >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 0x7fcdd8143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:06.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdcffff700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80a8f60 0x7fcdd80a99c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fcdc0002f70 tx=0x7fcdc0006eb0).ready entity=mon.0 client_cookie=e15a318ccb70f167 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd6ffd700 1 -- 172.21.15.26:0/825332066 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc000d570 con 0x7fcdd80a8f60 2022-07-20T02:33:06.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd6ffd700 1 -- 172.21.15.26:0/825332066 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc0000e90 con 0x7fcdd80a8f60 2022-07-20T02:33:06.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdd6ffd700 1 -- 172.21.15.26:0/825332066 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc000d900 con 0x7fcdd80a8f60 2022-07-20T02:33:06.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- 172.21.15.26:0/825332066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80a8f60 msgr2=0x7fcdd80a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80a8f60 0x7fcdd80a99c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fcdc0002f70 tx=0x7fcdc0006eb0).stop 2022-07-20T02:33:06.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- 172.21.15.26:0/825332066 shutdown_connections 2022-07-20T02:33:06.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/825332066 >> v2:172.21.15.120:3300/0 conn(0x7fcdd8143700 0x7fcdd8143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80a8f60 0x7fcdd80a99c0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/825332066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80b2920 0x7fcdd80a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- 172.21.15.26:0/825332066 >> 172.21.15.26:0/825332066 conn(0x7fcdd80b60a0 msgr2=0x7fcdd80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- 172.21.15.26:0/825332066 shutdown_connections 2022-07-20T02:33:06.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.520+0000 7fcdddb5d700 1 -- 172.21.15.26:0/825332066 wait complete. 2022-07-20T02:33:06.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 Processor -- start 2022-07-20T02:33:06.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- start start 2022-07-20T02:33:06.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80a8f60 0x7fcdd80adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdd8143700 0x7fcdd81561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8139090 con 0x7fcdd80b2920 2022-07-20T02:33:06.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80635a0 con 0x7fcdd8143700 2022-07-20T02:33:06.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8063840 con 0x7fcdd80a8f60 2022-07-20T02:33:06.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40922/0 (socket says 172.21.15.26:40922) 2022-07-20T02:33:06.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 -- 172.21.15.26:0/3967206598 learned_addr learned my addr 172.21.15.26:0/3967206598 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:06.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdd7fff700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdd8143700 0x7fcdd81561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdd77fe700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80a8f60 0x7fcdd80adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 -- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80a8f60 msgr2=0x7fcdd80adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80a8f60 0x7fcdd80adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 -- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdd8143700 msgr2=0x7fcdd81561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdd8143700 0x7fcdd81561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 -- 172.21.15.26:0/3967206598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc000b1e0 con 0x7fcdd80b2920 2022-07-20T02:33:06.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdcffff700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fcdc0002f70 tx=0x7fcdc0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc00117e0 con 0x7fcdd80b2920 2022-07-20T02:33:06.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd8063ae0 con 0x7fcdd80b2920 2022-07-20T02:33:06.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc0011940 con 0x7fcdd80b2920 2022-07-20T02:33:06.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc0011ca0 con 0x7fcdd80b2920 2022-07-20T02:33:06.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd8152350 con 0x7fcdd80b2920 2022-07-20T02:33:06.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.524+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd8060a60 con 0x7fcdd80b2920 2022-07-20T02:33:06.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.528+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcdc001a040 con 0x7fcdd80b2920 2022-07-20T02:33:06.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.528+0000 7fcdd57fa700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 0x7fcdb0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.528+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcdc006cfb0 con 0x7fcdd80b2920 2022-07-20T02:33:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.528+0000 7fcdd77fe700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 0x7fcdb0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.528+0000 7fcdd77fe700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 0x7fcdb0055e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcdc800c000 tx=0x7fcdc8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.536+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdc0043cd0 con 0x7fcdd80b2920 2022-07-20T02:33:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.660+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcdd80574e0 con 0x7fcdd80b2920 2022-07-20T02:33:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdd57fa700 1 -- 172.21.15.26:0/3967206598 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcdc0023020 con 0x7fcdd80b2920 2022-07-20T02:33:06.671 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 msgr2=0x7fcdb0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 0x7fcdb0055e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcdc800c000 tx=0x7fcdc8008040).stop 2022-07-20T02:33:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 msgr2=0x7fcdd8142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fcdc0002f70 tx=0x7fcdc0015000).stop 2022-07-20T02:33:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 shutdown_connections 2022-07-20T02:33:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdd80b2920 0x7fcdd8142de0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcdb0053950 0x7fcdb0055e10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdd8143700 0x7fcdd81561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 --2- 172.21.15.26:0/3967206598 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcdd80a8f60 0x7fcdd80adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 >> 172.21.15.26:0/3967206598 conn(0x7fcdd80b60a0 msgr2=0x7fcdd80b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 shutdown_connections 2022-07-20T02:33:06.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.664+0000 7fcdddb5d700 1 -- 172.21.15.26:0/3967206598 wait complete. 2022-07-20T02:33:06.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:06.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.800+0000 7f9dd2afc700 1 Processor -- start 2022-07-20T02:33:06.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.800+0000 7f9dd2afc700 1 -- start start 2022-07-20T02:33:06.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0b62d0 con 0x7f9dcc0a8c40 2022-07-20T02:33:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dcc147eb0 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc14aac0 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40932/0 (socket says 172.21.15.26:40932) 2022-07-20T02:33:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd12f9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 -- 172.21.15.26:0/1411199972 learned_addr learned my addr 172.21.15.26:0/1411199972 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd0af8700 1 -- 172.21.15.26:0/1411199972 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201676182 0 0) 0x7f9dcc0b62d0 con 0x7f9dcc0a8c40 2022-07-20T02:33:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 -- 172.21.15.26:0/1411199972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 msgr2=0x7f9dcc1479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 --2- 172.21.15.26:0/1411199972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc1479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 -- 172.21.15.26:0/1411199972 >> v1:172.21.15.120:6789/0 conn(0x7f9dcc0a8c40 legacy=0x7f9dcc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 -- 172.21.15.26:0/1411199972 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc14ad60 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd1afa700 1 --2- 172.21.15.26:0/1411199972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f9dc0002f70 tx=0x7f9dc0006eb0).ready entity=mon.0 client_cookie=67a068b389f3d0ed server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd0af8700 1 -- 172.21.15.26:0/1411199972 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000d570 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd0af8700 1 -- 172.21.15.26:0/1411199972 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0000e90 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd0af8700 1 -- 172.21.15.26:0/1411199972 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc000d8c0 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- 172.21.15.26:0/1411199972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 msgr2=0x7f9dcc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/1411199972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f9dc0002f70 tx=0x7f9dc0006eb0).stop 2022-07-20T02:33:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- 172.21.15.26:0/1411199972 shutdown_connections 2022-07-20T02:33:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/1411199972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc1479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/1411199972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc1474d0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- 172.21.15.26:0/1411199972 >> 172.21.15.26:0/1411199972 conn(0x7f9dcc060310 msgr2=0x7f9dcc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- 172.21.15.26:0/1411199972 shutdown_connections 2022-07-20T02:33:06.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 -- 172.21.15.26:0/1411199972 wait complete. 2022-07-20T02:33:06.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.804+0000 7f9dd2afc700 1 Processor -- start 2022-07-20T02:33:06.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- start start 2022-07-20T02:33:06.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dcc0a8c40 0x7f9dcc0abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc0b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc0aa860 con 0x7f9dcc0b5ef0 2022-07-20T02:33:06.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc0b4030 con 0x7f9dcc0a8c40 2022-07-20T02:33:06.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc0aa660 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd22fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc0b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34220/0 (socket says 172.21.15.26:34220) 2022-07-20T02:33:06.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd12f9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dcc0a8c40 0x7f9dcc0abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 -- 172.21.15.26:0/28929236 learned_addr learned my addr 172.21.15.26:0/28929236 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:06.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 -- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dcc0a8c40 msgr2=0x7f9dcc0abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dcc0a8c40 0x7f9dcc0abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 -- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 msgr2=0x7f9dcc0b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc0b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 -- 172.21.15.26:0/28929236 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc000b1e0 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd1afa700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9dc0002f70 tx=0x7f9dc0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc0011760 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc00118c0 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc0aaa90 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc0011c20 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc0b4950 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.808+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc067760 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.812+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9dc0025030 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.812+0000 7f9dbeffd700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 0x7f9dac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.812+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9dc006d120 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.812+0000 7f9dd12f9700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 0x7f9dac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.812+0000 7f9dd12f9700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 0x7f9dac055fc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f9dc800b000 tx=0x7f9dc8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:06.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.824+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9dcc067760 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9dcc067760 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dbeffd700 1 -- 172.21.15.26:0/28929236 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9dcc067760 con 0x7f9dcc0a7fc0 2022-07-20T02:33:06.953 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 msgr2=0x7f9dac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 0x7f9dac055fc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f9dc800b000 tx=0x7f9dc8006df0).stop 2022-07-20T02:33:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 msgr2=0x7f9dcc0ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9dc0002f70 tx=0x7f9dc0015000).stop 2022-07-20T02:33:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 shutdown_connections 2022-07-20T02:33:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9dcc0b5ef0 0x7f9dcc0b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9dac053b00 0x7f9dac055fc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dcc0a8c40 0x7f9dcc0abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 --2- 172.21.15.26:0/28929236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dcc0a7fc0 0x7f9dcc0ab820 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.948+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 >> 172.21.15.26:0/28929236 conn(0x7f9dcc060310 msgr2=0x7f9dcc05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.952+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 shutdown_connections 2022-07-20T02:33:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:06.952+0000 7f9dd2afc700 1 -- 172.21.15.26:0/28929236 wait complete. 2022-07-20T02:33:06.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:07.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 Processor -- start 2022-07-20T02:33:07.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- start start 2022-07-20T02:33:07.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7dc40b2920 0x7f7dc40a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7dc40a8f60 0x7f7dc40a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc4143700 0x7f7dc4143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40aad20 con 0x7f7dc40a8f60 2022-07-20T02:33:07.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40aaa40 con 0x7f7dc40b2920 2022-07-20T02:33:07.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40b1b30 con 0x7f7dc4143700 2022-07-20T02:33:07.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7dc40a8f60 0x7f7dc40a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7dc40a8f60 0x7f7dc40a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40944/0 (socket says 172.21.15.26:40944) 2022-07-20T02:33:07.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 -- 172.21.15.26:0/1455366551 learned_addr learned my addr 172.21.15.26:0/1455366551 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:07.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc37fe700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc4143700 0x7f7dc4143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc2ffd700 1 --2- 172.21.15.26:0/1455366551 >> v2:172.21.15.120:3300/0 conn(0x7f7dc40b2920 0x7f7dc40a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 -- 172.21.15.26:0/1455366551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc4143700 msgr2=0x7f7dc4143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc4143700 0x7f7dc4143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 -- 172.21.15.26:0/1455366551 >> v2:172.21.15.120:3300/0 conn(0x7f7dc40b2920 msgr2=0x7f7dc40a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1455366551 >> v2:172.21.15.120:3300/0 conn(0x7f7dc40b2920 0x7f7dc40a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 -- 172.21.15.26:0/1455366551 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40b1dd0 con 0x7f7dc40a8f60 2022-07-20T02:33:07.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40a99c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f7db40032f0 tx=0x7f7db4009290).ready entity=mon.0 client_cookie=a9d744aac05fd430 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc1ffb700 1 -- 172.21.15.26:0/1455366551 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db400b680 con 0x7f7dc40a8f60 2022-07-20T02:33:07.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc1ffb700 1 -- 172.21.15.26:0/1455366551 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400b7e0 con 0x7f7dc40a8f60 2022-07-20T02:33:07.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc1ffb700 1 -- 172.21.15.26:0/1455366551 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db400b950 con 0x7f7dc40a8f60 2022-07-20T02:33:07.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1455366551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 msgr2=0x7f7dc40a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40a99c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f7db40032f0 tx=0x7f7db4009290).stop 2022-07-20T02:33:07.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1455366551 shutdown_connections 2022-07-20T02:33:07.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc4143700 0x7f7dc4143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1455366551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40a99c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1455366551 >> v2:172.21.15.120:3300/0 conn(0x7f7dc40b2920 0x7f7dc40a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1455366551 >> 172.21.15.26:0/1455366551 conn(0x7f7dc40b60a0 msgr2=0x7f7dc40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:07.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1455366551 shutdown_connections 2022-07-20T02:33:07.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.084+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1455366551 wait complete. 2022-07-20T02:33:07.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 Processor -- start 2022-07-20T02:33:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- start start 2022-07-20T02:33:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40b2920 0x7f7dc4150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 0x7f7dc4150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40aec80 con 0x7f7dc40a8f60 2022-07-20T02:33:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4140180 con 0x7f7dc4143700 2022-07-20T02:33:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40ae5e0 con 0x7f7dc40b2920 2022-07-20T02:33:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40950/0 (socket says 172.21.15.26:40950) 2022-07-20T02:33:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 -- 172.21.15.26:0/1010765815 learned_addr learned my addr 172.21.15.26:0/1010765815 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40b2920 0x7f7dc4150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc37fe700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 0x7f7dc4150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 -- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40b2920 msgr2=0x7f7dc4150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40b2920 0x7f7dc4150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 -- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 msgr2=0x7f7dc4150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 0x7f7dc4150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 -- 172.21.15.26:0/1010765815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009530 con 0x7f7dc40a8f60 2022-07-20T02:33:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc37fe700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 0x7f7dc4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc2ffd700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f7dac00c000 tx=0x7f7dac006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac010780 con 0x7f7dc40a8f60 2022-07-20T02:33:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac0108e0 con 0x7f7dc40a8f60 2022-07-20T02:33:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc40acc90 con 0x7f7dc40a8f60 2022-07-20T02:33:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac010c40 con 0x7f7dc40a8f60 2022-07-20T02:33:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.088+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40adb70 con 0x7f7dc40a8f60 2022-07-20T02:33:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4060a60 con 0x7f7dc40a8f60 2022-07-20T02:33:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7dac01c460 con 0x7f7dc40a8f60 2022-07-20T02:33:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dbbfff700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 0x7f7da8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7dac06c560 con 0x7f7dc40a8f60 2022-07-20T02:33:07.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 0x7f7da8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.092+0000 7f7dc27fc700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 0x7f7da8055e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f7db401a000 tx=0x7f7db4017000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.100+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7dac043220 con 0x7f7dc40a8f60 2022-07-20T02:33:07.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7dc40574e0 con 0x7f7dc40a8f60 2022-07-20T02:33:07.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dbbfff700 1 -- 172.21.15.26:0/1010765815 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7dac046840 con 0x7f7dc40a8f60 2022-07-20T02:33:07.239 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:07.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 msgr2=0x7f7da8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 0x7f7da8055e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f7db401a000 tx=0x7f7db4017000).stop 2022-07-20T02:33:07.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 msgr2=0x7f7dc40ae0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f7dac00c000 tx=0x7f7dac006df0).stop 2022-07-20T02:33:07.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 shutdown_connections 2022-07-20T02:33:07.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7da8053950 0x7f7da8055e10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc4143700 0x7f7dc4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7dc40b2920 0x7f7dc4150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 --2- 172.21.15.26:0/1010765815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7dc40a8f60 0x7f7dc40ae0f0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 >> 172.21.15.26:0/1010765815 conn(0x7f7dc40b60a0 msgr2=0x7f7dc40b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:07.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 shutdown_connections 2022-07-20T02:33:07.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.228+0000 7f7dc96c7700 1 -- 172.21.15.26:0/1010765815 wait complete. 2022-07-20T02:33:07.248 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:07.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.348+0000 7f2d72621700 1 Processor -- start 2022-07-20T02:33:07.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.348+0000 7f2d72621700 1 -- start start 2022-07-20T02:33:07.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 0x7f2d6c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d6c061050 0x7f2d6c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6c0b1bc0 con 0x7f2d6c066a80 2022-07-20T02:33:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6c14b040 con 0x7f2d6c061050 2022-07-20T02:33:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0a8860 con 0x7f2d6c0b2920 2022-07-20T02:33:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d70bbe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2d6c066a80 0x7f2d6c066ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40242/0 (socket says 172.21.15.26:40242) 2022-07-20T02:33:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d70bbe700 1 -- 172.21.15.26:0/2646663937 learned_addr learned my addr 172.21.15.26:0/2646663937 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/2646663937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 0x7f2d6c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6b7fe700 1 --2- 172.21.15.26:0/2646663937 >> v2:172.21.15.120:3300/0 conn(0x7f2d6c061050 0x7f2d6c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6affd700 1 -- 172.21.15.26:0/2646663937 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164275692 0 0) 0x7f2d6c0b1bc0 con 0x7f2d6c066a80 2022-07-20T02:33:07.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6affd700 1 -- 172.21.15.26:0/2646663937 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d58003260 con 0x7f2d6c066a80 2022-07-20T02:33:07.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 -- 172.21.15.26:0/2646663937 >> v2:172.21.15.120:3300/0 conn(0x7f2d6c061050 msgr2=0x7f2d6c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:07.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/2646663937 >> v2:172.21.15.120:3300/0 conn(0x7f2d6c061050 0x7f2d6c148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 -- 172.21.15.26:0/2646663937 >> v1:172.21.15.26:6789/0 conn(0x7f2d6c066a80 legacy=0x7f2d6c066ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 -- 172.21.15.26:0/2646663937 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c147700 con 0x7f2d6c0b2920 2022-07-20T02:33:07.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/2646663937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 0x7f2d6c147d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2d60002f70 tx=0x7f2d60006df0).ready entity=mon.2 client_cookie=41d5c41deb271327 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6affd700 1 -- 172.21.15.26:0/2646663937 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d6000d4a0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6affd700 1 -- 172.21.15.26:0/2646663937 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60000e90 con 0x7f2d6c0b2920 2022-07-20T02:33:07.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d6affd700 1 -- 172.21.15.26:0/2646663937 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d6000d7f0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- 172.21.15.26:0/2646663937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 msgr2=0x7f2d6c147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 --2- 172.21.15.26:0/2646663937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 0x7f2d6c147d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2d60002f70 tx=0x7f2d60006df0).stop 2022-07-20T02:33:07.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- 172.21.15.26:0/2646663937 shutdown_connections 2022-07-20T02:33:07.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 --2- 172.21.15.26:0/2646663937 >> v2:172.21.15.120:3300/0 conn(0x7f2d6c061050 0x7f2d6c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 --2- 172.21.15.26:0/2646663937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c0b2920 0x7f2d6c147d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- 172.21.15.26:0/2646663937 >> 172.21.15.26:0/2646663937 conn(0x7f2d6c0b60a0 msgr2=0x7f2d6c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:07.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- 172.21.15.26:0/2646663937 shutdown_connections 2022-07-20T02:33:07.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.352+0000 7f2d72621700 1 -- 172.21.15.26:0/2646663937 wait complete. 2022-07-20T02:33:07.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 Processor -- start 2022-07-20T02:33:07.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 -- start start 2022-07-20T02:33:07.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d6c061050 0x7f2d6c1460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c066a80 0x7f2d6c14dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c065d90 con 0x7f2d6c0b2920 2022-07-20T02:33:07.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c066030 con 0x7f2d6c061050 2022-07-20T02:33:07.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.356+0000 7f2d72621700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0662d0 con 0x7f2d6c066a80 2022-07-20T02:33:07.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40968/0 (socket says 172.21.15.26:40968) 2022-07-20T02:33:07.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 -- 172.21.15.26:0/458414466 learned_addr learned my addr 172.21.15.26:0/458414466 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:07.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d6b7fe700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c066a80 0x7f2d6c14dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d6c061050 0x7f2d6c1460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 -- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c066a80 msgr2=0x7f2d6c14dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c066a80 0x7f2d6c14dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 -- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d6c061050 msgr2=0x7f2d6c1460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d6c061050 0x7f2d6c1460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 -- 172.21.15.26:0/458414466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6000b180 con 0x7f2d6c0b2920 2022-07-20T02:33:07.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d70bbe700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f2d64001490 tx=0x7f2d64007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d64005ba0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64005d00 con 0x7f2d6c0b2920 2022-07-20T02:33:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d640072c0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c066770 con 0x7f2d6c0b2920 2022-07-20T02:33:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c14e9b0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c066490 con 0x7f2d6c0b2920 2022-07-20T02:33:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.360+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d64010070 con 0x7f2d6c0b2920 2022-07-20T02:33:07.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.364+0000 7f2d697fa700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 0x7f2d4c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:07.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.364+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2d6406bb60 con 0x7f2d6c0b2920 2022-07-20T02:33:07.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.364+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 0x7f2d4c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:07.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.364+0000 7f2d6bfff700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 0x7f2d4c055fc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f2d6001a000 tx=0x7f2d60014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:07.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.372+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d64042880 con 0x7f2d6c0b2920 2022-07-20T02:33:07.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d6c14ada0 con 0x7f2d6c0b2920 2022-07-20T02:33:07.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d697fa700 1 -- 172.21.15.26:0/458414466 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f2d64018030 con 0x7f2d6c0b2920 2022-07-20T02:33:07.540 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:07.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 msgr2=0x7f2d4c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 0x7f2d4c055fc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f2d6001a000 tx=0x7f2d60014000).stop 2022-07-20T02:33:07.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 msgr2=0x7f2d6c159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:07.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f2d64001490 tx=0x7f2d64007e40).stop 2022-07-20T02:33:07.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 shutdown_connections 2022-07-20T02:33:07.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2d6c0b2920 0x7f2d6c159c90 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2d4c053b00 0x7f2d4c055fc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d6c066a80 0x7f2d6c14dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 --2- 172.21.15.26:0/458414466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d6c061050 0x7f2d6c1460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:07.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.532+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 >> 172.21.15.26:0/458414466 conn(0x7f2d6c0b60a0 msgr2=0x7f2d6c149b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:07.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.536+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 shutdown_connections 2022-07-20T02:33:07.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:07.536+0000 7f2d72621700 1 -- 172.21.15.26:0/458414466 wait complete. 2022-07-20T02:33:07.577 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:07.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:32:59.904474+0000', 'modified': '2022-07-20T02:33:00.907175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7053}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7053': {'gid': 7053, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/2339237946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 2339237946}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 2339237946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:07.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:07.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:08.024 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7053,"want_state":"up:active","state":"up:active","rank_uptime":8.1050324020000009,"mdsmap_epoch":48,"osdmap_epoch":98,"osdmap_epoch_barrier":97,"uptime":13.232890119} 2022-07-20T02:33:08.024 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7053, "mdsmap_epoch": 48, "osdmap_epoch": 98, "osdmap_epoch_barrier": 97, "rank_uptime": 8.105032402, "state": "up:active", "uptime": 13.232890119, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:08.025 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:33:08.026 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:33:08.026 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:33:08.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:08.027 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:08.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:33:08.087 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:08.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:08.181 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:08.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:08.183 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:33:08.197 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:33:08.198 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:08.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:33:08.267 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:08.267 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:08.278 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:08.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:08.281 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:33:08.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:33:08.319+0000 7f8912896080 -1 init, newargv = 0x55b2f26f3850 newargc=9ceph-fuse[22696]: starting ceph client 2022-07-20T02:33:08.322 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr: 2022-07-20T02:33:08.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22696]: starting fuse 2022-07-20T02:33:09.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:33:09.432 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:09.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:09.514 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:09.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:09.516 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:33:09.526 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:33:09.526 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:33:09.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:09.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:33:09.529 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:33:09.530 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:33:09.531 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:33:09.531 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:33:09.543 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:09.544 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:33:09.545 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:33:09.546 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:33:09.546 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:09.547 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:33:09.548 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:33:09.549 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:33:09.550 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:33:09.553 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:09.554 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:33:09.554 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:33:09.555 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:33:09.556 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:33:09.557 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:33:09.558 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:09.558 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:33:09.559 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:33:09.615 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22696.asok 2022-07-20T02:33:09.619 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22696.asok 2022-07-20T02:33:09.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:09.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22696.asok status 2022-07-20T02:33:09.735 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:33:09.735 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:33:09.735 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:33:09.735 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:33:09.736 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:33:09.736 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:33:09.736 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:33:09.736 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22696", 2022-07-20T02:33:09.737 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:33:09.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:09.737 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:33:09.737 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:33:09.738 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 7263, 2022-07-20T02:33:09.741 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:33:09.742 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:33:09.742 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:33:09.742 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 7263 2022-07-20T02:33:09.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:09.743 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:33:09.743 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:33:09.743 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:33:09.743 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 2680111673 2022-07-20T02:33:09.745 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:33:09.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:09.745 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:33:09.745 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:33:09.746 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:33:09.746 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 2680111673 2022-07-20T02:33:09.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:09.746 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.7263 172.21.15.176:0/2680111673", 2022-07-20T02:33:09.747 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/2680111673", 2022-07-20T02:33:09.747 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:33:09.747 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 48, 2022-07-20T02:33:09.771 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 98, 2022-07-20T02:33:09.771 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:33:09.771 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:33:09.771 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:33:09.772 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:33:09.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:09.773 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:09.773 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:09.823 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:33:09.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:09.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:09.848 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:09.876 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T02:33:09.877 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p foo 2022-07-20T02:33:09.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:33:09.913 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:33:09.913 DEBUG:teuthology.orchestra.run.smithi159:> import stat 2022-07-20T02:33:09.917 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:09.917 DEBUG:teuthology.orchestra.run.smithi159:> print(os.stat("/home/ubuntu/cephtest/mnt.0/foo").st_ino) 2022-07-20T02:33:09.918 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:33:10.038 INFO:teuthology.orchestra.run.smithi159.stdout:1099511627776 2022-07-20T02:33:10.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:10.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 Processor -- start 2022-07-20T02:33:10.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 -- start start 2022-07-20T02:33:10.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f92100b5ef0 0x7f92101474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100a8c40 0x7f92100af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92100b62d0 con 0x7f92100a7fc0 2022-07-20T02:33:10.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921014aac0 con 0x7f92100b5ef0 2022-07-20T02:33:10.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9216ca5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921014ad60 con 0x7f92100a8c40 2022-07-20T02:33:10.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9215ca3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f92100b5ef0 0x7f92101474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f92154a2700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92100a7fc0 0x7f92100aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40252/0 (socket says 172.21.15.26:40252) 2022-07-20T02:33:10.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f92154a2700 1 -- 172.21.15.26:0/1461237541 learned_addr learned my addr 172.21.15.26:0/1461237541 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:10.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f92164a4700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100a8c40 0x7f92100af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562162178 0 0) 0x7f92100b62d0 con 0x7f92100a7fc0 2022-07-20T02:33:10.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.352+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc003260 con 0x7f92100a7fc0 2022-07-20T02:33:10.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9215ca3700 1 -- 172.21.15.26:0/1461237541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100a8c40 msgr2=0x7f92100af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:10.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9215ca3700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100a8c40 0x7f92100af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9215ca3700 1 -- 172.21.15.26:0/1461237541 >> v1:172.21.15.26:6789/0 conn(0x7f92100a7fc0 legacy=0x7f92100aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421279864 0 0) 0x7f91fc003260 con 0x7f92100a7fc0 2022-07-20T02:33:10.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9215ca3700 1 -- 172.21.15.26:0/1461237541 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921014b000 con 0x7f92100b5ef0 2022-07-20T02:33:10.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9215ca3700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100b5ef0 0x7f92101474d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f9200001b80 tx=0x7f9204006df0).ready entity=mon.0 client_cookie=a64e163207f1bee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:10.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92000019e0 con 0x7f92100b5ef0 2022-07-20T02:33:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9204000e90 con 0x7f92100b5ef0 2022-07-20T02:33:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9214ca1700 1 -- 172.21.15.26:0/1461237541 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9204012460 con 0x7f92100b5ef0 2022-07-20T02:33:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- 172.21.15.26:0/1461237541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100b5ef0 msgr2=0x7f92101474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100b5ef0 0x7f92101474d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f9200001b80 tx=0x7f9204006df0).stop 2022-07-20T02:33:10.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- 172.21.15.26:0/1461237541 shutdown_connections 2022-07-20T02:33:10.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100a8c40 0x7f92100af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- 172.21.15.26:0/1461237541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100b5ef0 0x7f92101474d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- 172.21.15.26:0/1461237541 >> 172.21.15.26:0/1461237541 conn(0x7f9210060310 msgr2=0x7f92100b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:10.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- 172.21.15.26:0/1461237541 shutdown_connections 2022-07-20T02:33:10.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- 172.21.15.26:0/1461237541 wait complete. 2022-07-20T02:33:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 Processor -- start 2022-07-20T02:33:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- start start 2022-07-20T02:33:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92100a7fc0 0x7f92100af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100b5ef0 0x7f92100aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100abcf0 con 0x7f92100a8c40 2022-07-20T02:33:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210144cb0 con 0x7f92100a7fc0 2022-07-20T02:33:10.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.356+0000 7f9216ca5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92100aa730 con 0x7f92100b5ef0 2022-07-20T02:33:10.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40980/0 (socket says 172.21.15.26:40980) 2022-07-20T02:33:10.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f9215ca3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92100a7fc0 0x7f92100af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92164a4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100b5ef0 0x7f92100aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 -- 172.21.15.26:0/3141296211 learned_addr learned my addr 172.21.15.26:0/3141296211 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 -- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100b5ef0 msgr2=0x7f92100aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100b5ef0 0x7f92100aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 -- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92100a7fc0 msgr2=0x7f92100af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92100a7fc0 0x7f92100af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 -- 172.21.15.26:0/3141296211 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920400b180 con 0x7f92100a8c40 2022-07-20T02:33:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f92154a2700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f9200001bc0 tx=0x7f9200001ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9200005ba0 con 0x7f92100a8c40 2022-07-20T02:33:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200005d00 con 0x7f92100a8c40 2022-07-20T02:33:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92100626f0 con 0x7f92100a8c40 2022-07-20T02:33:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9200007300 con 0x7f92100a8c40 2022-07-20T02:33:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9210144940 con 0x7f92100a8c40 2022-07-20T02:33:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.360+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9200010030 con 0x7f92100a8c40 2022-07-20T02:33:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.364+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921014b2a0 con 0x7f92100a8c40 2022-07-20T02:33:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.364+0000 7f920effd700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 0x7f91f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.364+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f920006c1b0 con 0x7f92100a8c40 2022-07-20T02:33:10.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.364+0000 7f9215ca3700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 0x7f91f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:10.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.364+0000 7f9215ca3700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 0x7f91f0055fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f920401a000 tx=0x7f9204014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:10.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.368+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9200046e60 con 0x7f92100a8c40 2022-07-20T02:33:10.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.532+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92100574e0 con 0x7f92100a8c40 2022-07-20T02:33:10.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.532+0000 7f920effd700 1 -- 172.21.15.26:0/3141296211 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f9200042f70 con 0x7f92100a8c40 2022-07-20T02:33:10.536 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:10.537 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:10.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 msgr2=0x7f91f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 0x7f91f0055fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f920401a000 tx=0x7f9204014000).stop 2022-07-20T02:33:10.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 msgr2=0x7f921015a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:10.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f9200001bc0 tx=0x7f9200001ae0).stop 2022-07-20T02:33:10.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 shutdown_connections 2022-07-20T02:33:10.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92100a8c40 0x7f921015a590 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f91f0053b00 0x7f91f0055fc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f92100b5ef0 0x7f92100aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 --2- 172.21.15.26:0/3141296211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92100a7fc0 0x7f92100af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:10.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 >> 172.21.15.26:0/3141296211 conn(0x7f9210060310 msgr2=0x7f921005c340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:10.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 shutdown_connections 2022-07-20T02:33:10.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:10.536+0000 7f9216ca5700 1 -- 172.21.15.26:0/3141296211 wait complete. 2022-07-20T02:33:10.547 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:10.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 0x10000000000 2022-07-20T02:33:10.664 INFO:teuthology.orchestra.run.smithi026.stdout:{"path":"/foo","ino":1099511627776,"rdev":0,"ctime":"2022-07-20T02:33:09.903889+0000","btime":"2022-07-20T02:33:09.903889+0000","mode":16877,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-20T02:33:09.903889+0000","atime":"2022-07-20T02:33:09.903889+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":6275,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":6275,"want_loner":6275,"mds_caps_wanted":[]} 2022-07-20T02:33:10.676 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2022-07-20T02:33:09.903889+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "backtrace_version": 2, "btime": "2022-07-20T02:33:09.903889+0000", "change_attr": 0, "client_caps": [ { "client_id": 6275, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-20T02:33:09.903889+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "flocklock": {}, "gid": 0, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 6275, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2022-07-20T02:33:09.903889+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/foo", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": 6275, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "xattrs": [] } 2022-07-20T02:33:10.677 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:33:10.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:10.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:10.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:10.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:10.808 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:33:10.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:10.828 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:10.829 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:10.843 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:33:10.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:10.857 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:33:10.864 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:33:10.865 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:10.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23068]: fuse finished with error 0 and tester_r 0 2022-07-20T02:33:10.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:10.917 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:10.918 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:10.972 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:33:10.973 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:33:11.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:33:11.028 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:11.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:11.046 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:11.047 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:11.074 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:33:11.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:11.086 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:33:11.087 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:33:11.088 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:11.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22696]: fuse finished with error 0 and tester_r 0 2022-07-20T02:33:11.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:11.144 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:11.145 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:11.170 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:33:11.172 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:33:11.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:33:11.290 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:11.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:33:11.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 Processor -- start 2022-07-20T02:33:11.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- start start 2022-07-20T02:33:11.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff1dc0613b0 0x7ff1dc098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:11.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc0b0f20 con 0x7ff1dc143090 2022-07-20T02:33:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc143470 con 0x7ff1dc0950e0 2022-07-20T02:33:11.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc05fd90 con 0x7ff1dc0613b0 2022-07-20T02:33:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1daffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff1dc0613b0 0x7ff1dc098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:11.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1db7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff1dc143090 0x7ff1dc0b0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40266/0 (socket says 172.21.15.26:40266) 2022-07-20T02:33:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1db7fe700 1 -- 172.21.15.26:0/2998810788 learned_addr learned my addr 172.21.15.26:0/2998810788 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:11.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1daffd700 1 -- 172.21.15.26:0/2998810788 >> v1:172.21.15.120:6789/0 conn(0x7ff1dc0950e0 legacy=0x7ff1dc095530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1daffd700 1 -- 172.21.15.26:0/2998810788 >> v1:172.21.15.26:6789/0 conn(0x7ff1dc143090 legacy=0x7ff1dc0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:11.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1daffd700 1 -- 172.21.15.26:0/2998810788 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc097d50 con 0x7ff1dc0613b0 2022-07-20T02:33:11.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1daffd700 1 --2- 172.21.15.26:0/2998810788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc0613b0 0x7ff1dc098490 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7ff1cc002f70 tx=0x7ff1cc006df0).ready entity=mon.0 client_cookie=68b75f2a9ab5f571 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:11.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1da7fc700 1 -- 172.21.15.26:0/2998810788 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1cc00d610 con 0x7ff1dc0613b0 2022-07-20T02:33:11.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1da7fc700 1 -- 172.21.15.26:0/2998810788 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc000e90 con 0x7ff1dc0613b0 2022-07-20T02:33:11.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1da7fc700 1 -- 172.21.15.26:0/2998810788 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1cc00d960 con 0x7ff1dc0613b0 2022-07-20T02:33:11.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/2998810788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc0613b0 msgr2=0x7ff1dc098490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:11.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/2998810788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc0613b0 0x7ff1dc098490 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7ff1cc002f70 tx=0x7ff1cc006df0).stop 2022-07-20T02:33:11.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/2998810788 shutdown_connections 2022-07-20T02:33:11.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/2998810788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc0613b0 0x7ff1dc098490 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:11.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/2998810788 >> 172.21.15.26:0/2998810788 conn(0x7ff1dc0b5e70 msgr2=0x7ff1dc0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:11.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.416+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/2998810788 shutdown_connections 2022-07-20T02:33:11.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/2998810788 wait complete. 2022-07-20T02:33:11.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 Processor -- start 2022-07-20T02:33:11.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- start start 2022-07-20T02:33:11.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1dc0613b0 0x7ff1dc064800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:11.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1dc0950e0 0x7ff1dc09a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:11.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:11.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc063fb0 con 0x7ff1dc143090 2022-07-20T02:33:11.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc1441f0 con 0x7ff1dc0950e0 2022-07-20T02:33:11.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc144420 con 0x7ff1dc0613b0 2022-07-20T02:33:11.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:11.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40994/0 (socket says 172.21.15.26:40994) 2022-07-20T02:33:11.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 -- 172.21.15.26:0/338166054 learned_addr learned my addr 172.21.15.26:0/338166054 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1db7fe700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1dc0613b0 0x7ff1dc064800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:11.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1daffd700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1dc0950e0 0x7ff1dc09a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:11.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 -- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1dc0613b0 msgr2=0x7ff1dc064800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:11.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1dc0613b0 0x7ff1dc064800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 -- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1dc0950e0 msgr2=0x7ff1dc09a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1dc0950e0 0x7ff1dc09a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 -- 172.21.15.26:0/338166054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc00b180 con 0x7ff1dc143090 2022-07-20T02:33:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1dbfff700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7ff1dc06aea0 tx=0x7ff1d4007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:11.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1d400e4a0 con 0x7ff1dc143090 2022-07-20T02:33:11.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc1446c0 con 0x7ff1dc143090 2022-07-20T02:33:11.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d400e600 con 0x7ff1dc143090 2022-07-20T02:33:11.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1d400e960 con 0x7ff1dc143090 2022-07-20T02:33:11.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.420+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc09b9f0 con 0x7ff1dc143090 2022-07-20T02:33:11.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc144850 con 0x7ff1dc143090 2022-07-20T02:33:11.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff1d4019030 con 0x7ff1dc143090 2022-07-20T02:33:11.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1d8ff9700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 0x7ff1c0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff1d406b8d0 con 0x7ff1dc143090 2022-07-20T02:33:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1db7fe700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 0x7ff1c0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:11.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.424+0000 7ff1db7fe700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 0x7ff1c0055ff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7ff1c400b000 tx=0x7ff1c4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:11.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.428+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1d4042590 con 0x7ff1dc143090 2022-07-20T02:33:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.552+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff1dc097ff0 con 0x7ff1dc143090 2022-07-20T02:33:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.556+0000 7ff1d8ff9700 1 -- 172.21.15.26:0/338166054 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v98) v1 ==== 81+0+34088 (secure 0 0 0) 0x7ff1d4045bb0 con 0x7ff1dc143090 2022-07-20T02:33:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:11.563 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 98, 2022-07-20T02:33:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:33:11.570 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:33:11.570 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:33:00.907866+0000", 2022-07-20T02:33:11.571 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:11.571 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:11.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:33:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:33:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:33:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:33:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:33:11.572 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:33:11.573 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:33:11.573 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:11.573 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:33:11.573 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:33:11.574 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:33:11.575 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:33:11.575 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:33:11.575 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 18, 2022-07-20T02:33:11.576 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:33:11.576 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:33:11.576 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:33:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:33:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:33:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.578 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:33:11.579 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:33:11.579 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:33:11.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:11.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:11.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:11.594 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:33:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:33:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:33:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:33:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:33:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:11.617 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:11.617 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:11.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:33:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:11.628 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:11.628 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:11.630 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:11.631 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:11.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:11.632 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:11.633 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:11.635 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:11.635 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:11.636 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:11.636 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:11.637 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:11.637 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:11.638 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:11.642 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:11.642 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:11.643 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:11.643 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:11.643 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:11.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:11.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.644 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:11.654 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:11.656 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:11.656 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:11.657 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:11.658 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:11.659 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:11.659 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:11.660 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:11.660 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:11.661 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:11.661 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:11.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:33:11.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.663 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:11.663 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:33:11.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.671 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:33:11.672 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:33:11.673 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:33:11.673 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:11.673 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:11.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:11.674 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:11.689 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:11.690 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:11.691 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:11.691 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:11.692 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:11.693 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:11.695 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:11.695 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:11.695 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:11.696 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:11.713 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:11.713 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:11.714 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:11.714 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:11.715 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:11.715 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:11.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.716 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:33:11.717 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:11.717 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:11.717 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:11.718 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:11.718 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:11.719 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:11.719 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:11.720 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:11.720 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:11.721 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:11.723 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:11.725 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:11.726 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:11.726 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:11.727 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:11.727 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:11.728 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:11.728 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:11.729 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:11.731 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:11.733 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:11.734 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:33:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 17, 2022-07-20T02:33:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:33:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:32:57.859395+0000", 2022-07-20T02:33:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:11.779 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:11.779 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:11.779 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:11.779 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:11.780 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:11.780 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:11.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "97", 2022-07-20T02:33:11.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:11.781 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:11.781 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:11.797 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:11.797 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:11.797 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:11.797 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:11.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:11.798 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:11.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:11.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:11.799 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:11.799 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:11.799 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:11.799 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:11.800 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:11.800 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:11.800 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:11.800 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:11.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:11.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:11.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:11.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:11.802 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:11.802 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:11.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:11.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.803 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:11.803 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:11.804 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:11.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:11.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:11.804 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:11.804 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:11.805 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:11.805 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:11.805 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:11.805 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:11.806 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:11.806 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:33:11.806 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:33:11.806 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:33:11.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.807 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:11.808 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:33:11.808 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:33:11.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.809 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 18, 2022-07-20T02:33:11.809 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:33:11.810 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:32:58.906424+0000", 2022-07-20T02:33:11.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:11.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:11.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:11.811 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:11.811 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:11.811 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:11.812 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:11.812 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:11.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:11.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:11.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:11.814 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:11.814 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:11.814 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:11.815 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:11.815 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:11.815 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:11.815 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:11.815 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:11.816 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:11.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.816 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "98", 2022-07-20T02:33:11.816 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:11.817 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:11.817 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:11.825 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:11.825 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:11.825 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:11.825 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:11.826 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:11.826 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:11.826 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:33:11.826 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:33:11.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.827 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:11.828 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:33:11.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.829 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:11.829 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:33:11.829 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:11.829 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:11.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:11.830 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:11.830 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:11.830 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:11.830 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:11.831 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:11.831 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:11.831 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:11.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:11.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:33:11.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:33:11.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.835 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:11.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:33:11.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:33:11.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.839 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:11.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:33:11.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:33:11.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.842 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:11.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:33:11.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:33:11.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:11.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:33:11.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:33:11.846 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:33:11.846 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:33:11.847 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:11.847 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:11.847 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:11.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.848 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:11.848 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:33:11.848 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:11.849 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:11.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:11.850 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:11.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:11.850 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:11.850 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:11.851 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:11.851 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:11.851 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:11.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:11.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:33:11.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:33:11.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:33:11.856 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:33:11.857 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.858 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:11.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:33:11.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:33:11.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.863 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:11.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:33:11.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:33:11.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:11.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:33:11.866 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:33:11.867 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:33:11.867 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:33:11.867 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:11.867 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:11.867 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:11.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.868 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:11.869 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:33:11.869 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:11.869 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:11.869 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:11.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:11.872 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:11.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:11.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:11.873 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:11.873 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:11.874 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:11.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:33:11.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:33:11.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.880 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:11.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:33:11.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:33:11.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.888 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:11.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:33:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:33:11.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.894 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:11.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:33:11.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:33:11.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:11.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:33:11.916 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:33:11.916 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:33:11.917 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:33:11.917 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:11.918 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:11.918 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:11.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:33:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:33:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:33:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:33:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:33:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:33:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:33:11.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.977 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:11.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.979 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:33:11.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:11.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:33:11.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:11.982 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:11.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:33:11.984 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:33:11.984 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:33:11.985 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:33:11.985 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:11.985 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:11.986 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:11.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:11.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:11.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.988 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:11.988 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:33:11.989 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:11.990 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:11.993 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:11.994 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:11.994 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:11.995 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:11.995 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:11.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 95, 2022-07-20T02:33:11.997 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:11.997 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:11.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:11.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:11.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:11.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:11.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:33:11.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:33:12.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.005 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:12.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:33:12.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:33:12.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.010 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:12.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:33:12.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:33:12.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.015 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:12.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:33:12.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:33:12.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:12.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:33:12.021 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:33:12.021 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:33:12.023 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:33:12.023 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:12.023 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:12.024 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:12.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.026 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:12.026 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:33:12.027 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:12.027 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:12.027 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:12.028 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:12.028 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:12.029 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:12.029 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:12.030 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:33:12.030 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:12.030 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:12.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:12.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:33:12.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:33:12.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.037 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:12.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:33:12.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:33:12.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.043 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:12.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:33:12.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:33:12.048 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.049 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:12.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:33:12.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.053 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:33:12.054 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:12.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.055 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:33:12.055 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:33:12.056 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:33:12.057 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:33:12.057 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:12.057 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:12.058 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:12.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.061 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:12.062 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:33:12.062 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:12.063 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:12.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:12.063 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:12.064 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:12.064 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:12.066 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:12.066 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 95, 2022-07-20T02:33:12.067 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:12.067 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:12.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:33:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:33:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:33:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:33:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.104 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:12.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:33:12.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:33:12.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.112 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:33:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:33:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:33:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:33:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:33:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:33:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:33:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:33:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:33:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:33:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:33:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:33:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:33:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:33:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:33:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:33:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:33:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:33:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:33:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:33:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:33:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:33:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:33:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:33:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:33:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:33:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:33:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:33:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:33:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:33:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:33:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:33:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:33:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:33:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:33:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:33:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:33:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:33:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:33:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:12.213 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:33:12.213 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:33:12.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.215 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:33:12.215 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:12.216 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:12.216 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:33:12.217 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:12.217 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:12.218 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:33:12.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:12.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:12.219 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:33:12.220 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:33:12.220 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:33:12.220 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:33:12.221 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:33:12.221 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:12.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.556+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 msgr2=0x7ff1c0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.556+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 0x7ff1c0055ff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7ff1c400b000 tx=0x7ff1c4009040).stop 2022-07-20T02:33:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.556+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 msgr2=0x7ff1dc09ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.556+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7ff1dc06aea0 tx=0x7ff1d4007f40).stop 2022-07-20T02:33:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 shutdown_connections 2022-07-20T02:33:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff1dc143090 0x7ff1dc09ac70 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff1c0053b30 0x7ff1c0055ff0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1dc0950e0 0x7ff1dc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 --2- 172.21.15.26:0/338166054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1dc0613b0 0x7ff1dc064800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 >> 172.21.15.26:0/338166054 conn(0x7ff1dc0b5e70 msgr2=0x7ff1dc0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 shutdown_connections 2022-07-20T02:33:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:11.560+0000 7ff1e1d5c700 1 -- 172.21.15.26:0/338166054 wait complete. 2022-07-20T02:33:12.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:12.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.340+0000 7f8cbd501700 1 Processor -- start 2022-07-20T02:33:12.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.340+0000 7f8cbd501700 1 -- start start 2022-07-20T02:33:12.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cb80b0c50 0x7f8cb80957f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb8143470 con 0x7f8cb80b0810 2022-07-20T02:33:12.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb805fd90 con 0x7f8cb80b0c50 2022-07-20T02:33:12.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8097d50 con 0x7f8cb8143090 2022-07-20T02:33:12.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cb80b0c50 0x7f8cb80957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8cb80b0c50 0x7f8cb80957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41004/0 (socket says 172.21.15.26:41004) 2022-07-20T02:33:12.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 -- 172.21.15.26:0/1022549221 learned_addr learned my addr 172.21.15.26:0/1022549221 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:12.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb6ffd700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 -- 172.21.15.26:0/1022549221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 msgr2=0x7f8cb8098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:12.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 -- 172.21.15.26:0/1022549221 >> v1:172.21.15.120:6789/0 conn(0x7f8cb80b0810 legacy=0x7f8cb80950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 -- 172.21.15.26:0/1022549221 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb8097ff0 con 0x7f8cb80b0c50 2022-07-20T02:33:12.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb77fe700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0c50 0x7f8cb80957f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f8cb8098c00 tx=0x7f8cb000b530).ready entity=mon.0 client_cookie=41b35094401250c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb67fc700 1 -- 172.21.15.26:0/1022549221 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb00153f0 con 0x7f8cb80b0c50 2022-07-20T02:33:12.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb67fc700 1 -- 172.21.15.26:0/1022549221 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0015550 con 0x7f8cb80b0c50 2022-07-20T02:33:12.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cb67fc700 1 -- 172.21.15.26:0/1022549221 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb00158b0 con 0x7f8cb80b0c50 2022-07-20T02:33:12.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- 172.21.15.26:0/1022549221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0c50 msgr2=0x7f8cb80957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0c50 0x7f8cb80957f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f8cb8098c00 tx=0x7f8cb000b530).stop 2022-07-20T02:33:12.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- 172.21.15.26:0/1022549221 shutdown_connections 2022-07-20T02:33:12.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0c50 0x7f8cb80957f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 --2- 172.21.15.26:0/1022549221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- 172.21.15.26:0/1022549221 >> 172.21.15.26:0/1022549221 conn(0x7f8cb80b5e70 msgr2=0x7f8cb80b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- 172.21.15.26:0/1022549221 shutdown_connections 2022-07-20T02:33:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.344+0000 7f8cbd501700 1 -- 172.21.15.26:0/1022549221 wait complete. 2022-07-20T02:33:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 Processor -- start 2022-07-20T02:33:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- start start 2022-07-20T02:33:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cb80b0c50 0x7f8cb8066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80acea0 con 0x7f8cb80b0810 2022-07-20T02:33:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80aecd0 con 0x7f8cb80b0c50 2022-07-20T02:33:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80652e0 con 0x7f8cb8143090 2022-07-20T02:33:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41006/0 (socket says 172.21.15.26:41006) 2022-07-20T02:33:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 -- 172.21.15.26:0/2277091386 learned_addr learned my addr 172.21.15.26:0/2277091386 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8caffff700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cb80b0c50 0x7f8cb8066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb77fe700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 -- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 msgr2=0x7f8cb8151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 -- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cb80b0c50 msgr2=0x7f8cb8066d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cb80b0c50 0x7f8cb8066d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 -- 172.21.15.26:0/2277091386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb000b7d0 con 0x7f8cb80b0810 2022-07-20T02:33:12.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb6ffd700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f8cb8065ab0 tx=0x7f8ca8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca8010780 con 0x7f8cb80b0810 2022-07-20T02:33:12.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca80108e0 con 0x7f8cb80b0810 2022-07-20T02:33:12.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb8065510 con 0x7f8cb80b0810 2022-07-20T02:33:12.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca8010c40 con 0x7f8cb80b0810 2022-07-20T02:33:12.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.348+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80afd50 con 0x7f8cb80b0810 2022-07-20T02:33:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb80673e0 con 0x7f8cb80b0810 2022-07-20T02:33:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8ca801c460 con 0x7f8cb80b0810 2022-07-20T02:33:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8cb4ff9700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 0x7f8c9c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8ca806c4f0 con 0x7f8cb80b0810 2022-07-20T02:33:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8caffff700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 0x7f8c9c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.352+0000 7f8caffff700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 0x7f8c9c055ea0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8ca000d000 tx=0x7f8ca0007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.356+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ca80431b0 con 0x7f8cb80b0810 2022-07-20T02:33:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.524+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb8095ce0 con 0x7f8cb80b0810 2022-07-20T02:33:12.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.524+0000 7f8cb4ff9700 1 -- 172.21.15.26:0/2277091386 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2496 (secure 0 0 0) 0x7f8ca80467d0 con 0x7f8cb80b0810 2022-07-20T02:33:12.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:12.532 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:00.907175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7053":{"gid":7053,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/2339237946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2339237946},{"type":"v1","addr":"172.21.15.26:6837","nonce":2339237946}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 msgr2=0x7f8c9c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 0x7f8c9c055ea0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8ca000d000 tx=0x7f8ca0007b90).stop 2022-07-20T02:33:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 msgr2=0x7f8cb814eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f8cb8065ab0 tx=0x7f8ca8006df0).stop 2022-07-20T02:33:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 shutdown_connections 2022-07-20T02:33:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c9c0539e0 0x7f8c9c055ea0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8cb8143090 0x7f8cb8151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cb80b0c50 0x7f8cb8066d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 --2- 172.21.15.26:0/2277091386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8cb80b0810 0x7f8cb814eb80 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 >> 172.21.15.26:0/2277091386 conn(0x7f8cb80b5e70 msgr2=0x7f8cb8140970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 shutdown_connections 2022-07-20T02:33:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.528+0000 7f8cbd501700 1 -- 172.21.15.26:0/2277091386 wait complete. 2022-07-20T02:33:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 48 2022-07-20T02:33:12.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:33:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.672+0000 7efed8ae1700 1 Processor -- start 2022-07-20T02:33:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.672+0000 7efed8ae1700 1 -- start start 2022-07-20T02:33:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efed4060250 0x7efed4060700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40b0f50 con 0x7efed40aff40 2022-07-20T02:33:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed413e7f0 con 0x7efed4060250 2022-07-20T02:33:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40b0320 con 0x7efed40621f0 2022-07-20T02:33:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efed40aff40 0x7efed4066b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40294/0 (socket says 172.21.15.26:40294) 2022-07-20T02:33:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed37fe700 1 -- 172.21.15.26:0/1375732937 learned_addr learned my addr 172.21.15.26:0/1375732937 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 --2- 172.21.15.26:0/1375732937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed3fff700 1 --2- 172.21.15.26:0/1375732937 >> v2:172.21.15.120:3300/0 conn(0x7efed4060250 0x7efed4060700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed2ffd700 1 -- 172.21.15.26:0/1375732937 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981400211 0 0) 0x7efed40b0f50 con 0x7efed40aff40 2022-07-20T02:33:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed2ffd700 1 -- 172.21.15.26:0/1375732937 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003260 con 0x7efed40aff40 2022-07-20T02:33:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 -- 172.21.15.26:0/1375732937 >> v2:172.21.15.120:3300/0 conn(0x7efed4060250 msgr2=0x7efed4060700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:12.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 --2- 172.21.15.26:0/1375732937 >> v2:172.21.15.120:3300/0 conn(0x7efed4060250 0x7efed4060700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 -- 172.21.15.26:0/1375732937 >> v1:172.21.15.26:6789/0 conn(0x7efed40aff40 legacy=0x7efed4066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 -- 172.21.15.26:0/1375732937 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed405f8e0 con 0x7efed40621f0 2022-07-20T02:33:12.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efecbfff700 1 --2- 172.21.15.26:0/1375732937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4067240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efebc002f70 tx=0x7efebc006df0).ready entity=mon.2 client_cookie=b1ca3e64e2667bc1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed2ffd700 1 -- 172.21.15.26:0/1375732937 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc00d4a0 con 0x7efed40621f0 2022-07-20T02:33:12.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed2ffd700 1 -- 172.21.15.26:0/1375732937 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc000e90 con 0x7efed40621f0 2022-07-20T02:33:12.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed2ffd700 1 -- 172.21.15.26:0/1375732937 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc00d7f0 con 0x7efed40621f0 2022-07-20T02:33:12.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.676+0000 7efed8ae1700 1 -- 172.21.15.26:0/1375732937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 msgr2=0x7efed4067240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 --2- 172.21.15.26:0/1375732937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4067240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efebc002f70 tx=0x7efebc006df0).stop 2022-07-20T02:33:12.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 -- 172.21.15.26:0/1375732937 shutdown_connections 2022-07-20T02:33:12.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 --2- 172.21.15.26:0/1375732937 >> v2:172.21.15.120:3300/0 conn(0x7efed4060250 0x7efed4060700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 --2- 172.21.15.26:0/1375732937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4067240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 -- 172.21.15.26:0/1375732937 >> 172.21.15.26:0/1375732937 conn(0x7efed40b3050 msgr2=0x7efed40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 -- 172.21.15.26:0/1375732937 shutdown_connections 2022-07-20T02:33:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 -- 172.21.15.26:0/1375732937 wait complete. 2022-07-20T02:33:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 Processor -- start 2022-07-20T02:33:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.680+0000 7efed8ae1700 1 -- start start 2022-07-20T02:33:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4063b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed40aff40 0x7efed40b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed414fe60 con 0x7efed4060250 2022-07-20T02:33:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed4150050 con 0x7efed40aff40 2022-07-20T02:33:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed41502f0 con 0x7efed40621f0 2022-07-20T02:33:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41020/0 (socket says 172.21.15.26:41020) 2022-07-20T02:33:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 -- 172.21.15.26:0/3971531310 learned_addr learned my addr 172.21.15.26:0/3971531310 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed3fff700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed40aff40 0x7efed40b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efecbfff700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4063b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 -- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 msgr2=0x7efed4063b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4063b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 -- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed40aff40 msgr2=0x7efed40b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed40aff40 0x7efed40b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 -- 172.21.15.26:0/3971531310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc00b180 con 0x7efed4060250 2022-07-20T02:33:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed37fe700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7efec4001110 tx=0x7efec4007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec40070d0 con 0x7efed4060250 2022-07-20T02:33:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec4007230 con 0x7efed4060250 2022-07-20T02:33:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec4005610 con 0x7efed4060250 2022-07-20T02:33:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed4150590 con 0x7efed4060250 2022-07-20T02:33:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed4064020 con 0x7efed4060250 2022-07-20T02:33:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.684+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed4150720 con 0x7efed4060250 2022-07-20T02:33:12.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.688+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efec401a070 con 0x7efed4060250 2022-07-20T02:33:12.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.688+0000 7efed17fa700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 0x7efeac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:12.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.688+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6420+0+0 (secure 0 0 0) 0x7efec406b2b0 con 0x7efed4060250 2022-07-20T02:33:12.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.688+0000 7efecbfff700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 0x7efeac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:12.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.688+0000 7efecbfff700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 0x7efeac055ea0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7efebc01a000 tx=0x7efebc014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:12.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.696+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efec4042010 con 0x7efed4060250 2022-07-20T02:33:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:12.860+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efed405fb10 con 0x7efed4060250 2022-07-20T02:33:13.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:33:13.444+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:33:13.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.456+0000 7efed17fa700 1 -- 172.21.15.26:0/3971531310 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7efec4018070 con 0x7efed4060250 2022-07-20T02:33:13.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 msgr2=0x7efeac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 0x7efeac055ea0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7efebc01a000 tx=0x7efebc014000).stop 2022-07-20T02:33:13.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 msgr2=0x7efed4150a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7efec4001110 tx=0x7efec4007b20).stop 2022-07-20T02:33:13.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 shutdown_connections 2022-07-20T02:33:13.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeac0539e0 0x7efeac055ea0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed40aff40 0x7efed40b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed40621f0 0x7efed4063b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 --2- 172.21.15.26:0/3971531310 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed4060250 0x7efed4150a30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 >> 172.21.15.26:0/3971531310 conn(0x7efed40b3050 msgr2=0x7efed413ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:13.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 shutdown_connections 2022-07-20T02:33:13.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.460+0000 7efed8ae1700 1 -- 172.21.15.26:0/3971531310 wait complete. 2022-07-20T02:33:13.467 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:33:13.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:13.492 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:33:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 Processor -- start 2022-07-20T02:33:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 -- start start 2022-07-20T02:33:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58ac143070 0x7f58ac1434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac146750 con 0x7f58ac0b5d50 2022-07-20T02:33:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac067ba0 con 0x7f58ac143070 2022-07-20T02:33:13.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b289a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1469f0 con 0x7f58ac140320 2022-07-20T02:33:13.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b1097700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58ac143070 0x7f58ac1434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b2099700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.576+0000 7f58b1898700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f58ac0b5d50 0x7f58ac067490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:54532/0 (socket says 172.21.15.26:54532) 2022-07-20T02:33:13.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34328/0 (socket says 172.21.15.26:34328) 2022-07-20T02:33:13.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b1898700 1 -- 172.21.15.26:0/1542463301 learned_addr learned my addr 172.21.15.26:0/1542463301 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:13.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b0896700 1 -- 172.21.15.26:0/1542463301 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336744270 0 0) 0x7f58ac146750 con 0x7f58ac0b5d50 2022-07-20T02:33:13.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 -- 172.21.15.26:0/1542463301 >> v2:172.21.15.120:3300/0 conn(0x7f58ac143070 msgr2=0x7f58ac1434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:13.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 --2- 172.21.15.26:0/1542463301 >> v2:172.21.15.120:3300/0 conn(0x7f58ac143070 0x7f58ac1434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 -- 172.21.15.26:0/1542463301 >> v1:172.21.15.120:6789/0 conn(0x7f58ac0b5d50 legacy=0x7f58ac067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 -- 172.21.15.26:0/1542463301 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac146c90 con 0x7f58ac140320 2022-07-20T02:33:13.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b2099700 1 --2- 172.21.15.26:0/1542463301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f58a4002f70 tx=0x7f58a4006df0).ready entity=mon.2 client_cookie=f0a64f42aaa5b9c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b0896700 1 -- 172.21.15.26:0/1542463301 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a400d570 con 0x7f58ac140320 2022-07-20T02:33:13.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b0896700 1 -- 172.21.15.26:0/1542463301 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4000e90 con 0x7f58ac140320 2022-07-20T02:33:13.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b0896700 1 -- 172.21.15.26:0/1542463301 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a400d8c0 con 0x7f58ac140320 2022-07-20T02:33:13.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 -- 172.21.15.26:0/1542463301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 msgr2=0x7f58ac1407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 --2- 172.21.15.26:0/1542463301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f58a4002f70 tx=0x7f58a4006df0).stop 2022-07-20T02:33:13.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 -- 172.21.15.26:0/1542463301 shutdown_connections 2022-07-20T02:33:13.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 --2- 172.21.15.26:0/1542463301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac140320 0x7f58ac1407d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 --2- 172.21.15.26:0/1542463301 >> v2:172.21.15.120:3300/0 conn(0x7f58ac143070 0x7f58ac1434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 -- 172.21.15.26:0/1542463301 >> 172.21.15.26:0/1542463301 conn(0x7f58ac060310 msgr2=0x7f58ac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:13.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 -- 172.21.15.26:0/1542463301 shutdown_connections 2022-07-20T02:33:13.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.580+0000 7f58b289a700 1 -- 172.21.15.26:0/1542463301 wait complete. 2022-07-20T02:33:13.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 Processor -- start 2022-07-20T02:33:13.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 -- start start 2022-07-20T02:33:13.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac0b5d50 0x7f58ac142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 0x7f58ac0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1458b0 con 0x7f58ac140320 2022-07-20T02:33:13.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac13d4f0 con 0x7f58ac143070 2022-07-20T02:33:13.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b289a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac13d720 con 0x7f58ac0b5d50 2022-07-20T02:33:13.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.584+0000 7f58b1097700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1898700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac0b5d50 0x7f58ac142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41048/0 (socket says 172.21.15.26:41048) 2022-07-20T02:33:13.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 -- 172.21.15.26:0/2179500214 learned_addr learned my addr 172.21.15.26:0/2179500214 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:13.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b2099700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 0x7f58ac0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 -- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac0b5d50 msgr2=0x7f58ac142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac0b5d50 0x7f58ac142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 -- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 msgr2=0x7f58ac0b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 0x7f58ac0b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 -- 172.21.15.26:0/2179500214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a400b180 con 0x7f58ac140320 2022-07-20T02:33:13.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b2099700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 0x7f58ac0b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:13.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b1097700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f58ac142390 tx=0x7f58a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a0010780 con 0x7f58ac140320 2022-07-20T02:33:13.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a00108e0 con 0x7f58ac140320 2022-07-20T02:33:13.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac13d9c0 con 0x7f58ac140320 2022-07-20T02:33:13.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58a0010c40 con 0x7f58ac140320 2022-07-20T02:33:13.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac140fe0 con 0x7f58ac140320 2022-07-20T02:33:13.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.588+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac146f30 con 0x7f58ac140320 2022-07-20T02:33:13.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.592+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f58a001c460 con 0x7f58ac140320 2022-07-20T02:33:13.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.592+0000 7f58aaffd700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 0x7f588c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.592+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f58a006c400 con 0x7f58ac140320 2022-07-20T02:33:13.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.592+0000 7f58b1898700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 0x7f588c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.592+0000 7f58b1898700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 0x7f588c055f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f589c00d000 tx=0x7f589c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.596+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58a0043110 con 0x7f58ac140320 2022-07-20T02:33:13.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.756+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ac0574e0 con 0x7f58ac140320 2022-07-20T02:33:13.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.756+0000 7f58aaffd700 1 -- 172.21.15.26:0/2179500214 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f58a0046730 con 0x7f58ac140320 2022-07-20T02:33:13.766 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:13.766 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":49,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":49,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:32:59.904474+0000","modified":"2022-07-20T02:33:13.445003+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T02:33:13.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 msgr2=0x7f588c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 0x7f588c055f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f589c00d000 tx=0x7f589c009040).stop 2022-07-20T02:33:13.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 msgr2=0x7f58ac0b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f58ac142390 tx=0x7f58a0006df0).stop 2022-07-20T02:33:13.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 shutdown_connections 2022-07-20T02:33:13.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58ac140320 0x7f58ac0b38f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f588c053a70 0x7f588c055f30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58ac143070 0x7f58ac0b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 --2- 172.21.15.26:0/2179500214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58ac0b5d50 0x7f58ac142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 >> 172.21.15.26:0/2179500214 conn(0x7f58ac060310 msgr2=0x7f58ac0b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:13.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 shutdown_connections 2022-07-20T02:33:13.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.760+0000 7f58b289a700 1 -- 172.21.15.26:0/2179500214 wait complete. 2022-07-20T02:33:13.773 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 49 2022-07-20T02:33:13.775 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:33:13.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.892+0000 7f70c49aa700 1 Processor -- start 2022-07-20T02:33:13.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.892+0000 7f70c49aa700 1 -- start start 2022-07-20T02:33:13.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70c01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f70c00a7fc0 0x7f70c01479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c00b62d0 con 0x7f70c00a8c40 2022-07-20T02:33:13.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c0147eb0 con 0x7f70c00a7fc0 2022-07-20T02:33:13.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c014aac0 con 0x7f70c00b5ef0 2022-07-20T02:33:13.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f70c00a8c40 0x7f70c00aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40334/0 (socket says 172.21.15.26:40334) 2022-07-20T02:33:13.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 -- 172.21.15.26:0/912263481 learned_addr learned my addr 172.21.15.26:0/912263481 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:13.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022539918 0 0) 0x7f70c00b62d0 con 0x7f70c00a8c40 2022-07-20T02:33:13.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70be59c700 1 --2- 172.21.15.26:0/912263481 >> v2:172.21.15.120:3300/0 conn(0x7f70c00a7fc0 0x7f70c01479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bed9d700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70c01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a4003260 con 0x7f70c00a8c40 2022-07-20T02:33:13.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608537821 0 0) 0x7f70a4003260 con 0x7f70c00a8c40 2022-07-20T02:33:13.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 msgr2=0x7f70c01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:13.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70c01474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 >> v2:172.21.15.120:3300/0 conn(0x7f70c00a7fc0 msgr2=0x7f70c01479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:13.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 --2- 172.21.15.26:0/912263481 >> v2:172.21.15.120:3300/0 conn(0x7f70c00a7fc0 0x7f70c01479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c014ad60 con 0x7f70c00a8c40 2022-07-20T02:33:13.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f70b8001660 con 0x7f70c00a8c40 2022-07-20T02:33:13.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bed9d700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70c01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:13.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 >> v1:172.21.15.26:6789/0 conn(0x7f70c00a8c40 legacy=0x7f70c00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70a4003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70a4003c60 0x7f70a4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70a4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c0147eb0 con 0x7f70a40049f0 2022-07-20T02:33:13.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c014aac0 con 0x7f70a4003c60 2022-07-20T02:33:13.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a400b750 con 0x7f70c00b5ef0 2022-07-20T02:33:13.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70a4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70be59c700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70a4003c60 0x7f70a4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bed9d700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70a4003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 -- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 msgr2=0x7f70a4003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:13.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70a4003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 -- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70a4003c60 msgr2=0x7f70a4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:13.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70a4003c60 0x7f70a4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 -- 172.21.15.26:0/912263481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4003260 con 0x7f70a40049f0 2022-07-20T02:33:13.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bed9d700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c00b5ef0 0x7f70a4003750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:13.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70a4004e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f70a40054d0 tx=0x7f70b8001e50).ready entity=mon.0 client_cookie=928080621adaec7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b80070d0 con 0x7f70a40049f0 2022-07-20T02:33:13.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b8007230 con 0x7f70a40049f0 2022-07-20T02:33:13.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bdd9b700 1 -- 172.21.15.26:0/912263481 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b800c980 con 0x7f70a40049f0 2022-07-20T02:33:13.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- 172.21.15.26:0/912263481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 msgr2=0x7f70a4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 --2- 172.21.15.26:0/912263481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70a4004e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f70a40054d0 tx=0x7f70b8001e50).stop 2022-07-20T02:33:13.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70bf59e700 1 -- 172.21.15.26:0/912263481 reap_dead start 2022-07-20T02:33:13.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- 172.21.15.26:0/912263481 shutdown_connections 2022-07-20T02:33:13.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- 172.21.15.26:0/912263481 >> 172.21.15.26:0/912263481 conn(0x7f70c0060310 msgr2=0x7f70c00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:13.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- 172.21.15.26:0/912263481 shutdown_connections 2022-07-20T02:33:13.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.896+0000 7f70c49aa700 1 -- 172.21.15.26:0/912263481 wait complete. 2022-07-20T02:33:13.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.900+0000 7f70c49aa700 1 Processor -- start 2022-07-20T02:33:13.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.900+0000 7f70c49aa700 1 -- start start 2022-07-20T02:33:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c0062a30 0x7f70c0062eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70c0159b80 0x7f70c015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c00ab4b0 con 0x7f70a40049f0 2022-07-20T02:33:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c00aac80 con 0x7f70c0159b80 2022-07-20T02:33:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c00aa9e0 con 0x7f70c0062a30 2022-07-20T02:33:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41066/0 (socket says 172.21.15.26:41066) 2022-07-20T02:33:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 -- 172.21.15.26:0/1476007768 learned_addr learned my addr 172.21.15.26:0/1476007768 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bf59e700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70c0159b80 0x7f70c015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70be59c700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c0062a30 0x7f70c0062eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 -- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c0062a30 msgr2=0x7f70c0062eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c0062a30 0x7f70c0062eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 -- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70c0159b80 msgr2=0x7f70c015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:13.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70c0159b80 0x7f70c015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:13.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 -- 172.21.15.26:0/1476007768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c00ab210 con 0x7f70a40049f0 2022-07-20T02:33:13.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70bed9d700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f70b4008000 tx=0x7f70b4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b400c070 con 0x7f70a40049f0 2022-07-20T02:33:13.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4000e90 con 0x7f70a40049f0 2022-07-20T02:33:13.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c0065210 con 0x7f70a40049f0 2022-07-20T02:33:13.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b40133f0 con 0x7f70a40049f0 2022-07-20T02:33:13.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c00b52d0 con 0x7f70a40049f0 2022-07-20T02:33:13.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.904+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c00ac2d0 con 0x7f70a40049f0 2022-07-20T02:33:13.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.908+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70b4019020 con 0x7f70a40049f0 2022-07-20T02:33:13.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.908+0000 7f70affff700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 0x7f709c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:13.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.908+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f70b406b660 con 0x7f70a40049f0 2022-07-20T02:33:13.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.908+0000 7f70be59c700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 0x7f709c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:13.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.908+0000 7f70be59c700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 0x7f709c055fc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f70b000b000 tx=0x7f70b0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:13.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:13.912+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70b40423d0 con 0x7f70a40049f0 2022-07-20T02:33:14.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.076+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f70c014b000 con 0x7f70a40049f0 2022-07-20T02:33:14.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.464+0000 7f70affff700 1 -- 172.21.15.26:0/1476007768 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f70b40459f0 con 0x7f70a40049f0 2022-07-20T02:33:14.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 msgr2=0x7f709c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 0x7f709c055fc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f70b000b000 tx=0x7f70b0008040).stop 2022-07-20T02:33:14.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 msgr2=0x7f70c00b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f70b4008000 tx=0x7f70b4008040).stop 2022-07-20T02:33:14.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 shutdown_connections 2022-07-20T02:33:14.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f709c053b00 0x7f709c055fc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70c0159b80 0x7f70c015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f70c0062a30 0x7f70c0062eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 --2- 172.21.15.26:0/1476007768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f70a40049f0 0x7f70c00b4780 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 >> 172.21.15.26:0/1476007768 conn(0x7f70c0060310 msgr2=0x7f70c00ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:14.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 shutdown_connections 2022-07-20T02:33:14.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.468+0000 7f70c49aa700 1 -- 172.21.15.26:0/1476007768 wait complete. 2022-07-20T02:33:14.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:33:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.616+0000 7fc292174700 1 Processor -- start 2022-07-20T02:33:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.616+0000 7fc292174700 1 -- start start 2022-07-20T02:33:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 0x7fc28c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc28c0957f0 0x7fc28c1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c147080 con 0x7fc28c067440 2022-07-20T02:33:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c143000 con 0x7fc28c0957f0 2022-07-20T02:33:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c1472b0 con 0x7fc28c0b2f20 2022-07-20T02:33:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc28c0957f0 0x7fc28c1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc28c0957f0 0x7fc28c1433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41078/0 (socket says 172.21.15.26:41078) 2022-07-20T02:33:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 -- 172.21.15.26:0/2951067396 learned_addr learned my addr 172.21.15.26:0/2951067396 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc290971700 1 --1- 172.21.15.26:0/2951067396 >> v1:172.21.15.26:6789/0 conn(0x7fc28c067440 0x7fc28c0950e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40360/0 (socket says 172.21.15.26:40360) 2022-07-20T02:33:14.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291172700 1 --2- 172.21.15.26:0/2951067396 >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 0x7fc28c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 -- 172.21.15.26:0/2951067396 >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 msgr2=0x7fc28c0b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:14.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 --2- 172.21.15.26:0/2951067396 >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 0x7fc28c0b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 -- 172.21.15.26:0/2951067396 >> v1:172.21.15.26:6789/0 conn(0x7fc28c067440 legacy=0x7fc28c0950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 -- 172.21.15.26:0/2951067396 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0a9920 con 0x7fc28c0957f0 2022-07-20T02:33:14.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291172700 1 --2- 172.21.15.26:0/2951067396 >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 0x7fc28c0b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:14.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc291973700 1 --2- 172.21.15.26:0/2951067396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c1433f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7fc2840032f0 tx=0x7fc284009290).ready entity=mon.0 client_cookie=8371efcccca748fc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:14.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc27ffff700 1 -- 172.21.15.26:0/2951067396 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc28400b870 con 0x7fc28c0957f0 2022-07-20T02:33:14.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc27ffff700 1 -- 172.21.15.26:0/2951067396 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28400b9d0 con 0x7fc28c0957f0 2022-07-20T02:33:14.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc27ffff700 1 -- 172.21.15.26:0/2951067396 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc28400bb40 con 0x7fc28c0957f0 2022-07-20T02:33:14.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- 172.21.15.26:0/2951067396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 msgr2=0x7fc28c1433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 --2- 172.21.15.26:0/2951067396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c1433f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7fc2840032f0 tx=0x7fc284009290).stop 2022-07-20T02:33:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- 172.21.15.26:0/2951067396 shutdown_connections 2022-07-20T02:33:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 --2- 172.21.15.26:0/2951067396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c1433f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 --2- 172.21.15.26:0/2951067396 >> v2:172.21.15.120:3300/0 conn(0x7fc28c0b2f20 0x7fc28c0b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- 172.21.15.26:0/2951067396 >> 172.21.15.26:0/2951067396 conn(0x7fc28c0b67f0 msgr2=0x7fc28c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:14.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- 172.21.15.26:0/2951067396 shutdown_connections 2022-07-20T02:33:14.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.620+0000 7fc292174700 1 -- 172.21.15.26:0/2951067396 wait complete. 2022-07-20T02:33:14.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 Processor -- start 2022-07-20T02:33:14.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- start start 2022-07-20T02:33:14.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28c067440 0x7fc28c09b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28c0b2f20 0x7fc28c0904b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c1564a0 con 0x7fc28c0957f0 2022-07-20T02:33:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c09a070 con 0x7fc28c0b2f20 2022-07-20T02:33:14.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0afea0 con 0x7fc28c067440 2022-07-20T02:33:14.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41084/0 (socket says 172.21.15.26:41084) 2022-07-20T02:33:14.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 -- 172.21.15.26:0/659324175 learned_addr learned my addr 172.21.15.26:0/659324175 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:14.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc291172700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28c067440 0x7fc28c09b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc291973700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28c0b2f20 0x7fc28c0904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 -- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28c067440 msgr2=0x7fc28c09b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28c067440 0x7fc28c09b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 -- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28c0b2f20 msgr2=0x7fc28c0904b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:14.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28c0b2f20 0x7fc28c0904b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:14.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 -- 172.21.15.26:0/659324175 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284009530 con 0x7fc28c0957f0 2022-07-20T02:33:14.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc290971700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7fc28c0b5c80 tx=0x7fc280007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:14.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc28000ce30 con 0x7fc28c0957f0 2022-07-20T02:33:14.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c1566d0 con 0x7fc28c0957f0 2022-07-20T02:33:14.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2800072e0 con 0x7fc28c0957f0 2022-07-20T02:33:14.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c09aae0 con 0x7fc28c0957f0 2022-07-20T02:33:14.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.624+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc280005740 con 0x7fc28c0957f0 2022-07-20T02:33:14.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c156860 con 0x7fc28c0957f0 2022-07-20T02:33:14.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc280010070 con 0x7fc28c0957f0 2022-07-20T02:33:14.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc27e7fc700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 0x7fc26c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:14.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fc28006b6c0 con 0x7fc28c0957f0 2022-07-20T02:33:14.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc291172700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 0x7fc26c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:14.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.628+0000 7fc291172700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 0x7fc26c055e10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fc28800b000 tx=0x7fc288008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:14.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.636+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc280042430 con 0x7fc28c0957f0 2022-07-20T02:33:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:14.764+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc28c0a95a0 con 0x7fc28c0957f0 2022-07-20T02:33:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.468+0000 7fc27e7fc700 1 -- 172.21.15.26:0/659324175 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 185+0+0 (secure 0 0 0) 0x7fc280045a50 con 0x7fc28c0957f0 2022-07-20T02:33:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 msgr2=0x7fc26c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 0x7fc26c055e10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fc28800b000 tx=0x7fc288008040).stop 2022-07-20T02:33:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 msgr2=0x7fc28c0b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7fc28c0b5c80 tx=0x7fc280007c00).stop 2022-07-20T02:33:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 shutdown_connections 2022-07-20T02:33:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc28c0957f0 0x7fc28c0b1910 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc26c053950 0x7fc26c055e10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28c0b2f20 0x7fc28c0904b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 --2- 172.21.15.26:0/659324175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28c067440 0x7fc28c09b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 >> 172.21.15.26:0/659324175 conn(0x7fc28c0b67f0 msgr2=0x7fc28c0b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 shutdown_connections 2022-07-20T02:33:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.472+0000 7fc292174700 1 -- 172.21.15.26:0/659324175 wait complete. 2022-07-20T02:33:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:33:15.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:33:15.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.612+0000 7fe1bffff700 1 Processor -- start 2022-07-20T02:33:15.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.612+0000 7fe1bffff700 1 -- start start 2022-07-20T02:33:15.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe1c005cfd0 0x7fe1c0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1c0060f00 0x7fe1c013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c01474e0 con 0x7fe1c0147000 2022-07-20T02:33:15.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c00b4e40 con 0x7fe1c0060f00 2022-07-20T02:33:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c005d3b0 con 0x7fe1c005cfd0 2022-07-20T02:33:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1c0060f00 0x7fe1c013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1c0060f00 0x7fe1c013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41094/0 (socket says 172.21.15.26:41094) 2022-07-20T02:33:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 -- 172.21.15.26:0/3259108552 learned_addr learned my addr 172.21.15.26:0/3259108552 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1beffd700 1 --2- 172.21.15.26:0/3259108552 >> v2:172.21.15.120:3300/0 conn(0x7fe1c005cfd0 0x7fe1c0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 -- 172.21.15.26:0/3259108552 >> v2:172.21.15.120:3300/0 conn(0x7fe1c005cfd0 msgr2=0x7fe1c0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 --2- 172.21.15.26:0/3259108552 >> v2:172.21.15.120:3300/0 conn(0x7fe1c005cfd0 0x7fe1c0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 -- 172.21.15.26:0/3259108552 >> v1:172.21.15.120:6789/0 conn(0x7fe1c0147000 legacy=0x7fe1c00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 -- 172.21.15.26:0/3259108552 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c01431e0 con 0x7fe1c0060f00 2022-07-20T02:33:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bf7fe700 1 --2- 172.21.15.26:0/3259108552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0060f00 0x7fe1c013c9b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fe1b80032f0 tx=0x7fe1b8009290).ready entity=mon.0 client_cookie=f82776e2ed42574c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bdffb700 1 -- 172.21.15.26:0/3259108552 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1b800b870 con 0x7fe1c0060f00 2022-07-20T02:33:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bdffb700 1 -- 172.21.15.26:0/3259108552 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b800b9d0 con 0x7fe1c0060f00 2022-07-20T02:33:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bdffb700 1 -- 172.21.15.26:0/3259108552 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1b800bb40 con 0x7fe1c0060f00 2022-07-20T02:33:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- 172.21.15.26:0/3259108552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0060f00 msgr2=0x7fe1c013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 --2- 172.21.15.26:0/3259108552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0060f00 0x7fe1c013c9b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fe1b80032f0 tx=0x7fe1b8009290).stop 2022-07-20T02:33:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- 172.21.15.26:0/3259108552 shutdown_connections 2022-07-20T02:33:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 --2- 172.21.15.26:0/3259108552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0060f00 0x7fe1c013c9b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 --2- 172.21.15.26:0/3259108552 >> v2:172.21.15.120:3300/0 conn(0x7fe1c005cfd0 0x7fe1c0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- 172.21.15.26:0/3259108552 >> 172.21.15.26:0/3259108552 conn(0x7fe1c00620c0 msgr2=0x7fe1c005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- 172.21.15.26:0/3259108552 shutdown_connections 2022-07-20T02:33:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.616+0000 7fe1bffff700 1 -- 172.21.15.26:0/3259108552 wait complete. 2022-07-20T02:33:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 Processor -- start 2022-07-20T02:33:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- start start 2022-07-20T02:33:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1c0060f00 0x7fe1c00b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 0x7fe1c00b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0142ea0 con 0x7fe1c0147000 2022-07-20T02:33:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0152d80 con 0x7fe1c00b7b10 2022-07-20T02:33:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00b3580 con 0x7fe1c0060f00 2022-07-20T02:33:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41098/0 (socket says 172.21.15.26:41098) 2022-07-20T02:33:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 -- 172.21.15.26:0/4131916793 learned_addr learned my addr 172.21.15.26:0/4131916793 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1beffd700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1c0060f00 0x7fe1c00b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bf7fe700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 0x7fe1c00b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 -- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1c0060f00 msgr2=0x7fe1c00b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1c0060f00 0x7fe1c00b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 -- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 msgr2=0x7fe1c00b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 0x7fe1c00b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 -- 172.21.15.26:0/4131916793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009530 con 0x7fe1c0147000 2022-07-20T02:33:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bf7fe700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 0x7fe1c00b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1be7fc700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fe1c0068660 tx=0x7fe1a8007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1a800ce30 con 0x7fe1c0147000 2022-07-20T02:33:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a8007240 con 0x7fe1c0147000 2022-07-20T02:33:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c0140f00 con 0x7fe1c0147000 2022-07-20T02:33:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1a8006700 con 0x7fe1c0147000 2022-07-20T02:33:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.620+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c0152f90 con 0x7fe1c0147000 2022-07-20T02:33:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe1a8010030 con 0x7fe1c0147000 2022-07-20T02:33:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0153150 con 0x7fe1c0147000 2022-07-20T02:33:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1b7fff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 0x7fe1a4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fe1a806bc50 con 0x7fe1c0147000 2022-07-20T02:33:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1beffd700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 0x7fe1a4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.624+0000 7fe1beffd700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 0x7fe1a4055f30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fe1b000b000 tx=0x7fe1b0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.632+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1a8042ad0 con 0x7fe1c0147000 2022-07-20T02:33:15.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:15.760+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1c0143480 con 0x7fe1c0147000 2022-07-20T02:33:16.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.480+0000 7fe1b7fff700 1 -- 172.21.15.26:0/4131916793 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe1a80460f0 con 0x7fe1c0147000 2022-07-20T02:33:16.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 msgr2=0x7fe1a4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 0x7fe1a4055f30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fe1b000b000 tx=0x7fe1b0006df0).stop 2022-07-20T02:33:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 msgr2=0x7fe1c00b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fe1c0068660 tx=0x7fe1a8007be0).stop 2022-07-20T02:33:16.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 shutdown_connections 2022-07-20T02:33:16.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1c0147000 0x7fe1c00b2ba0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe1a4053a70 0x7fe1a4055f30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1c00b7b10 0x7fe1c00b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 --2- 172.21.15.26:0/4131916793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1c0060f00 0x7fe1c00b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 >> 172.21.15.26:0/4131916793 conn(0x7fe1c005cfd0 msgr2=0x7fe1c0144320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:16.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 shutdown_connections 2022-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.484+0000 7fe1bffff700 1 -- 172.21.15.26:0/4131916793 wait complete. 2022-07-20T02:33:16.500 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:33:16.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:33:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 Processor -- start 2022-07-20T02:33:16.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 -- start start 2022-07-20T02:33:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11d80b5d50 0x7f11d8067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8140320 0x7f11d81407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8146750 con 0x7f11d8143070 2022-07-20T02:33:16.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d81469f0 con 0x7f11d80b5d50 2022-07-20T02:33:16.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11dc856700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d8146c90 con 0x7f11d8140320 2022-07-20T02:33:16.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11d80b5d50 0x7f11d8067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11d80b5d50 0x7f11d8067490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41104/0 (socket says 172.21.15.26:41104) 2022-07-20T02:33:16.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 -- 172.21.15.26:0/689317312 learned_addr learned my addr 172.21.15.26:0/689317312 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:16.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d7fff700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8140320 0x7f11d81407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 -- 172.21.15.26:0/689317312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8140320 msgr2=0x7f11d81407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8140320 0x7f11d81407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 -- 172.21.15.26:0/689317312 >> v1:172.21.15.120:6789/0 conn(0x7f11d8143070 legacy=0x7f11d81434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.612+0000 7f11d77fe700 1 -- 172.21.15.26:0/689317312 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8146f30 con 0x7f11d80b5d50 2022-07-20T02:33:16.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11d77fe700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d8067490 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f11d81472b0 tx=0x7f11d0006eb0).ready entity=mon.0 client_cookie=de057e44fd37432c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11d67fc700 1 -- 172.21.15.26:0/689317312 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11d000d570 con 0x7f11d80b5d50 2022-07-20T02:33:16.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11d67fc700 1 -- 172.21.15.26:0/689317312 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0000e90 con 0x7f11d80b5d50 2022-07-20T02:33:16.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11d67fc700 1 -- 172.21.15.26:0/689317312 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11d000d8c0 con 0x7f11d80b5d50 2022-07-20T02:33:16.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- 172.21.15.26:0/689317312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 msgr2=0x7f11d8067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d8067490 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f11d81472b0 tx=0x7f11d0006eb0).stop 2022-07-20T02:33:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- 172.21.15.26:0/689317312 shutdown_connections 2022-07-20T02:33:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8140320 0x7f11d81407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- 172.21.15.26:0/689317312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d8067490 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- 172.21.15.26:0/689317312 >> 172.21.15.26:0/689317312 conn(0x7f11d8060310 msgr2=0x7f11d80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:16.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- 172.21.15.26:0/689317312 shutdown_connections 2022-07-20T02:33:16.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- 172.21.15.26:0/689317312 wait complete. 2022-07-20T02:33:16.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 Processor -- start 2022-07-20T02:33:16.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- start start 2022-07-20T02:33:16.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 0x7f11d81447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 0x7f11d813b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d81459e0 con 0x7f11d80b5d50 2022-07-20T02:33:16.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d8145c80 con 0x7f11d8140320 2022-07-20T02:33:16.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.616+0000 7f11dc856700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d8145f20 con 0x7f11d8143070 2022-07-20T02:33:16.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41110/0 (socket says 172.21.15.26:41110) 2022-07-20T02:33:16.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 -- 172.21.15.26:0/251606916 learned_addr learned my addr 172.21.15.26:0/251606916 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:16.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d7fff700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 0x7f11d813b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d6ffd700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 0x7f11d81447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 -- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 msgr2=0x7f11d813b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 0x7f11d813b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 -- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 msgr2=0x7f11d81447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 0x7f11d81447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 -- 172.21.15.26:0/251606916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d000b1e0 con 0x7f11d80b5d50 2022-07-20T02:33:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d7fff700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 0x7f11d813b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d6ffd700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 0x7f11d81447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:16.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d77fe700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f11d81472b0 tx=0x7f11d0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11d0011760 con 0x7f11d80b5d50 2022-07-20T02:33:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d00118c0 con 0x7f11d80b5d50 2022-07-20T02:33:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8142ab0 con 0x7f11d80b5d50 2022-07-20T02:33:16.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11d0011c20 con 0x7f11d80b5d50 2022-07-20T02:33:16.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8144e70 con 0x7f11d80b5d50 2022-07-20T02:33:16.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d8140cc0 con 0x7f11d80b5d50 2022-07-20T02:33:16.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.620+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11d0025030 con 0x7f11d80b5d50 2022-07-20T02:33:16.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.624+0000 7f11d4ff9700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 0x7f11bc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.624+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f11d006cd60 con 0x7f11d80b5d50 2022-07-20T02:33:16.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.624+0000 7f11d6ffd700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 0x7f11bc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.624+0000 7f11d6ffd700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 0x7f11bc055e10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f11c000c000 tx=0x7f11c0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.628+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11d0043e00 con 0x7f11d80b5d50 2022-07-20T02:33:16.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.756+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f11d8144e70 con 0x7f11d80b5d50 2022-07-20T02:33:16.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11d4ff9700 1 -- 172.21.15.26:0/251606916 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f11d8144e70 con 0x7f11d80b5d50 2022-07-20T02:33:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 msgr2=0x7f11bc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 0x7f11bc055e10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f11c000c000 tx=0x7f11c0008040).stop 2022-07-20T02:33:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 msgr2=0x7f11d81423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f11d81472b0 tx=0x7f11d0015000).stop 2022-07-20T02:33:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 shutdown_connections 2022-07-20T02:33:16.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f11bc053950 0x7f11bc055e10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11d8143070 0x7f11d813b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11d8140320 0x7f11d81447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 --2- 172.21.15.26:0/251606916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11d80b5d50 0x7f11d81423d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 >> 172.21.15.26:0/251606916 conn(0x7f11d8060310 msgr2=0x7f11d80b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 shutdown_connections 2022-07-20T02:33:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.764+0000 7f11dc856700 1 -- 172.21.15.26:0/251606916 wait complete. 2022-07-20T02:33:16.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-20T02:33:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 Processor -- start 2022-07-20T02:33:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- start start 2022-07-20T02:33:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 0x7fe1600a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe160060f10 0x7fe1600b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b2f10 con 0x7fe160147660 2022-07-20T02:33:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1600a8e30 con 0x7fe160060f10 2022-07-20T02:33:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1600a7310 con 0x7fe1600b2b30 2022-07-20T02:33:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe160060f10 0x7fe1600b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15ed9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe160147660 0x7fe160066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40402/0 (socket says 172.21.15.26:40402) 2022-07-20T02:33:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15ed9d700 1 -- 172.21.15.26:0/1220215215 learned_addr learned my addr 172.21.15.26:0/1220215215 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15e59c700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 0x7fe1600a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 -- 172.21.15.26:0/1220215215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 msgr2=0x7fe1600a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 0x7fe1600a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 -- 172.21.15.26:0/1220215215 >> v1:172.21.15.26:6789/0 conn(0x7fe160147660 legacy=0x7fe160066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 -- 172.21.15.26:0/1220215215 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600a7500 con 0x7fe160060f10 2022-07-20T02:33:16.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15e59c700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 0x7fe1600a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15dd9b700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe160060f10 0x7fe1600b1990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fe150002f70 tx=0x7fe150006df0).ready entity=mon.0 client_cookie=e3b090692af3350a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15d59a700 1 -- 172.21.15.26:0/1220215215 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe15000d570 con 0x7fe160060f10 2022-07-20T02:33:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15d59a700 1 -- 172.21.15.26:0/1220215215 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe150000e90 con 0x7fe160060f10 2022-07-20T02:33:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe15d59a700 1 -- 172.21.15.26:0/1220215215 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe15000d8c0 con 0x7fe160060f10 2022-07-20T02:33:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- 172.21.15.26:0/1220215215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe160060f10 msgr2=0x7fe1600b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe160060f10 0x7fe1600b1990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fe150002f70 tx=0x7fe150006df0).stop 2022-07-20T02:33:16.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- 172.21.15.26:0/1220215215 shutdown_connections 2022-07-20T02:33:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe160060f10 0x7fe1600b1990 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 --2- 172.21.15.26:0/1220215215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1600b2b30 0x7fe1600a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- 172.21.15.26:0/1220215215 >> 172.21.15.26:0/1220215215 conn(0x7fe1600b6110 msgr2=0x7fe1600b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:16.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- 172.21.15.26:0/1220215215 shutdown_connections 2022-07-20T02:33:16.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.892+0000 7fe164cfa700 1 -- 172.21.15.26:0/1220215215 wait complete. 2022-07-20T02:33:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 Processor -- start 2022-07-20T02:33:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- start start 2022-07-20T02:33:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160147660 0x7fe160157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16015af70 con 0x7fe1600b2b30 2022-07-20T02:33:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe160063ba0 con 0x7fe160147660 2022-07-20T02:33:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe160063e40 con 0x7fe160060f10 2022-07-20T02:33:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15e59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41126/0 (socket says 172.21.15.26:41126) 2022-07-20T02:33:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15e59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34408/0 (socket says 172.21.15.26:34408) 2022-07-20T02:33:16.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 -- 172.21.15.26:0/2359831626 learned_addr learned my addr 172.21.15.26:0/2359831626 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:16.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15ed9d700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160147660 0x7fe160157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 -- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 msgr2=0x7fe1600b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 -- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160147660 msgr2=0x7fe160157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:16.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160147660 0x7fe160157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:16.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 -- 172.21.15.26:0/2359831626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15000b180 con 0x7fe1600b2b30 2022-07-20T02:33:16.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15e59c700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:16.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe15dd9b700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fe150002f70 tx=0x7fe150015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe150006b20 con 0x7fe1600b2b30 2022-07-20T02:33:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15000d8f0 con 0x7fe1600b2b30 2022-07-20T02:33:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1600640e0 con 0x7fe1600b2b30 2022-07-20T02:33:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe150011d70 con 0x7fe1600b2b30 2022-07-20T02:33:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.896+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1600651d0 con 0x7fe1600b2b30 2022-07-20T02:33:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe160156d40 con 0x7fe1600b2b30 2022-07-20T02:33:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe150025030 con 0x7fe1600b2b30 2022-07-20T02:33:16.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe14f7fe700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 0x7fe13c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:16.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe15006cf90 con 0x7fe1600b2b30 2022-07-20T02:33:16.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe15e59c700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 0x7fe13c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:16.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.900+0000 7fe15e59c700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 0x7fe13c055fc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fe15400c000 tx=0x7fe154008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:16.904+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe150043f60 con 0x7fe1600b2b30 2022-07-20T02:33:17.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.080+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7fe1600a77a0 con 0x7fe1600b2b30 2022-07-20T02:33:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.484+0000 7fe14f7fe700 1 -- 172.21.15.26:0/2359831626 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v350) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe1500230d0 con 0x7fe1600b2b30 2022-07-20T02:33:17.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:17.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 msgr2=0x7fe13c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 0x7fe13c055fc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fe15400c000 tx=0x7fe154008040).stop 2022-07-20T02:33:17.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 msgr2=0x7fe16015aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fe150002f70 tx=0x7fe150015000).stop 2022-07-20T02:33:17.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 shutdown_connections 2022-07-20T02:33:17.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1600b2b30 0x7fe16015aa80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe13c053b00 0x7fe13c055fc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160147660 0x7fe160157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 --2- 172.21.15.26:0/2359831626 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe160060f10 0x7fe1600b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 >> 172.21.15.26:0/2359831626 conn(0x7fe1600b6110 msgr2=0x7fe1600aa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:17.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 shutdown_connections 2022-07-20T02:33:17.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.488+0000 7fe164cfa700 1 -- 172.21.15.26:0/2359831626 wait complete. 2022-07-20T02:33:17.517 INFO:tasks.cephfs_test_runner:test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:33:17.517 INFO:tasks.cephfs_test_runner:Starting test: test_filtered_df (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:33:17.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-20T02:33:17.537 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:17.543 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:17.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:17.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 Processor -- start 2022-07-20T02:33:17.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 -- start start 2022-07-20T02:33:17.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efe08066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe080b1bc0 con 0x7efe08061050 2022-07-20T02:33:17.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe08147700 con 0x7efe080b2920 2022-07-20T02:33:17.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe0d29f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0814b040 con 0x7efe081484a0 2022-07-20T02:33:17.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe077fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efe08061050 0x7efe08066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40416/0 (socket says 172.21.15.26:40416) 2022-07-20T02:33:17.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe077fe700 1 -- 172.21.15.26:0/3033821257 learned_addr learned my addr 172.21.15.26:0/3033821257 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:17.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efdfffff700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efe08066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525732229 0 0) 0x7efe080b1bc0 con 0x7efe08061050 2022-07-20T02:33:17.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde8003260 con 0x7efe08061050 2022-07-20T02:33:17.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780353809 0 0) 0x7efe08147700 con 0x7efe080b2920 2022-07-20T02:33:17.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe080b1bc0 con 0x7efe080b2920 2022-07-20T02:33:17.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 968886719 0 0) 0x7efde8003260 con 0x7efe08061050 2022-07-20T02:33:17.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 msgr2=0x7efe08066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:17.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efe08066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 >> v1:172.21.15.120:6789/0 conn(0x7efe080b2920 legacy=0x7efe08147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08147930 con 0x7efe08061050 2022-07-20T02:33:17.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7efe000019e0 con 0x7efe08061050 2022-07-20T02:33:17.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 >> v1:172.21.15.26:6789/0 conn(0x7efe08061050 legacy=0x7efe08066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8003910 0x7efde8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efde8004500 0x7efde8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efde8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0814b040 con 0x7efde8003910 2022-07-20T02:33:17.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde800b7b0 con 0x7efde8004500 2022-07-20T02:33:17.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde800b970 con 0x7efe081484a0 2022-07-20T02:33:17.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe06ffd700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8003910 0x7efde8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efdfffff700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efde8004500 0x7efde8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.680+0000 7efe077fe700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efde8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 -- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 msgr2=0x7efde8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:17.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efde8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 -- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efde8004500 msgr2=0x7efde8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:17.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efde8004500 0x7efde8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 -- 172.21.15.26:0/3033821257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde8003260 con 0x7efde8003910 2022-07-20T02:33:17.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe077fe700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe081484a0 0x7efde8004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:17.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe06ffd700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8003910 0x7efde8003d10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7efdf0001110 tx=0x7efdf0007a00).ready entity=mon.0 client_cookie=fc22fb0e4898d275 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:17.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdf0007d10 con 0x7efde8003910 2022-07-20T02:33:17.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0007030 con 0x7efde8003910 2022-07-20T02:33:17.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe067fc700 1 -- 172.21.15.26:0/3033821257 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdf0007390 con 0x7efde8003910 2022-07-20T02:33:17.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- 172.21.15.26:0/3033821257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8003910 msgr2=0x7efde8003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 --2- 172.21.15.26:0/3033821257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8003910 0x7efde8003d10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7efdf0001110 tx=0x7efdf0007a00).stop 2022-07-20T02:33:17.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe077fe700 1 -- 172.21.15.26:0/3033821257 reap_dead start 2022-07-20T02:33:17.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- 172.21.15.26:0/3033821257 shutdown_connections 2022-07-20T02:33:17.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- 172.21.15.26:0/3033821257 >> 172.21.15.26:0/3033821257 conn(0x7efe080b60a0 msgr2=0x7efe080b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:17.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- 172.21.15.26:0/3033821257 shutdown_connections 2022-07-20T02:33:17.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- 172.21.15.26:0/3033821257 wait complete. 2022-07-20T02:33:17.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 Processor -- start 2022-07-20T02:33:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.684+0000 7efe0d29f700 1 -- start start 2022-07-20T02:33:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe080aa930 0x7efe08065ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 0x7efe080663e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe081546c0 con 0x7efe08066db0 2022-07-20T02:33:17.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe080ae330 con 0x7efe080aa930 2022-07-20T02:33:17.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe080aad10 con 0x7efe080af530 2022-07-20T02:33:17.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41148/0 (socket says 172.21.15.26:41148) 2022-07-20T02:33:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 -- 172.21.15.26:0/1938525104 learned_addr learned my addr 172.21.15.26:0/1938525104 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efdfffff700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 0x7efe080663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe06ffd700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe080aa930 0x7efe08065ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 -- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 msgr2=0x7efe080663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 0x7efe080663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 -- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe080aa930 msgr2=0x7efe08065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe080aa930 0x7efe08065ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 -- 172.21.15.26:0/1938525104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080ac220 con 0x7efe08066db0 2022-07-20T02:33:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efdfffff700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 0x7efe080663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe077fe700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7efe00001210 tx=0x7efe08062220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe00010710 con 0x7efe08066db0 2022-07-20T02:33:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe00010870 con 0x7efe08066db0 2022-07-20T02:33:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe080ae560 con 0x7efe08066db0 2022-07-20T02:33:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe00010bd0 con 0x7efe08066db0 2022-07-20T02:33:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe080668d0 con 0x7efe08066db0 2022-07-20T02:33:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efe00010d30 con 0x7efe08066db0 2022-07-20T02:33:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.688+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe080ae6f0 con 0x7efe08066db0 2022-07-20T02:33:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.692+0000 7efe04ff9700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 0x7efdec055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.692+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7efe0006bdb0 con 0x7efe08066db0 2022-07-20T02:33:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.692+0000 7efe06ffd700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 0x7efdec055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.692+0000 7efe06ffd700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 0x7efdec055fc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7efdf0019000 tx=0x7efdf0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.696+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe00042df0 con 0x7efe08066db0 2022-07-20T02:33:17.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:17.864+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7efe0814ada0 con 0x7efe08066db0 2022-07-20T02:33:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.488+0000 7efe04ff9700 1 -- 172.21.15.26:0/1938525104 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v351) v1 ==== 130+0+0 (secure 0 0 0) 0x7efe00046410 con 0x7efe08066db0 2022-07-20T02:33:18.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 msgr2=0x7efdec055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 0x7efdec055fc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7efdf0019000 tx=0x7efdf0016000).stop 2022-07-20T02:33:18.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 msgr2=0x7efe081569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7efe00001210 tx=0x7efe08062220).stop 2022-07-20T02:33:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 shutdown_connections 2022-07-20T02:33:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe08066db0 0x7efe081569b0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efdec053b00 0x7efdec055fc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe080af530 0x7efe080663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 --2- 172.21.15.26:0/1938525104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe080aa930 0x7efe08065ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 >> 172.21.15.26:0/1938525104 conn(0x7efe080b60a0 msgr2=0x7efe0814a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:18.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 shutdown_connections 2022-07-20T02:33:18.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.496+0000 7efe0d29f700 1 -- 172.21.15.26:0/1938525104 wait complete. 2022-07-20T02:33:18.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:33:18.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 Processor -- start 2022-07-20T02:33:18.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 -- start start 2022-07-20T02:33:18.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0ea00b3000 0x7f0ea00b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea009a500 0x7f0ea0143830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea009bd10 con 0x7f0ea0061170 2022-07-20T02:33:18.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea0144430 con 0x7f0ea00b3000 2022-07-20T02:33:18.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.656+0000 7f0ea878e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea01446d0 con 0x7f0ea009a500 2022-07-20T02:33:18.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0ea00b3000 0x7f0ea00b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea6d2b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0ea0061170 0x7f0ea0098fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40436/0 (socket says 172.21.15.26:40436) 2022-07-20T02:33:18.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea6d2b700 1 -- 172.21.15.26:0/3582006124 learned_addr learned my addr 172.21.15.26:0/3582006124 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:18.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea5d29700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea009a500 0x7f0ea0143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 -- 172.21.15.26:0/3582006124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea009a500 msgr2=0x7f0ea0143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea009a500 0x7f0ea0143830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 -- 172.21.15.26:0/3582006124 >> v1:172.21.15.26:6789/0 conn(0x7f0ea0061170 legacy=0x7f0ea0098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 -- 172.21.15.26:0/3582006124 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea0061740 con 0x7f0ea00b3000 2022-07-20T02:33:18.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea5528700 1 -- 172.21.15.26:0/3582006124 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877031156 0 0) 0x7f0ea009bd10 con 0x7f0ea0061170 2022-07-20T02:33:18.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea652a700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea00b3000 0x7f0ea00b33e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f0e9c002f70 tx=0x7f0e9c006df0).ready entity=mon.0 client_cookie=f6f4d48cab8d6736 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:18.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea5528700 1 -- 172.21.15.26:0/3582006124 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9c00d570 con 0x7f0ea00b3000 2022-07-20T02:33:18.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea5528700 1 -- 172.21.15.26:0/3582006124 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c000e90 con 0x7f0ea00b3000 2022-07-20T02:33:18.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea5528700 1 -- 172.21.15.26:0/3582006124 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9c00d8c0 con 0x7f0ea00b3000 2022-07-20T02:33:18.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 -- 172.21.15.26:0/3582006124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea00b3000 msgr2=0x7f0ea00b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea00b3000 0x7f0ea00b33e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f0e9c002f70 tx=0x7f0e9c006df0).stop 2022-07-20T02:33:18.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 -- 172.21.15.26:0/3582006124 shutdown_connections 2022-07-20T02:33:18.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea009a500 0x7f0ea0143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 --2- 172.21.15.26:0/3582006124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea00b3000 0x7f0ea00b33e0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 -- 172.21.15.26:0/3582006124 >> 172.21.15.26:0/3582006124 conn(0x7f0ea00b6920 msgr2=0x7f0ea00b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 -- 172.21.15.26:0/3582006124 shutdown_connections 2022-07-20T02:33:18.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.660+0000 7f0ea878e700 1 -- 172.21.15.26:0/3582006124 wait complete. 2022-07-20T02:33:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 Processor -- start 2022-07-20T02:33:18.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 -- start start 2022-07-20T02:33:18.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0061170 0x7f0ea00b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea00b3000 0x7f0ea00aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0140c20 con 0x7f0ea009a500 2022-07-20T02:33:18.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0066690 con 0x7f0ea0061170 2022-07-20T02:33:18.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea878e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0142230 con 0x7f0ea00b3000 2022-07-20T02:33:18.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea5d29700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea5d29700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41160/0 (socket says 172.21.15.26:41160) 2022-07-20T02:33:18.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.664+0000 7f0ea5d29700 1 -- 172.21.15.26:0/349580651 learned_addr learned my addr 172.21.15.26:0/349580651 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea652a700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0061170 0x7f0ea00b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea6d2b700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea00b3000 0x7f0ea00aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 -- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea00b3000 msgr2=0x7f0ea00aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea00b3000 0x7f0ea00aab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 -- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0061170 msgr2=0x7f0ea00b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0061170 0x7f0ea00b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 -- 172.21.15.26:0/349580651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c00b180 con 0x7f0ea009a500 2022-07-20T02:33:18.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea5d29700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f0e9000c000 tx=0x7f0e90006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:18.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e90010780 con 0x7f0ea009a500 2022-07-20T02:33:18.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea00663f0 con 0x7f0ea009a500 2022-07-20T02:33:18.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e900108e0 con 0x7f0ea009a500 2022-07-20T02:33:18.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea00afbb0 con 0x7f0ea009a500 2022-07-20T02:33:18.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e90010c40 con 0x7f0ea009a500 2022-07-20T02:33:18.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea00619e0 con 0x7f0ea009a500 2022-07-20T02:33:18.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e9001c460 con 0x7f0ea009a500 2022-07-20T02:33:18.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 0x7f0e80055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:18.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0e9006c2d0 con 0x7f0ea009a500 2022-07-20T02:33:18.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea652a700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 0x7f0e80055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:18.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.668+0000 7f0ea652a700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 0x7f0e80055fc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f0e9c01a000 tx=0x7f0e9c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:18.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.676+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e90043270 con 0x7f0ea009a500 2022-07-20T02:33:18.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.808+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0ea00574e0 con 0x7f0ea009a500 2022-07-20T02:33:18.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0e8f7fe700 1 -- 172.21.15.26:0/349580651 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0e90046890 con 0x7f0ea009a500 2022-07-20T02:33:18.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 msgr2=0x7f0e80055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 0x7f0e80055fc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f0e9c01a000 tx=0x7f0e9c014000).stop 2022-07-20T02:33:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 msgr2=0x7f0ea00aa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f0e9000c000 tx=0x7f0e90006df0).stop 2022-07-20T02:33:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 shutdown_connections 2022-07-20T02:33:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ea009a500 0x7f0ea00aa6a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e80053b00 0x7f0e80055fc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea00b3000 0x7f0ea00aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 --2- 172.21.15.26:0/349580651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0061170 0x7f0ea00b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 >> 172.21.15.26:0/349580651 conn(0x7f0ea00b6920 msgr2=0x7f0ea00b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:18.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 shutdown_connections 2022-07-20T02:33:18.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:18.812+0000 7f0ea878e700 1 -- 172.21.15.26:0/349580651 wait complete. 2022-07-20T02:33:18.828 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:33:18.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:18.867 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:18.887 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:18.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:18.947 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:33:18.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:18.949 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:18.950 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:33:18.964 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:33:18.965 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:18.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:18.967 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:18.968 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:33:18.990 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:33:18.991 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:18.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:33:19.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 Processor -- start 2022-07-20T02:33:19.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 -- start start 2022-07-20T02:33:19.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 0x7f246012c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24601468c0 0x7f24600b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2460061e00 con 0x7f2460061a20 2022-07-20T02:33:19.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246012cc40 con 0x7f24601468c0 2022-07-20T02:33:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f2467a80700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2460147040 con 0x7f24601432e0 2022-07-20T02:33:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246581c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2460061a20 0x7f246012c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40446/0 (socket says 172.21.15.26:40446) 2022-07-20T02:33:19.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246581c700 1 -- 172.21.15.26:0/1271070036 learned_addr learned my addr 172.21.15.26:0/1271070036 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:19.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 --2- 172.21.15.26:0/1271070036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 0x7f246012c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246601d700 1 --2- 172.21.15.26:0/1271070036 >> v2:172.21.15.120:3300/0 conn(0x7f24601468c0 0x7f24600b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246481a700 1 -- 172.21.15.26:0/1271070036 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072427698 0 0) 0x7f2460061e00 con 0x7f2460061a20 2022-07-20T02:33:19.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246481a700 1 -- 172.21.15.26:0/1271070036 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244c003260 con 0x7f2460061a20 2022-07-20T02:33:19.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 -- 172.21.15.26:0/1271070036 >> v2:172.21.15.120:3300/0 conn(0x7f24601468c0 msgr2=0x7f24600b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:19.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 --2- 172.21.15.26:0/1271070036 >> v2:172.21.15.120:3300/0 conn(0x7f24601468c0 0x7f24600b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:19.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 -- 172.21.15.26:0/1271070036 >> v1:172.21.15.26:6789/0 conn(0x7f2460061a20 legacy=0x7f246012c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:19.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 -- 172.21.15.26:0/1271070036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24601472e0 con 0x7f24601432e0 2022-07-20T02:33:19.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246501b700 1 --2- 172.21.15.26:0/1271070036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 0x7f246012c750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f24580032f0 tx=0x7f24580091d0).ready entity=mon.2 client_cookie=81ae3ef6269865ac server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:19.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246481a700 1 -- 172.21.15.26:0/1271070036 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800b600 con 0x7f24601432e0 2022-07-20T02:33:19.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246481a700 1 -- 172.21.15.26:0/1271070036 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800b760 con 0x7f24601432e0 2022-07-20T02:33:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.116+0000 7f246481a700 1 -- 172.21.15.26:0/1271070036 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800b8d0 con 0x7f24601432e0 2022-07-20T02:33:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- 172.21.15.26:0/1271070036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 msgr2=0x7f246012c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:19.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 --2- 172.21.15.26:0/1271070036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 0x7f246012c750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f24580032f0 tx=0x7f24580091d0).stop 2022-07-20T02:33:19.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- 172.21.15.26:0/1271070036 shutdown_connections 2022-07-20T02:33:19.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 --2- 172.21.15.26:0/1271070036 >> v2:172.21.15.120:3300/0 conn(0x7f24601468c0 0x7f24600b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:19.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 --2- 172.21.15.26:0/1271070036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601432e0 0x7f246012c750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:19.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- 172.21.15.26:0/1271070036 >> 172.21.15.26:0/1271070036 conn(0x7f246005faf0 msgr2=0x7f2460066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:19.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- 172.21.15.26:0/1271070036 shutdown_connections 2022-07-20T02:33:19.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- 172.21.15.26:0/1271070036 wait complete. 2022-07-20T02:33:19.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 Processor -- start 2022-07-20T02:33:19.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.120+0000 7f2467a80700 1 -- start start 2022-07-20T02:33:19.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2460061a20 0x7f2460129e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601468c0 0x7f24600b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460145770 con 0x7f24601432e0 2022-07-20T02:33:19.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460145d00 con 0x7f2460061a20 2022-07-20T02:33:19.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24600b9820 con 0x7f24601468c0 2022-07-20T02:33:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246581c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2460061a20 0x7f2460129e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41172/0 (socket says 172.21.15.26:41172) 2022-07-20T02:33:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 -- 172.21.15.26:0/3806573551 learned_addr learned my addr 172.21.15.26:0/3806573551 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246601d700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601468c0 0x7f24600b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 -- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601468c0 msgr2=0x7f24600b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601468c0 0x7f24600b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 -- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2460061a20 msgr2=0x7f2460129e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2460061a20 0x7f2460129e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 -- 172.21.15.26:0/3806573551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458009470 con 0x7f24601432e0 2022-07-20T02:33:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f246501b700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f24580032f0 tx=0x7f2458016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800b8d0 con 0x7f24601432e0 2022-07-20T02:33:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245801f3f0 con 0x7f24601432e0 2022-07-20T02:33:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24600b9ac0 con 0x7f24601432e0 2022-07-20T02:33:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245800b600 con 0x7f24601432e0 2022-07-20T02:33:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.124+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24600b9500 con 0x7f24601432e0 2022-07-20T02:33:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460147580 con 0x7f24601432e0 2022-07-20T02:33:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2458026030 con 0x7f24601432e0 2022-07-20T02:33:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f2456ffd700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 0x7f2440055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f245806ca40 con 0x7f24601432e0 2022-07-20T02:33:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f246581c700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 0x7f2440055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.128+0000 7f246581c700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 0x7f2440055ea0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f245c00d000 tx=0x7f245c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.136+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2458043950 con 0x7f24601432e0 2022-07-20T02:33:19.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.268+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f24600574e0 con 0x7f24601432e0 2022-07-20T02:33:19.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.268+0000 7f2456ffd700 1 -- 172.21.15.26:0/3806573551 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v101) v1 ==== 81+0+29269 (secure 0 0 0) 0x7f2458024090 con 0x7f24601432e0 2022-07-20T02:33:19.271 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:19.272 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:19.272 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 101, 2022-07-20T02:33:19.273 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:33:19.273 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:33:19.277 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:33:16.467262+0000", 2022-07-20T02:33:19.277 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:19.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:19.278 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:33:19.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:33:19.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:33:19.280 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:33:19.280 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:33:19.281 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:33:19.282 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:33:19.283 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:19.283 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:33:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:33:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:33:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:33:19.285 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:33:19.285 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 18, 2022-07-20T02:33:19.287 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:33:19.287 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:33:19.287 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:33:19.288 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:33:19.288 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:33:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:33:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:33:19.290 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:33:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:19.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:19.293 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:19.293 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:19.294 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:19.298 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:19.304 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:19.304 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:33:19.305 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:33:19.305 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:33:19.305 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:33:19.306 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:33:19.306 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:19.307 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:19.308 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:19.309 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:19.309 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:19.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.310 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:33:19.310 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:19.311 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:19.311 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:19.311 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:19.312 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:19.312 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:19.315 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:19.315 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:19.316 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:19.316 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:19.317 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:19.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:19.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:19.318 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:19.318 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:19.319 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:19.320 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:19.320 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:19.320 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:19.324 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:19.326 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:19.327 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:19.327 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:19.328 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:19.328 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:19.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:19.330 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:19.331 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:19.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:19.333 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:19.334 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:19.334 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:19.336 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:19.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:33:19.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.338 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:19.338 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:33:19.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:33:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:33:19.341 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:33:19.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:19.343 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:19.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:19.344 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:19.344 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:19.345 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:19.345 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:19.346 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:19.346 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:19.347 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:19.347 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:19.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:19.348 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:19.349 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:19.349 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:19.349 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:19.350 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:19.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:19.351 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:19.351 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:19.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.352 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:33:19.353 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:19.353 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:19.354 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:19.355 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:19.355 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:19.356 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:19.356 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:19.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:19.357 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:19.358 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:19.358 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:19.359 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:19.359 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:19.360 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:19.360 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:19.361 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:19.361 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:19.362 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:19.362 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:19.363 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:19.364 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:19.365 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:19.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:19.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.367 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:19.367 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:19.368 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:19.368 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:19.369 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:19.370 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:19.370 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:19.370 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:19.371 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:33:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.373 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:19.373 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:33:19.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.374 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:19.374 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:33:19.375 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.375 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.376 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.376 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.377 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.377 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:33:19.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:33:19.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.387 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:33:19.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:33:19.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:33:19.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:33:19.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.410 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.411 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:33:19.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:33:19.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:33:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:33:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:33:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:33:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:33:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.464 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:33:19.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:33:19.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.473 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:33:19.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:33:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.483 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:33:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:33:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.490 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.491 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:33:19.492 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:33:19.494 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:19.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:33:19.505 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:33:19.506 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:33:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:33:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.507 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.508 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.510 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:19.510 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:33:19.511 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.511 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.512 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.513 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.526 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:19.528 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.529 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:33:19.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:33:19.534 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.536 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:33:19.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:33:19.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:33:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:33:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:33:19.561 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:33:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:19.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:33:19.566 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:33:19.567 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:33:19.567 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:33:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:19.571 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:33:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.572 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.573 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:19.574 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:33:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:33:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.584 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.586 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:33:19.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:33:19.589 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.591 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.591 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:33:19.594 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:33:19.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.611 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:33:19.783 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:33:19.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:19.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.787 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:33:19.787 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:33:19.788 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:33:19.788 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:33:19.789 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.789 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.790 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:19.792 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:33:19.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.793 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.794 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.794 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.795 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.795 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.795 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.796 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 95, 2022-07-20T02:33:19.796 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.796 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.797 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:33:19.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:33:19.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.803 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:33:19.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:33:19.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.814 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:33:19.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:33:19.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.819 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:33:19.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:33:19.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:19.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:33:19.823 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:33:19.823 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:33:19.823 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:33:19.823 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.824 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.824 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.825 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:19.825 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:33:19.825 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.826 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.826 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.826 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.827 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.827 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.827 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 85, 2022-07-20T02:33:19.827 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.828 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:33:19.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:33:19.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.835 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:33:19.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:33:19.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.841 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:33:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:33:19.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.847 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:33:19.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:33:19.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:19.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:33:19.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:33:19.864 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:33:19.865 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:33:19.865 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.865 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.865 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.867 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:19.867 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:33:19.868 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.868 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.868 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.869 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.869 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.870 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.870 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.871 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 95, 2022-07-20T02:33:19.871 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.871 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:33:19.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:33:19.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:33:19.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:33:19.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.886 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:33:19.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:33:19.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.890 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:33:19.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:33:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:33:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:33:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:33:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:33:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:33:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 96, 2022-07-20T02:33:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:33:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:33:19.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.910 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:19.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:33:19.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:33:19.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.917 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:19.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:33:19.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:33:19.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.932 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:19.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:19.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:19.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:33:19.933 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:19.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:33:19.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:19.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:33:19.937 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:33:19.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:33:19.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:33:19.938 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:19.940 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:19.940 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:19.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:19.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.941 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:19.941 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:33:19.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:19.942 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.942 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.945 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.946 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.946 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:33:19.947 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:19.952 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.952 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.953 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.953 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.954 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.954 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:33:19.955 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:19.957 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.958 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.959 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:33:19.964 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:19.966 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.967 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.967 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.968 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.968 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:33:19.969 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:19.971 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.971 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.972 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.972 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.972 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.973 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:33:19.973 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:19.976 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.976 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.977 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.977 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.978 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.978 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:33:19.978 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:19.980 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.981 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.983 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.983 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.984 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.984 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:33:19.984 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:19.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:19.986 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:19.987 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:19.988 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:19.988 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:19.989 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:33:19.989 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:19.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:19.994 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:19.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:33:19.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:33:19.996 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:33:19.996 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:33:19.996 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:33:19.997 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/2339237946": "2022-07-21T02:33:12.864462+0000", 2022-07-20T02:33:19.997 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/2339237946": "2022-07-21T02:33:12.864462+0000" 2022-07-20T02:33:19.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:19.998 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:33:19.998 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:33:19.999 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:33:19.999 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:19.999 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:33:20.000 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:33:20.000 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:20.001 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:33:20.001 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:20.002 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:20.003 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:33:20.003 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:33:20.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:20.004 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:33:20.005 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:20.005 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:20.006 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:33:20.006 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:20.006 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:20.007 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:33:20.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:20.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:20.008 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:33:20.009 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:33:20.009 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:33:20.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:33:20.010 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:33:20.011 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:20.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 msgr2=0x7f2440055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 0x7f2440055ea0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f245c00d000 tx=0x7f245c009040).stop 2022-07-20T02:33:20.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 msgr2=0x7f24600b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f24580032f0 tx=0x7f2458016000).stop 2022-07-20T02:33:20.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 shutdown_connections 2022-07-20T02:33:20.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f24601432e0 0x7f24600b8110 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f24400539e0 0x7f2440055ea0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24601468c0 0x7f24600b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 --2- 172.21.15.26:0/3806573551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2460061a20 0x7f2460129e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 >> 172.21.15.26:0/3806573551 conn(0x7f246005faf0 msgr2=0x7f24600b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 shutdown_connections 2022-07-20T02:33:20.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:19.272+0000 7f2467a80700 1 -- 172.21.15.26:0/3806573551 wait complete. 2022-07-20T02:33:20.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:20.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 Processor -- start 2022-07-20T02:33:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- start start 2022-07-20T02:33:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1ce81484a0 0x7f1ce8066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b1bc0 con 0x7f1ce80b2920 2022-07-20T02:33:20.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8147700 con 0x7f1ce8061050 2022-07-20T02:33:20.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce814b040 con 0x7f1ce81484a0 2022-07-20T02:33:20.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ceccb8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1ce81484a0 0x7f1ce8066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ced4b9700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1ce80b2920 0x7f1ce8147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40458/0 (socket says 172.21.15.26:40458) 2022-07-20T02:33:20.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ced4b9700 1 -- 172.21.15.26:0/711448615 learned_addr learned my addr 172.21.15.26:0/711448615 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:20.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ceccb8700 1 -- 172.21.15.26:0/711448615 >> v1:172.21.15.120:6789/0 conn(0x7f1ce8061050 legacy=0x7f1ce8066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ceccb8700 1 -- 172.21.15.26:0/711448615 >> v1:172.21.15.26:6789/0 conn(0x7f1ce80b2920 legacy=0x7f1ce8147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ceccb8700 1 -- 172.21.15.26:0/711448615 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8147930 con 0x7f1ce81484a0 2022-07-20T02:33:20.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1ceccb8700 1 --2- 172.21.15.26:0/711448615 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce81484a0 0x7f1ce8066a80 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f1ce40032f0 tx=0x7f1ce40091d0).ready entity=mon.0 client_cookie=16a9bd06e8cb91ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cdbfff700 1 -- 172.21.15.26:0/711448615 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ce4009ea0 con 0x7f1ce81484a0 2022-07-20T02:33:20.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cdbfff700 1 -- 172.21.15.26:0/711448615 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce400b8a0 con 0x7f1ce81484a0 2022-07-20T02:33:20.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cdbfff700 1 -- 172.21.15.26:0/711448615 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ce400ba10 con 0x7f1ce81484a0 2022-07-20T02:33:20.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- 172.21.15.26:0/711448615 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce81484a0 msgr2=0x7f1ce8066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 --2- 172.21.15.26:0/711448615 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce81484a0 0x7f1ce8066a80 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f1ce40032f0 tx=0x7f1ce40091d0).stop 2022-07-20T02:33:20.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- 172.21.15.26:0/711448615 shutdown_connections 2022-07-20T02:33:20.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 --2- 172.21.15.26:0/711448615 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce81484a0 0x7f1ce8066a80 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.148+0000 7f1cef71d700 1 -- 172.21.15.26:0/711448615 >> 172.21.15.26:0/711448615 conn(0x7f1ce80b60a0 msgr2=0x7f1ce80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- 172.21.15.26:0/711448615 shutdown_connections 2022-07-20T02:33:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- 172.21.15.26:0/711448615 wait complete. 2022-07-20T02:33:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 Processor -- start 2022-07-20T02:33:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- start start 2022-07-20T02:33:20.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ce8061050 0x7f1ce80ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ce81484a0 0x7f1ce8063a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80a9aa0 con 0x7f1ce80b2920 2022-07-20T02:33:20.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80a9d40 con 0x7f1ce81484a0 2022-07-20T02:33:20.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80a9fe0 con 0x7f1ce8061050 2022-07-20T02:33:20.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41186/0 (socket says 172.21.15.26:41186) 2022-07-20T02:33:20.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 -- 172.21.15.26:0/3068009425 learned_addr learned my addr 172.21.15.26:0/3068009425 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:20.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ced4b9700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ce8061050 0x7f1ce80ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cedcba700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ce81484a0 0x7f1ce8063a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 -- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ce8061050 msgr2=0x7f1ce80ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ce8061050 0x7f1ce80ab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 -- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ce81484a0 msgr2=0x7f1ce8063a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ce81484a0 0x7f1ce8063a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 -- 172.21.15.26:0/3068009425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4009470 con 0x7f1ce80b2920 2022-07-20T02:33:20.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1ceccb8700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f1ce40032f0 tx=0x7f1ce4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ce4009ea0 con 0x7f1ce80b2920 2022-07-20T02:33:20.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce401f3f0 con 0x7f1ce80b2920 2022-07-20T02:33:20.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce80aa280 con 0x7f1ce80b2920 2022-07-20T02:33:20.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ce401f750 con 0x7f1ce80b2920 2022-07-20T02:33:20.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.152+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce80aa720 con 0x7f1ce80b2920 2022-07-20T02:33:20.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce80aa410 con 0x7f1ce80b2920 2022-07-20T02:33:20.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ce400f070 con 0x7f1ce80b2920 2022-07-20T02:33:20.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1cda7fc700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 0x7f1cd0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1ce406c6d0 con 0x7f1ce80b2920 2022-07-20T02:33:20.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1ced4b9700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 0x7f1cd0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.156+0000 7f1ced4b9700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 0x7f1cd0055f30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f1cdc00d000 tx=0x7f1cdc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.168+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ce40436d0 con 0x7f1ce80b2920 2022-07-20T02:33:20.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.328+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce814ada0 con 0x7f1ce80b2920 2022-07-20T02:33:20.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cda7fc700 1 -- 172.21.15.26:0/3068009425 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1565 (secure 0 0 0) 0x7f1ce4024090 con 0x7f1ce80b2920 2022-07-20T02:33:20.334 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:20.334 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/3730411156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3730411156},{"type":"v1","addr":"172.21.15.26:6837","nonce":3730411156}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":50}],"filesystems":[]} 2022-07-20T02:33:20.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 msgr2=0x7f1cd0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 0x7f1cd0055f30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f1cdc00d000 tx=0x7f1cdc009040).stop 2022-07-20T02:33:20.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 msgr2=0x7f1ce80ab610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f1ce40032f0 tx=0x7f1ce4016000).stop 2022-07-20T02:33:20.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 shutdown_connections 2022-07-20T02:33:20.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ce80b2920 0x7f1ce80ab610 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1cd0053a70 0x7f1cd0055f30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ce81484a0 0x7f1ce8063a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 --2- 172.21.15.26:0/3068009425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ce8061050 0x7f1ce80ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 >> 172.21.15.26:0/3068009425 conn(0x7f1ce80b60a0 msgr2=0x7f1ce814b180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 shutdown_connections 2022-07-20T02:33:20.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.332+0000 7f1cef71d700 1 -- 172.21.15.26:0/3068009425 wait complete. 2022-07-20T02:33:20.339 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 50 2022-07-20T02:33:20.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:20.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 Processor -- start 2022-07-20T02:33:20.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 -- start start 2022-07-20T02:33:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feb18147000 0x7feb18143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 0x7feb1813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb181474e0 con 0x7feb1805cfd0 2022-07-20T02:33:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb180b4e40 con 0x7feb18147000 2022-07-20T02:33:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1e02f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb1805d3b0 con 0x7feb18060f00 2022-07-20T02:33:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7feb18147000 0x7feb18143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1d02d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb1805cfd0 0x7feb180b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40472/0 (socket says 172.21.15.26:40472) 2022-07-20T02:33:20.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1d02d700 1 -- 172.21.15.26:0/1868606659 learned_addr learned my addr 172.21.15.26:0/1868606659 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:20.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1d82e700 1 --2- 172.21.15.26:0/1868606659 >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 0x7feb1813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 -- 172.21.15.26:0/1868606659 >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 msgr2=0x7feb1813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:20.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 --2- 172.21.15.26:0/1868606659 >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 0x7feb1813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 -- 172.21.15.26:0/1868606659 >> v1:172.21.15.26:6789/0 conn(0x7feb1805cfd0 legacy=0x7feb180b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb07fff700 1 -- 172.21.15.26:0/1868606659 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282167581 0 0) 0x7feb181474e0 con 0x7feb1805cfd0 2022-07-20T02:33:20.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 -- 172.21.15.26:0/1868606659 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb181431e0 con 0x7feb18147000 2022-07-20T02:33:20.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1d82e700 1 --2- 172.21.15.26:0/1868606659 >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 0x7feb1813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:20.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb1c82c700 1 --2- 172.21.15.26:0/1868606659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18143bc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7feb08002f70 tx=0x7feb08006df0).ready entity=mon.0 client_cookie=f9c7863e4d629bea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb07fff700 1 -- 172.21.15.26:0/1868606659 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0800d5c0 con 0x7feb18147000 2022-07-20T02:33:20.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb07fff700 1 -- 172.21.15.26:0/1868606659 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08000e90 con 0x7feb18147000 2022-07-20T02:33:20.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.452+0000 7feb07fff700 1 -- 172.21.15.26:0/1868606659 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0800d910 con 0x7feb18147000 2022-07-20T02:33:20.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- 172.21.15.26:0/1868606659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 msgr2=0x7feb18143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- 172.21.15.26:0/1868606659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18143bc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7feb08002f70 tx=0x7feb08006df0).stop 2022-07-20T02:33:20.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- 172.21.15.26:0/1868606659 shutdown_connections 2022-07-20T02:33:20.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- 172.21.15.26:0/1868606659 >> v2:172.21.15.120:3300/0 conn(0x7feb18060f00 0x7feb1813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- 172.21.15.26:0/1868606659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18143bc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- 172.21.15.26:0/1868606659 >> 172.21.15.26:0/1868606659 conn(0x7feb180620c0 msgr2=0x7feb1805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- 172.21.15.26:0/1868606659 shutdown_connections 2022-07-20T02:33:20.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- 172.21.15.26:0/1868606659 wait complete. 2022-07-20T02:33:20.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 Processor -- start 2022-07-20T02:33:20.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- start start 2022-07-20T02:33:20.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 0x7feb18152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 0x7feb18155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18153210 con 0x7feb18147000 2022-07-20T02:33:20.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb180b7520 con 0x7feb18060f00 2022-07-20T02:33:20.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1e02f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb180b77c0 con 0x7feb180b03a0 2022-07-20T02:33:20.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41200/0 (socket says 172.21.15.26:41200) 2022-07-20T02:33:20.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 -- 172.21.15.26:0/4138440695 learned_addr learned my addr 172.21.15.26:0/4138440695 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:20.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1d02d700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 0x7feb18152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1d82e700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 0x7feb18155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 -- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 msgr2=0x7feb18155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 0x7feb18155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 -- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 msgr2=0x7feb18152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.456+0000 7feb1c82c700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 0x7feb18152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1c82c700 1 -- 172.21.15.26:0/4138440695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0800b180 con 0x7feb18147000 2022-07-20T02:33:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1d82e700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 0x7feb18155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1d02d700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 0x7feb18152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1c82c700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7feb08002f70 tx=0x7feb08014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0801e400 con 0x7feb18147000 2022-07-20T02:33:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0801e560 con 0x7feb18147000 2022-07-20T02:33:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb18142520 con 0x7feb18147000 2022-07-20T02:33:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0801e8c0 con 0x7feb18147000 2022-07-20T02:33:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb18155d00 con 0x7feb18147000 2022-07-20T02:33:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb18145c00 con 0x7feb18147000 2022-07-20T02:33:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feb0801c070 con 0x7feb18147000 2022-07-20T02:33:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 0x7feaf8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.460+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5686+0+0 (secure 0 0 0) 0x7feb0806c8f0 con 0x7feb18147000 2022-07-20T02:33:20.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.464+0000 7feb1d02d700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 0x7feaf8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.464+0000 7feb1d02d700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 0x7feaf8055f60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7feb1400b000 tx=0x7feb14009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.468+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb08043990 con 0x7feb18147000 2022-07-20T02:33:20.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.632+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb180b7980 con 0x7feb18147000 2022-07-20T02:33:20.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb067fc700 1 -- 172.21.15.26:0/4138440695 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1565 (secure 0 0 0) 0x7feb08023070 con 0x7feb18147000 2022-07-20T02:33:20.640 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:20.640 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4116312481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4116312481},{"type":"v1","addr":"172.21.15.120:6833","nonce":4116312481}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/3495118646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":3495118646},{"type":"v1","addr":"172.21.15.26:6835","nonce":3495118646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46},{"gid":7084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/3730411156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3730411156},{"type":"v1","addr":"172.21.15.26:6837","nonce":3730411156}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":50}],"filesystems":[]} 2022-07-20T02:33:20.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 msgr2=0x7feaf8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 0x7feaf8055f60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7feb1400b000 tx=0x7feb14009040).stop 2022-07-20T02:33:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 msgr2=0x7feb18142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7feb08002f70 tx=0x7feb08014000).stop 2022-07-20T02:33:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 shutdown_connections 2022-07-20T02:33:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feb18147000 0x7feb18142030 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feaf8053aa0 0x7feaf8055f60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb180b03a0 0x7feb18155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 --2- 172.21.15.26:0/4138440695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb18060f00 0x7feb18152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 >> 172.21.15.26:0/4138440695 conn(0x7feb1805cfd0 msgr2=0x7feb18149ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 shutdown_connections 2022-07-20T02:33:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.636+0000 7feb1e02f700 1 -- 172.21.15.26:0/4138440695 wait complete. 2022-07-20T02:33:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 50 2022-07-20T02:33:20.651 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:33:20.651 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:33:20.653 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:33:20.653 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:20.654 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:33:20.654 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:33:20.655 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:33:20.655 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:20.655 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:33:20.656 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:33:20.656 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:33:20.657 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:20.676 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:33:20.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:33:20.677 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:33:20.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:33:20.679 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:33:20.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:33:20.686 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:33:20.687 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:20.721 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:33:20.722 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:20.723 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:33:20.723 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:20.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:33:20.759 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:33:20.799 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:33:20.807 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:33:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 Processor -- start 2022-07-20T02:33:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- start start 2022-07-20T02:33:20.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f89c4147d70 0x7f89c40b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c40b2f10 con 0x7f89c4060f10 2022-07-20T02:33:20.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c40a8e30 con 0x7f89c4147d70 2022-07-20T02:33:20.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40a7310 con 0x7f89c40b2b30 2022-07-20T02:33:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34514/0 (socket says 172.21.15.26:34514) 2022-07-20T02:33:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 -- 172.21.15.26:0/1672852218 learned_addr learned my addr 172.21.15.26:0/1672852218 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9d61700 1 --2- 172.21.15.26:0/1672852218 >> v2:172.21.15.120:3300/0 conn(0x7f89c4147d70 0x7f89c40b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 -- 172.21.15.26:0/1672852218 >> v2:172.21.15.120:3300/0 conn(0x7f89c4147d70 msgr2=0x7f89c40b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:20.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 --2- 172.21.15.26:0/1672852218 >> v2:172.21.15.120:3300/0 conn(0x7f89c4147d70 0x7f89c40b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 -- 172.21.15.26:0/1672852218 >> v1:172.21.15.120:6789/0 conn(0x7f89c4060f10 legacy=0x7f89c4147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 -- 172.21.15.26:0/1672852218 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c40a7500 con 0x7f89c40b2b30 2022-07-20T02:33:20.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89c9560700 1 --2- 172.21.15.26:0/1672852218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f89c0002f70 tx=0x7f89c0006eb0).ready entity=mon.2 client_cookie=45f77c465e230c99 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89bbfff700 1 -- 172.21.15.26:0/1672852218 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c000d570 con 0x7f89c40b2b30 2022-07-20T02:33:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89bbfff700 1 -- 172.21.15.26:0/1672852218 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0000e90 con 0x7f89c40b2b30 2022-07-20T02:33:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89bbfff700 1 -- 172.21.15.26:0/1672852218 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c000d900 con 0x7f89c40b2b30 2022-07-20T02:33:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1672852218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 msgr2=0x7f89c40a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1672852218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f89c0002f70 tx=0x7f89c0006eb0).stop 2022-07-20T02:33:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1672852218 shutdown_connections 2022-07-20T02:33:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1672852218 >> v2:172.21.15.120:3300/0 conn(0x7f89c4147d70 0x7f89c40b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1672852218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c40b2b30 0x7f89c40a7ee0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1672852218 >> 172.21.15.26:0/1672852218 conn(0x7f89c40b6110 msgr2=0x7f89c40b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1672852218 shutdown_connections 2022-07-20T02:33:20.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1672852218 wait complete. 2022-07-20T02:33:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 Processor -- start 2022-07-20T02:33:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- start start 2022-07-20T02:33:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 0x7f89c40ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c40b2b30 0x7f89c4143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40633d0 con 0x7f89c4147d70 2022-07-20T02:33:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c4063b00 con 0x7f89c40b2b30 2022-07-20T02:33:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c4156890 con 0x7f89c4060f10 2022-07-20T02:33:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9560700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 0x7f89c40ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c8d5f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c40b2b30 0x7f89c4143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9560700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 0x7f89c40ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34520/0 (socket says 172.21.15.26:34520) 2022-07-20T02:33:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41240/0 (socket says 172.21.15.26:41240) 2022-07-20T02:33:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9560700 1 -- 172.21.15.26:0/1540057992 learned_addr learned my addr 172.21.15.26:0/1540057992 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 -- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 msgr2=0x7f89c40ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 0x7f89c40ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 -- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c40b2b30 msgr2=0x7f89c4143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c40b2b30 0x7f89c4143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 -- 172.21.15.26:0/1540057992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c000b1e0 con 0x7f89c4147d70 2022-07-20T02:33:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89c9d61700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f89c4062c70 tx=0x7f89bc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89bc012600 con 0x7f89c4147d70 2022-07-20T02:33:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc012760 con 0x7f89c4147d70 2022-07-20T02:33:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c40635c0 con 0x7f89c4147d70 2022-07-20T02:33:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89bc012ac0 con 0x7f89c4147d70 2022-07-20T02:33:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.836+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c4144310 con 0x7f89c4147d70 2022-07-20T02:33:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c40b0690 con 0x7f89c4147d70 2022-07-20T02:33:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f89bc022070 con 0x7f89c4147d70 2022-07-20T02:33:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89ba7fc700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 0x7f89ac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f89bc06c260 con 0x7f89c4147d70 2022-07-20T02:33:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89c9560700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 0x7f89ac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.840+0000 7f89c9560700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 0x7f89ac055f30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f89c001c000 tx=0x7f89c0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.844+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89bc0432a0 con 0x7f89c4147d70 2022-07-20T02:33:20.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:20.984+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f89c40a77a0 con 0x7f89c4147d70 2022-07-20T02:33:21.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.828+0000 7f89ba7fc700 1 -- 172.21.15.26:0/1540057992 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f89bc016080 con 0x7f89c4147d70 2022-07-20T02:33:21.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 msgr2=0x7f89ac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:21.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 0x7f89ac055f30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f89c001c000 tx=0x7f89c0015000).stop 2022-07-20T02:33:21.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 msgr2=0x7f89c4143e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:21.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f89c4062c70 tx=0x7f89bc0091d0).stop 2022-07-20T02:33:21.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 shutdown_connections 2022-07-20T02:33:21.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f89c4147d70 0x7f89c4143e20 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:21.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f89ac053a70 0x7f89ac055f30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:21.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c40b2b30 0x7f89c4143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:21.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 --2- 172.21.15.26:0/1540057992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4060f10 0x7f89c40ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:21.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 >> 172.21.15.26:0/1540057992 conn(0x7f89c40b6110 msgr2=0x7f89c4147660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:21.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 shutdown_connections 2022-07-20T02:33:21.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.832+0000 7f89cb7c4700 1 -- 172.21.15.26:0/1540057992 wait complete. 2022-07-20T02:33:21.842 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:33:21.850 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:33:21.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:33:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 Processor -- start 2022-07-20T02:33:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 -- start start 2022-07-20T02:33:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e540b5ef0 0x7f5e541474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e540a8c40 0x7f5e540af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e540b62d0 con 0x7f5e540a7fc0 2022-07-20T02:33:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5414aac0 con 0x7f5e540b5ef0 2022-07-20T02:33:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e58827700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5414ad60 con 0x7f5e540a8c40 2022-07-20T02:33:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e52d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e540b5ef0 0x7f5e541474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e5259c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f5e540a7fc0 0x7f5e540aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:54738/0 (socket says 172.21.15.26:54738) 2022-07-20T02:33:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e52d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e540b5ef0 0x7f5e541474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41248/0 (socket says 172.21.15.26:41248) 2022-07-20T02:33:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.960+0000 7f5e5259c700 1 -- 172.21.15.26:0/1427822855 learned_addr learned my addr 172.21.15.26:0/1427822855 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e5359e700 1 --2- 172.21.15.26:0/1427822855 >> v2:172.21.15.120:3300/0 conn(0x7f5e540a8c40 0x7f5e540af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e51d9b700 1 -- 172.21.15.26:0/1427822855 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560707707 0 0) 0x7f5e540b62d0 con 0x7f5e540a7fc0 2022-07-20T02:33:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e51d9b700 1 -- 172.21.15.26:0/1427822855 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e38003260 con 0x7f5e540a7fc0 2022-07-20T02:33:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e52d9d700 1 -- 172.21.15.26:0/1427822855 >> v2:172.21.15.120:3300/0 conn(0x7f5e540a8c40 msgr2=0x7f5e540af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e52d9d700 1 --2- 172.21.15.26:0/1427822855 >> v2:172.21.15.120:3300/0 conn(0x7f5e540a8c40 0x7f5e540af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e52d9d700 1 -- 172.21.15.26:0/1427822855 >> v1:172.21.15.120:6789/0 conn(0x7f5e540a7fc0 legacy=0x7f5e540aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e52d9d700 1 -- 172.21.15.26:0/1427822855 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5414b000 con 0x7f5e540b5ef0 2022-07-20T02:33:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e52d9d700 1 --2- 172.21.15.26:0/1427822855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540b5ef0 0x7f5e541474d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f5e48002f70 tx=0x7f5e48006df0).ready entity=mon.0 client_cookie=2d315e8ad3a726d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e51d9b700 1 -- 172.21.15.26:0/1427822855 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e4800d4f0 con 0x7f5e540b5ef0 2022-07-20T02:33:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e51d9b700 1 -- 172.21.15.26:0/1427822855 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e48000e90 con 0x7f5e540b5ef0 2022-07-20T02:33:22.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e51d9b700 1 -- 172.21.15.26:0/1427822855 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e4800d840 con 0x7f5e540b5ef0 2022-07-20T02:33:22.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 -- 172.21.15.26:0/1427822855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540b5ef0 msgr2=0x7f5e541474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 --2- 172.21.15.26:0/1427822855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540b5ef0 0x7f5e541474d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f5e48002f70 tx=0x7f5e48006df0).stop 2022-07-20T02:33:22.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 -- 172.21.15.26:0/1427822855 shutdown_connections 2022-07-20T02:33:22.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 --2- 172.21.15.26:0/1427822855 >> v2:172.21.15.120:3300/0 conn(0x7f5e540a8c40 0x7f5e540af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 --2- 172.21.15.26:0/1427822855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540b5ef0 0x7f5e541474d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 -- 172.21.15.26:0/1427822855 >> 172.21.15.26:0/1427822855 conn(0x7f5e54060310 msgr2=0x7f5e540b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 -- 172.21.15.26:0/1427822855 shutdown_connections 2022-07-20T02:33:22.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.964+0000 7f5e58827700 1 -- 172.21.15.26:0/1427822855 wait complete. 2022-07-20T02:33:22.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 Processor -- start 2022-07-20T02:33:22.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- start start 2022-07-20T02:33:22.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e540a7fc0 0x7f5e540aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e540b5ef0 0x7f5e54156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e541566e0 con 0x7f5e540a8c40 2022-07-20T02:33:22.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e540af260 con 0x7f5e540a7fc0 2022-07-20T02:33:22.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e540aefc0 con 0x7f5e540b5ef0 2022-07-20T02:33:22.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41254/0 (socket says 172.21.15.26:41254) 2022-07-20T02:33:22.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 -- 172.21.15.26:0/2266942543 learned_addr learned my addr 172.21.15.26:0/2266942543 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:22.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e52d9d700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e540a7fc0 0x7f5e540aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5359e700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e540b5ef0 0x7f5e54156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 -- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e540b5ef0 msgr2=0x7f5e54156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e540b5ef0 0x7f5e54156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 -- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e540a7fc0 msgr2=0x7f5e540aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e540a7fc0 0x7f5e540aae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 -- 172.21.15.26:0/2266942543 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4800b180 con 0x7f5e540a8c40 2022-07-20T02:33:22.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e5259c700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f5e44001490 tx=0x7f5e44007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e44005ba0 con 0x7f5e540a8c40 2022-07-20T02:33:22.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44005d00 con 0x7f5e540a8c40 2022-07-20T02:33:22.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e54156940 con 0x7f5e540a8c40 2022-07-20T02:33:22.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e440072c0 con 0x7f5e540a8c40 2022-07-20T02:33:22.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.968+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e541572d0 con 0x7f5e540a8c40 2022-07-20T02:33:22.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.972+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e54156ad0 con 0x7f5e540a8c40 2022-07-20T02:33:22.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.972+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e44010070 con 0x7f5e540a8c40 2022-07-20T02:33:22.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.972+0000 7f5e43fff700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 0x7f5e30055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.972+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f5e4406b7c0 con 0x7f5e540a8c40 2022-07-20T02:33:22.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.972+0000 7f5e52d9d700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 0x7f5e30055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.976+0000 7f5e52d9d700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 0x7f5e30055fc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f5e4801a000 tx=0x7f5e48014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:21.984+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e440428c0 con 0x7f5e540a8c40 2022-07-20T02:33:22.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.148+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5e5414b2a0 con 0x7f5e540a8c40 2022-07-20T02:33:22.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.148+0000 7f5e43fff700 1 -- 172.21.15.26:0/2266942543 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f5e44045ee0 con 0x7f5e540a8c40 2022-07-20T02:33:22.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:22.164 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:22.164 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:33:22.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.165 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:33:22.165 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:33:22.171 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:22.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.173 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.176 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:33:22.176 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:33:22.176 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.177 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:22.177 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.178 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.178 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.181 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:33:22.182 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:33:22.182 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:22.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.184 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.192 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:33:22.193 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:33:22.193 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.193 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.194 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.194 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.195 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:33:22.195 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:33:22.195 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.196 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.196 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.196 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.197 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:33:22.197 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:33:22.198 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.198 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.200 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:33:22.200 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:33:22.200 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.200 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.201 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.201 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.202 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:33:22.202 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:33:22.203 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.203 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.203 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.203 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.204 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:33:22.205 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:33:22.205 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.205 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.205 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.206 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.208 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:33:22.208 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:33:22.208 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.208 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.209 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.209 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.210 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:33:22.210 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:33:22.210 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.211 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.212 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.214 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:33:22.214 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:33:22.215 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.215 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:22.215 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:33:22.215 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:33:22.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.216 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:33:22.217 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:33:22.226 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.227 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:22.227 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:33:22.228 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:33:22.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.231 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:33:22.232 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:33:22.232 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.233 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:22.233 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:22.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:22.236 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.238 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:33:22.239 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:33:22.239 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:22.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:33:22.241 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:22.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:22.243 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:33:22.243 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:33:22.244 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:22.244 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:22.245 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:33:22.245 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:22.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:22.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:22.249 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:22.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 msgr2=0x7f5e30055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 0x7f5e30055fc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f5e4801a000 tx=0x7f5e48014000).stop 2022-07-20T02:33:22.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 msgr2=0x7f5e540a92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f5e44001490 tx=0x7f5e44007e40).stop 2022-07-20T02:33:22.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 shutdown_connections 2022-07-20T02:33:22.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e540a8c40 0x7f5e540a92b0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e30053b00 0x7f5e30055fc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e540b5ef0 0x7f5e54156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 --2- 172.21.15.26:0/2266942543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e540a7fc0 0x7f5e540aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 >> 172.21.15.26:0/2266942543 conn(0x7f5e54060310 msgr2=0x7f5e54067950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 shutdown_connections 2022-07-20T02:33:22.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.152+0000 7f5e58827700 1 -- 172.21.15.26:0/2266942543 wait complete. 2022-07-20T02:33:22.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:22.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 Processor -- start 2022-07-20T02:33:22.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 -- start start 2022-07-20T02:33:22.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7648060f00 0x7f7648143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76481474e0 con 0x7f7648147000 2022-07-20T02:33:22.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480b4e40 con 0x7f764805cfd0 2022-07-20T02:33:22.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764805d3b0 con 0x7f7648060f00 2022-07-20T02:33:22.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7645d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7648147000 0x7f764813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40542/0 (socket says 172.21.15.26:40542) 2022-07-20T02:33:22.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7645d9b700 1 -- 172.21.15.26:0/3598366162 learned_addr learned my addr 172.21.15.26:0/3598366162 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:22.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7646d9d700 1 --2- 172.21.15.26:0/3598366162 >> v2:172.21.15.120:3300/0 conn(0x7f7648060f00 0x7f7648143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172171267 0 0) 0x7f76481474e0 con 0x7f7648147000 2022-07-20T02:33:22.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f762c003260 con 0x7f7648147000 2022-07-20T02:33:22.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55910596 0 0) 0x7f76480b4e40 con 0x7f764805cfd0 2022-07-20T02:33:22.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76481474e0 con 0x7f764805cfd0 2022-07-20T02:33:22.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7646d9d700 1 -- 172.21.15.26:0/3598366162 >> v1:172.21.15.120:6789/0 conn(0x7f764805cfd0 legacy=0x7f76480b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7646d9d700 1 -- 172.21.15.26:0/3598366162 >> v1:172.21.15.26:6789/0 conn(0x7f7648147000 legacy=0x7f764813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7646d9d700 1 -- 172.21.15.26:0/3598366162 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76481431e0 con 0x7f7648060f00 2022-07-20T02:33:22.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.372+0000 7f7646d9d700 1 --2- 172.21.15.26:0/3598366162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648060f00 0x7f7648143bc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f76400032f0 tx=0x7f76400091d0).ready entity=mon.1 client_cookie=26d3fe39b45f943f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f764000b600 con 0x7f7648060f00 2022-07-20T02:33:22.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764000b760 con 0x7f7648060f00 2022-07-20T02:33:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764559a700 1 -- 172.21.15.26:0/3598366162 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f764000b8d0 con 0x7f7648060f00 2022-07-20T02:33:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- 172.21.15.26:0/3598366162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648060f00 msgr2=0x7f7648143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 --2- 172.21.15.26:0/3598366162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648060f00 0x7f7648143bc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f76400032f0 tx=0x7f76400091d0).stop 2022-07-20T02:33:22.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- 172.21.15.26:0/3598366162 shutdown_connections 2022-07-20T02:33:22.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 --2- 172.21.15.26:0/3598366162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648060f00 0x7f7648143bc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- 172.21.15.26:0/3598366162 >> 172.21.15.26:0/3598366162 conn(0x7f76480620c0 msgr2=0x7f764805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- 172.21.15.26:0/3598366162 shutdown_connections 2022-07-20T02:33:22.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- 172.21.15.26:0/3598366162 wait complete. 2022-07-20T02:33:22.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 Processor -- start 2022-07-20T02:33:22.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 -- start start 2022-07-20T02:33:22.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 0x7f7648146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.376+0000 7f764759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 0x7f76481420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648142b50 con 0x7f7648060f00 2022-07-20T02:33:22.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648142df0 con 0x7f7648144e00 2022-07-20T02:33:22.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7648152e20 con 0x7f7648147000 2022-07-20T02:33:22.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 0x7f7648146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7646d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 0x7f76481420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34550/0 (socket says 172.21.15.26:34550) 2022-07-20T02:33:22.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 -- 172.21.15.26:0/4132784775 learned_addr learned my addr 172.21.15.26:0/4132784775 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:22.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7646d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 0x7f76481420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:60962/0 (socket says 172.21.15.26:60962) 2022-07-20T02:33:22.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764659c700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 0x7f7648146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41268/0 (socket says 172.21.15.26:41268) 2022-07-20T02:33:22.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 -- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 msgr2=0x7f76481420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 0x7f76481420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 -- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 msgr2=0x7f7648146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 0x7f7648146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 -- 172.21.15.26:0/4132784775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7640009470 con 0x7f7648147000 2022-07-20T02:33:22.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f7645d9b700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7638001110 tx=0x7f7638007af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76380070d0 con 0x7f7648147000 2022-07-20T02:33:22.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7638007230 con 0x7f7648147000 2022-07-20T02:33:22.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7648152b20 con 0x7f7648147000 2022-07-20T02:33:22.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7638005610 con 0x7f7648147000 2022-07-20T02:33:22.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648142760 con 0x7f7648147000 2022-07-20T02:33:22.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76480ad9d0 con 0x7f7648147000 2022-07-20T02:33:22.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.380+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7638010070 con 0x7f7648147000 2022-07-20T02:33:22.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.384+0000 7f76377fe700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 0x7f7624055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.384+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f763806b3f0 con 0x7f7648147000 2022-07-20T02:33:22.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.384+0000 7f764659c700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 0x7f7624055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.384+0000 7f764659c700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 0x7f7624055f60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f763c00d000 tx=0x7f763c007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.392+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76480ad9d0 con 0x7f7648147000 2022-07-20T02:33:22.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.524+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76480ad9d0 con 0x7f7648147000 2022-07-20T02:33:22.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f76377fe700 1 -- 172.21.15.26:0/4132784775 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+13464 (secure 0 0 0) 0x7f76480ad9d0 con 0x7f7648147000 2022-07-20T02:33:22.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:22.533 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":104,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:21.818453+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:22.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 msgr2=0x7f7624055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 0x7f7624055f60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f763c00d000 tx=0x7f763c007ad0).stop 2022-07-20T02:33:22.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 msgr2=0x7f7648144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7638001110 tx=0x7f7638007af0).stop 2022-07-20T02:33:22.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 shutdown_connections 2022-07-20T02:33:22.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7624053aa0 0x7f7624055f60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648144e00 0x7f76481420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7648147000 0x7f7648144570 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 --2- 172.21.15.26:0/4132784775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7648060f00 0x7f7648146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 >> 172.21.15.26:0/4132784775 conn(0x7f764805cfd0 msgr2=0x7f764813ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 shutdown_connections 2022-07-20T02:33:22.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.528+0000 7f764759e700 1 -- 172.21.15.26:0/4132784775 wait complete. 2022-07-20T02:33:22.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:33:22.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:22.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 Processor -- start 2022-07-20T02:33:22.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 -- start start 2022-07-20T02:33:22.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0734147be0 0x7f07340aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07340b62d0 con 0x7f07340b5ef0 2022-07-20T02:33:22.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073414aac0 con 0x7f07340a7fc0 2022-07-20T02:33:22.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f073a689700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073414ad60 con 0x7f0734147be0 2022-07-20T02:33:22.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.668+0000 7f0738e86700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0734147be0 0x7f07340aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0739687700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f07340b5ef0 0x7f07341474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40554/0 (socket says 172.21.15.26:40554) 2022-07-20T02:33:22.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0739687700 1 -- 172.21.15.26:0/2932121754 learned_addr learned my addr 172.21.15.26:0/2932121754 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:22.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0738e86700 1 -- 172.21.15.26:0/2932121754 >> v1:172.21.15.120:6789/0 conn(0x7f07340a7fc0 legacy=0x7f07340af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0738e86700 1 -- 172.21.15.26:0/2932121754 >> v1:172.21.15.26:6789/0 conn(0x7f07340b5ef0 legacy=0x7f07341474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f072bfff700 1 -- 172.21.15.26:0/2932121754 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491796826 0 0) 0x7f07340b62d0 con 0x7f07340b5ef0 2022-07-20T02:33:22.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0738e86700 1 -- 172.21.15.26:0/2932121754 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073414b000 con 0x7f0734147be0 2022-07-20T02:33:22.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f0738e86700 1 --2- 172.21.15.26:0/2932121754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0734147be0 0x7f07340aee40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f07300032f0 tx=0x7f07300091d0).ready entity=mon.0 client_cookie=e0a4f890dae583d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f072bfff700 1 -- 172.21.15.26:0/2932121754 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0730009ea0 con 0x7f0734147be0 2022-07-20T02:33:22.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f072bfff700 1 -- 172.21.15.26:0/2932121754 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073000b8a0 con 0x7f0734147be0 2022-07-20T02:33:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f072bfff700 1 -- 172.21.15.26:0/2932121754 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f073000ba10 con 0x7f0734147be0 2022-07-20T02:33:22.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 -- 172.21.15.26:0/2932121754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0734147be0 msgr2=0x7f07340aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 --2- 172.21.15.26:0/2932121754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0734147be0 0x7f07340aee40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f07300032f0 tx=0x7f07300091d0).stop 2022-07-20T02:33:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 -- 172.21.15.26:0/2932121754 shutdown_connections 2022-07-20T02:33:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 --2- 172.21.15.26:0/2932121754 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0734147be0 0x7f07340aee40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 -- 172.21.15.26:0/2932121754 >> 172.21.15.26:0/2932121754 conn(0x7f0734060310 msgr2=0x7f07340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 -- 172.21.15.26:0/2932121754 shutdown_connections 2022-07-20T02:33:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 -- 172.21.15.26:0/2932121754 wait complete. 2022-07-20T02:33:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.672+0000 7f073a689700 1 Processor -- start 2022-07-20T02:33:22.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- start start 2022-07-20T02:33:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 0x7f07340b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0734147be0 0x7f0734159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07340a92b0 con 0x7f07340a7fc0 2022-07-20T02:33:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07340aaf60 con 0x7f07340b5ef0 2022-07-20T02:33:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07340aa230 con 0x7f0734147be0 2022-07-20T02:33:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41280/0 (socket says 172.21.15.26:41280) 2022-07-20T02:33:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 -- 172.21.15.26:0/2293000258 learned_addr learned my addr 172.21.15.26:0/2293000258 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0738e86700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 0x7f07340b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739e88700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0734147be0 0x7f0734159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 -- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0734147be0 msgr2=0x7f0734159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0734147be0 0x7f0734159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 -- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 msgr2=0x7f07340b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 0x7f07340b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 -- 172.21.15.26:0/2293000258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0730009470 con 0x7f07340a7fc0 2022-07-20T02:33:22.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0738e86700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 0x7f07340b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:22.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f0739687700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f0724000bc0 tx=0x7f0724007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07240072e0 con 0x7f07340a7fc0 2022-07-20T02:33:22.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724007440 con 0x7f07340a7fc0 2022-07-20T02:33:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07340ab1c0 con 0x7f07340a7fc0 2022-07-20T02:33:22.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07240057e0 con 0x7f07340a7fc0 2022-07-20T02:33:22.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07340aaba0 con 0x7f07340a7fc0 2022-07-20T02:33:22.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.676+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0734147a20 con 0x7f07340a7fc0 2022-07-20T02:33:22.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.680+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0724010070 con 0x7f07340a7fc0 2022-07-20T02:33:22.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.680+0000 7f072a7fc700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 0x7f071c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.680+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f072406b570 con 0x7f07340a7fc0 2022-07-20T02:33:22.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.680+0000 7f0738e86700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 0x7f071c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.680+0000 7f0738e86700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 0x7f071c055f30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f0730019000 tx=0x7f0730016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.684+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0724042670 con 0x7f07340a7fc0 2022-07-20T02:33:22.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.808+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f073414b2a0 con 0x7f07340a7fc0 2022-07-20T02:33:22.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.808+0000 7f072a7fc700 1 -- 172.21.15.26:0/2293000258 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v104) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0724045c90 con 0x7f07340a7fc0 2022-07-20T02:33:22.828 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:22.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.808+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 msgr2=0x7f071c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.808+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 0x7f071c055f30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f0730019000 tx=0x7f0730016000).stop 2022-07-20T02:33:22.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 msgr2=0x7f0734147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f0724000bc0 tx=0x7f0724007af0).stop 2022-07-20T02:33:22.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 shutdown_connections 2022-07-20T02:33:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f071c053a70 0x7f071c055f30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0734147be0 0x7f0734159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07340b5ef0 0x7f07340b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 --2- 172.21.15.26:0/2293000258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07340a7fc0 0x7f0734147390 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 >> 172.21.15.26:0/2293000258 conn(0x7f0734060310 msgr2=0x7f07340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 shutdown_connections 2022-07-20T02:33:22.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.812+0000 7f073a689700 1 -- 172.21.15.26:0/2293000258 wait complete. 2022-07-20T02:33:22.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:22.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 Processor -- start 2022-07-20T02:33:22.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 -- start start 2022-07-20T02:33:22.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b3405c720 0x7f2b34147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 0x7f2b34147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:22.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3405d4d0 con 0x7f2b340b1200 2022-07-20T02:33:22.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b340a9ff0 con 0x7f2b3405c720 2022-07-20T02:33:22.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b33fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b34067e60 con 0x7f2b340afe40 2022-07-20T02:33:22.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b3405c720 0x7f2b34147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b3405c720 0x7f2b34147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41288/0 (socket says 172.21.15.26:41288) 2022-07-20T02:33:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 -- 172.21.15.26:0/3910830071 learned_addr learned my addr 172.21.15.26:0/3910830071 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b327fc700 1 --2- 172.21.15.26:0/3910830071 >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 0x7f2b34147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 -- 172.21.15.26:0/3910830071 >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 msgr2=0x7f2b34147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/3910830071 >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 0x7f2b34147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 -- 172.21.15.26:0/3910830071 >> v1:172.21.15.120:6789/0 conn(0x7f2b340b1200 legacy=0x7f2b340abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 -- 172.21.15.26:0/3910830071 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b34067bc0 con 0x7f2b3405c720 2022-07-20T02:33:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b327fc700 1 --2- 172.21.15.26:0/3910830071 >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 0x7f2b34147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/3910830071 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b3405c720 0x7f2b34147550 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24006eb0).ready entity=mon.0 client_cookie=cc1b300769731a85 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b31ffb700 1 -- 172.21.15.26:0/3910830071 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2400d610 con 0x7f2b3405c720 2022-07-20T02:33:22.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b31ffb700 1 -- 172.21.15.26:0/3910830071 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b24000e90 con 0x7f2b3405c720 2022-07-20T02:33:22.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.944+0000 7f2b31ffb700 1 -- 172.21.15.26:0/3910830071 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2400d9a0 con 0x7f2b3405c720 2022-07-20T02:33:22.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- 172.21.15.26:0/3910830071 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b3405c720 msgr2=0x7f2b34147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:22.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- 172.21.15.26:0/3910830071 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b3405c720 0x7f2b34147550 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24006eb0).stop 2022-07-20T02:33:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- 172.21.15.26:0/3910830071 shutdown_connections 2022-07-20T02:33:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- 172.21.15.26:0/3910830071 >> v2:172.21.15.120:3300/0 conn(0x7f2b340afe40 0x7f2b34147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- 172.21.15.26:0/3910830071 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b3405c720 0x7f2b34147550 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- 172.21.15.26:0/3910830071 >> 172.21.15.26:0/3910830071 conn(0x7f2b34060310 msgr2=0x7f2b340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- 172.21.15.26:0/3910830071 shutdown_connections 2022-07-20T02:33:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- 172.21.15.26:0/3910830071 wait complete. 2022-07-20T02:33:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 Processor -- start 2022-07-20T02:33:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- start start 2022-07-20T02:33:23.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 0x7f2b340b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340afe40 0x7f2b34065390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b341568e0 con 0x7f2b340b1200 2022-07-20T02:33:23.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3415ad80 con 0x7f2b340afe40 2022-07-20T02:33:23.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b33fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b34067330 con 0x7f2b3405c720 2022-07-20T02:33:23.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b337fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b337fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41298/0 (socket says 172.21.15.26:41298) 2022-07-20T02:33:23.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b337fe700 1 -- 172.21.15.26:0/1724723786 learned_addr learned my addr 172.21.15.26:0/1724723786 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:23.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b327fc700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340afe40 0x7f2b34065390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.948+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 0x7f2b340b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 -- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 msgr2=0x7f2b340b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 0x7f2b340b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 -- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340afe40 msgr2=0x7f2b34065390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340afe40 0x7f2b34065390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 -- 172.21.15.26:0/1724723786 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2400b1e0 con 0x7f2b340b1200 2022-07-20T02:33:23.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 0x7f2b340b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b337fe700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f2b3405d220 tx=0x7f2b2c0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2c006aa0 con 0x7f2b340b1200 2022-07-20T02:33:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c006c00 con 0x7f2b340b1200 2022-07-20T02:33:23.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3415a840 con 0x7f2b340b1200 2022-07-20T02:33:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2c007030 con 0x7f2b340b1200 2022-07-20T02:33:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b34156d70 con 0x7f2b340b1200 2022-07-20T02:33:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b2c010030 con 0x7f2b340b1200 2022-07-20T02:33:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b34156f30 con 0x7f2b340b1200 2022-07-20T02:33:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 0x7f2b10055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 0x7f2b10055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f2b2c06bc90 con 0x7f2b340b1200 2022-07-20T02:33:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.952+0000 7f2b32ffd700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 0x7f2b10055e10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f2b2401a000 tx=0x7f2b24015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:22.960+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b2c042c80 con 0x7f2b340b1200 2022-07-20T02:33:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.084+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2b340aa250 con 0x7f2b340b1200 2022-07-20T02:33:23.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.084+0000 7f2b23fff700 1 -- 172.21.15.26:0/1724723786 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v104) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2b2c0462a0 con 0x7f2b340b1200 2022-07-20T02:33:23.089 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:23.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 msgr2=0x7f2b10055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 0x7f2b10055e10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f2b2401a000 tx=0x7f2b24015000).stop 2022-07-20T02:33:23.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 msgr2=0x7f2b34157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f2b3405d220 tx=0x7f2b2c0013a0).stop 2022-07-20T02:33:23.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 shutdown_connections 2022-07-20T02:33:23.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b340b1200 0x7f2b34157210 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b10053950 0x7f2b10055e10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340afe40 0x7f2b34065390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1724723786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b3405c720 0x7f2b340b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 >> 172.21.15.26:0/1724723786 conn(0x7f2b34060310 msgr2=0x7f2b340a7440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:23.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 shutdown_connections 2022-07-20T02:33:23.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.088+0000 7f2b33fff700 1 -- 172.21.15.26:0/1724723786 wait complete. 2022-07-20T02:33:23.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:33:23.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.212+0000 7f33c7111700 1 Processor -- start 2022-07-20T02:33:23.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.212+0000 7f33c7111700 1 -- start start 2022-07-20T02:33:23.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f33c00b5ef0 0x7f33c01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c01479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00b62d0 con 0x7f33c00a8c40 2022-07-20T02:33:23.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c0147eb0 con 0x7f33c00b5ef0 2022-07-20T02:33:23.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c014aac0 con 0x7f33c00a7fc0 2022-07-20T02:33:23.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f33c00b5ef0 0x7f33c01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f33c00b5ef0 0x7f33c01474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41304/0 (socket says 172.21.15.26:41304) 2022-07-20T02:33:23.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 -- 172.21.15.26:0/2083560866 learned_addr learned my addr 172.21.15.26:0/2083560866 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:23.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c590e700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c01479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 -- 172.21.15.26:0/2083560866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 msgr2=0x7f33c01479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:23.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c01479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 -- 172.21.15.26:0/2083560866 >> v1:172.21.15.120:6789/0 conn(0x7f33c00a8c40 legacy=0x7f33c00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 -- 172.21.15.26:0/2083560866 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c014ad60 con 0x7f33c00b5ef0 2022-07-20T02:33:23.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c610f700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00b5ef0 0x7f33c01474d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f33b0002f70 tx=0x7f33b0006eb0).ready entity=mon.0 client_cookie=7a1141181d4e200a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c510d700 1 -- 172.21.15.26:0/2083560866 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33b000d570 con 0x7f33c00b5ef0 2022-07-20T02:33:23.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c510d700 1 -- 172.21.15.26:0/2083560866 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b0000e90 con 0x7f33c00b5ef0 2022-07-20T02:33:23.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c510d700 1 -- 172.21.15.26:0/2083560866 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33b000d900 con 0x7f33c00b5ef0 2022-07-20T02:33:23.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- 172.21.15.26:0/2083560866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00b5ef0 msgr2=0x7f33c01474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00b5ef0 0x7f33c01474d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f33b0002f70 tx=0x7f33b0006eb0).stop 2022-07-20T02:33:23.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- 172.21.15.26:0/2083560866 shutdown_connections 2022-07-20T02:33:23.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c01479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 --2- 172.21.15.26:0/2083560866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00b5ef0 0x7f33c01474d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- 172.21.15.26:0/2083560866 >> 172.21.15.26:0/2083560866 conn(0x7f33c0060310 msgr2=0x7f33c00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:23.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- 172.21.15.26:0/2083560866 shutdown_connections 2022-07-20T02:33:23.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.216+0000 7f33c7111700 1 -- 172.21.15.26:0/2083560866 wait complete. 2022-07-20T02:33:23.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 Processor -- start 2022-07-20T02:33:23.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 -- start start 2022-07-20T02:33:23.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33c00b5ef0 0x7f33c0147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00ab3c0 con 0x7f33c00a8c40 2022-07-20T02:33:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0062870 con 0x7f33c00b5ef0 2022-07-20T02:33:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c7111700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00654a0 con 0x7f33c00a7fc0 2022-07-20T02:33:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41310/0 (socket says 172.21.15.26:41310) 2022-07-20T02:33:23.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 -- 172.21.15.26:0/4199794715 learned_addr learned my addr 172.21.15.26:0/4199794715 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:23.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c610f700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c6910700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33c00b5ef0 0x7f33c0147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 -- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 msgr2=0x7f33c00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c00b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 -- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33c00b5ef0 msgr2=0x7f33c0147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33c00b5ef0 0x7f33c0147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c590e700 1 -- 172.21.15.26:0/4199794715 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b000b1e0 con 0x7f33c00a8c40 2022-07-20T02:33:23.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.220+0000 7f33c610f700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33c590e700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f33bc00c000 tx=0x7f33bc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33bc010590 con 0x7f33c00a8c40 2022-07-20T02:33:23.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc0106f0 con 0x7f33c00a8c40 2022-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33bc010a50 con 0x7f33c00a8c40 2022-07-20T02:33:23.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c00b48b0 con 0x7f33c00a8c40 2022-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c00aad30 con 0x7f33c00a8c40 2022-07-20T02:33:23.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c00ab530 con 0x7f33c00a8c40 2022-07-20T02:33:23.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.224+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f33bc010bb0 con 0x7f33c00a8c40 2022-07-20T02:33:23.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.228+0000 7f33af7fe700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 0x7f33a0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.228+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f33bc06bef0 con 0x7f33c00a8c40 2022-07-20T02:33:23.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.228+0000 7f33c610f700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 0x7f33a0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.228+0000 7f33c610f700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 0x7f33a0055e10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f33b001a000 tx=0x7f33b0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.232+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33c00ab530 con 0x7f33c00a8c40 2022-07-20T02:33:23.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f33c00ab500 con 0x7f33c00a8c40 2022-07-20T02:33:23.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33af7fe700 1 -- 172.21.15.26:0/4199794715 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+143 (secure 0 0 0) 0x7f33bc042f50 con 0x7f33c00a8c40 2022-07-20T02:33:23.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:23.365 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:33:23.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:23.365 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:33:23.365 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:33:23.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:23.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:23.366 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:33:23.366 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:33:23.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:23.367 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:33:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 msgr2=0x7f33a0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 0x7f33a0055e10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f33b001a000 tx=0x7f33b0015000).stop 2022-07-20T02:33:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 msgr2=0x7f33c0147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f33bc00c000 tx=0x7f33bc006df0).stop 2022-07-20T02:33:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 shutdown_connections 2022-07-20T02:33:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33c00a8c40 0x7f33c0147390 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f33a0053950 0x7f33a0055e10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33c00b5ef0 0x7f33c0147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 --2- 172.21.15.26:0/4199794715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f33c00a7fc0 0x7f33c00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 >> 172.21.15.26:0/4199794715 conn(0x7f33c0060310 msgr2=0x7f33c005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 shutdown_connections 2022-07-20T02:33:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.360+0000 7f33c7111700 1 -- 172.21.15.26:0/4199794715 wait complete. 2022-07-20T02:33:23.379 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:33:23.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:33:23.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 Processor -- start 2022-07-20T02:33:23.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 -- start start 2022-07-20T02:33:23.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59d0147000 0x7f59d0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d01474e0 con 0x7f59d005cfd0 2022-07-20T02:33:23.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00b4e40 con 0x7f59d0147000 2022-07-20T02:33:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.496+0000 7f59cf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d005d3b0 con 0x7f59d0060f00 2022-07-20T02:33:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59ce59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f59d005cfd0 0x7f59d00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40596/0 (socket says 172.21.15.26:40596) 2022-07-20T02:33:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59ce59c700 1 -- 172.21.15.26:0/2880323217 learned_addr learned my addr 172.21.15.26:0/2880323217 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:23.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 --2- 172.21.15.26:0/2880323217 >> v2:172.21.15.120:3300/0 conn(0x7f59d0147000 0x7f59d0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59ced9d700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cd59a700 1 -- 172.21.15.26:0/2880323217 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033566290 0 0) 0x7f59d01474e0 con 0x7f59d005cfd0 2022-07-20T02:33:23.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cd59a700 1 -- 172.21.15.26:0/2880323217 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b0003260 con 0x7f59d005cfd0 2022-07-20T02:33:23.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 -- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 msgr2=0x7f59d013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:23.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 -- 172.21.15.26:0/2880323217 >> v1:172.21.15.26:6789/0 conn(0x7f59d005cfd0 legacy=0x7f59d00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 -- 172.21.15.26:0/2880323217 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d01431e0 con 0x7f59d0147000 2022-07-20T02:33:23.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cdd9b700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0143bc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f59c0002f70 tx=0x7f59c0006df0).ready entity=mon.1 client_cookie=28c446e539045a4f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cd59a700 1 -- 172.21.15.26:0/2880323217 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c000d4a0 con 0x7f59d0147000 2022-07-20T02:33:23.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cd59a700 1 -- 172.21.15.26:0/2880323217 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0000e90 con 0x7f59d0147000 2022-07-20T02:33:23.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cd59a700 1 -- 172.21.15.26:0/2880323217 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c000d7f0 con 0x7f59d0147000 2022-07-20T02:33:23.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 msgr2=0x7f59d0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0143bc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f59c0002f70 tx=0x7f59c0006df0).stop 2022-07-20T02:33:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- 172.21.15.26:0/2880323217 shutdown_connections 2022-07-20T02:33:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 --2- 172.21.15.26:0/2880323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0143bc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- 172.21.15.26:0/2880323217 >> 172.21.15.26:0/2880323217 conn(0x7f59d00620c0 msgr2=0x7f59d005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:23.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- 172.21.15.26:0/2880323217 shutdown_connections 2022-07-20T02:33:23.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- 172.21.15.26:0/2880323217 wait complete. 2022-07-20T02:33:23.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 Processor -- start 2022-07-20T02:33:23.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.500+0000 7f59cf59e700 1 -- start start 2022-07-20T02:33:23.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00b0850 con 0x7f59d013cc70 2022-07-20T02:33:23.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00b0180 con 0x7f59d0147000 2022-07-20T02:33:23.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00ba640 con 0x7f59d0060f00 2022-07-20T02:33:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41326/0 (socket says 172.21.15.26:41326) 2022-07-20T02:33:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 -- 172.21.15.26:0/253150423 learned_addr learned my addr 172.21.15.26:0/253150423 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ce59c700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cdd9b700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 -- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 msgr2=0x7f59d013c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 -- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 msgr2=0x7f59d0155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:23.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:23.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 -- 172.21.15.26:0/253150423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c000b180 con 0x7f59d013cc70 2022-07-20T02:33:23.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ce59c700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:23.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59ced9d700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f59c8006e00 tx=0x7f59c80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c800c7f0 con 0x7f59d013cc70 2022-07-20T02:33:23.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800c950 con 0x7f59d013cc70 2022-07-20T02:33:23.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d0152d60 con 0x7f59d013cc70 2022-07-20T02:33:23.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c800ccb0 con 0x7f59d013cc70 2022-07-20T02:33:23.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d0153080 con 0x7f59d013cc70 2022-07-20T02:33:23.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.504+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d00aff10 con 0x7f59d013cc70 2022-07-20T02:33:23.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.508+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f59c8022070 con 0x7f59d013cc70 2022-07-20T02:33:23.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.508+0000 7f59c77fe700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 0x7f59b4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:23.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.508+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f59c806c360 con 0x7f59d013cc70 2022-07-20T02:33:23.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.508+0000 7f59ce59c700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 0x7f59b4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:23.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.508+0000 7f59ce59c700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 0x7f59b4055ff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f59b800b000 tx=0x7f59b8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:23.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.512+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59c8043460 con 0x7f59d013cc70 2022-07-20T02:33:23.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:23.636+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f59d0143480 con 0x7f59d013cc70 2022-07-20T02:33:24.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.548+0000 7f59c77fe700 1 -- 172.21.15.26:0/253150423 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v105) v1 ==== 133+0+0 (secure 0 0 0) 0x7f59c80160a0 con 0x7f59d013cc70 2022-07-20T02:33:24.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 msgr2=0x7f59b4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:24.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 0x7f59b4055ff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f59b800b000 tx=0x7f59b8009040).stop 2022-07-20T02:33:24.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 msgr2=0x7f59d0155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:24.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f59c8006e00 tx=0x7f59c80091d0).stop 2022-07-20T02:33:24.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 shutdown_connections 2022-07-20T02:33:24.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59d013cc70 0x7f59d0155c20 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f59b4053b30 0x7f59b4055ff0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0147000 0x7f59d0155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 --2- 172.21.15.26:0/253150423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59d0060f00 0x7f59d013c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 >> 172.21.15.26:0/253150423 conn(0x7f59d005cfd0 msgr2=0x7f59d00b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:24.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 shutdown_connections 2022-07-20T02:33:24.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.552+0000 7f59cf59e700 1 -- 172.21.15.26:0/253150423 wait complete. 2022-07-20T02:33:24.562 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:33:24.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:33:24.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 Processor -- start 2022-07-20T02:33:24.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 -- start start 2022-07-20T02:33:24.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0048140320 0x7f0048140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f004805c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 0x7f0048143040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00480b1ce0 con 0x7f0048140320 2022-07-20T02:33:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00480ab530 con 0x7f004805ca70 2022-07-20T02:33:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f004ce09700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00480aab50 con 0x7f0048147020 2022-07-20T02:33:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0048140320 0x7f0048140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0048140320 0x7f0048140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41330/0 (socket says 172.21.15.26:41330) 2022-07-20T02:33:24.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 -- 172.21.15.26:0/3887875635 learned_addr learned my addr 172.21.15.26:0/3887875635 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f0047fff700 1 --2- 172.21.15.26:0/3887875635 >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 0x7f0048143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f0046ffd700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f004805c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 -- 172.21.15.26:0/3887875635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 msgr2=0x7f004805c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f004805c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 -- 172.21.15.26:0/3887875635 >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 msgr2=0x7f0048143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:24.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 --2- 172.21.15.26:0/3887875635 >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 0x7f0048143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 -- 172.21.15.26:0/3887875635 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00480aadf0 con 0x7f0048140320 2022-07-20T02:33:24.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f0047fff700 1 --2- 172.21.15.26:0/3887875635 >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 0x7f0048143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.692+0000 7f00477fe700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048140320 0x7f0048140700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f0038002f70 tx=0x7f0038006eb0).ready entity=mon.0 client_cookie=eb0f668253ec2c79 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f00467fc700 1 -- 172.21.15.26:0/3887875635 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003800d570 con 0x7f0048140320 2022-07-20T02:33:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f00467fc700 1 -- 172.21.15.26:0/3887875635 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0038000e90 con 0x7f0048140320 2022-07-20T02:33:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f00467fc700 1 -- 172.21.15.26:0/3887875635 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003800d900 con 0x7f0048140320 2022-07-20T02:33:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- 172.21.15.26:0/3887875635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048140320 msgr2=0x7f0048140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048140320 0x7f0048140700 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f0038002f70 tx=0x7f0038006eb0).stop 2022-07-20T02:33:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- 172.21.15.26:0/3887875635 shutdown_connections 2022-07-20T02:33:24.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 --2- 172.21.15.26:0/3887875635 >> v2:172.21.15.120:3300/0 conn(0x7f004805ca70 0x7f0048143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f004805c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 --2- 172.21.15.26:0/3887875635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048140320 0x7f0048140700 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- 172.21.15.26:0/3887875635 >> 172.21.15.26:0/3887875635 conn(0x7f0048060310 msgr2=0x7f00480b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:24.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- 172.21.15.26:0/3887875635 shutdown_connections 2022-07-20T02:33:24.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- 172.21.15.26:0/3887875635 wait complete. 2022-07-20T02:33:24.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 Processor -- start 2022-07-20T02:33:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.696+0000 7f004ce09700 1 -- start start 2022-07-20T02:33:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 0x7f00480b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048142440 0x7f00480aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00480ade60 con 0x7f0048142440 2022-07-20T02:33:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0048152580 con 0x7f0048140320 2022-07-20T02:33:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0048143a50 con 0x7f0048147020 2022-07-20T02:33:24.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f00477fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 0x7f00480b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34622/0 (socket says 172.21.15.26:34622) 2022-07-20T02:33:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0047fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048142440 0x7f00480aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 -- 172.21.15.26:0/3374938993 learned_addr learned my addr 172.21.15.26:0/3374938993 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:24.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 -- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 msgr2=0x7f00480b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:24.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 0x7f00480b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 -- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048142440 msgr2=0x7f00480aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048142440 0x7f00480aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 -- 172.21.15.26:0/3374938993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040009470 con 0x7f0048147020 2022-07-20T02:33:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f00477fe700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 0x7f00480b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0046ffd700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0048143d20 tx=0x7f003c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003c011750 con 0x7f0048147020 2022-07-20T02:33:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c0118b0 con 0x7f0048147020 2022-07-20T02:33:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003800b1e0 con 0x7f0048147020 2022-07-20T02:33:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003c011c10 con 0x7f0048147020 2022-07-20T02:33:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.700+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0048142de0 con 0x7f0048147020 2022-07-20T02:33:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0048142fa0 con 0x7f0048147020 2022-07-20T02:33:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f003c011d70 con 0x7f0048147020 2022-07-20T02:33:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f0044ff9700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 0x7f0028055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f003c06c110 con 0x7f0048147020 2022-07-20T02:33:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f00477fe700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 0x7f0028055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.704+0000 7f00477fe700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 0x7f0028055e10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f003801c000 tx=0x7f0038015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.712+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f003c043000 con 0x7f0048147020 2022-07-20T02:33:24.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:24.848+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f00480ab090 con 0x7f0048147020 2022-07-20T02:33:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.816+0000 7f0044ff9700 1 -- 172.21.15.26:0/3374938993 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v107) v1 ==== 125+0+0 (secure 0 0 0) 0x7f003c046620 con 0x7f0048147020 2022-07-20T02:33:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 msgr2=0x7f0028055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 0x7f0028055e10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f003801c000 tx=0x7f0038015000).stop 2022-07-20T02:33:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 msgr2=0x7f00480ae670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0048143d20 tx=0x7f003c006df0).stop 2022-07-20T02:33:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 shutdown_connections 2022-07-20T02:33:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0048142440 0x7f00480aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0028053950 0x7f0028055e10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0048147020 0x7f00480ae670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 --2- 172.21.15.26:0/3374938993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048140320 0x7f00480b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 >> 172.21.15.26:0/3374938993 conn(0x7f0048060310 msgr2=0x7f004813b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 shutdown_connections 2022-07-20T02:33:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.820+0000 7f004ce09700 1 -- 172.21.15.26:0/3374938993 wait complete. 2022-07-20T02:33:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:33:25.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:33:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 Processor -- start 2022-07-20T02:33:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 -- start start 2022-07-20T02:33:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f13440adf90 0x7f13440b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f1344060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13440ae370 con 0x7f1344060e40 2022-07-20T02:33:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13440b28b0 con 0x7f13440adf90 2022-07-20T02:33:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134c781700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13440b0610 con 0x7f13440b14b0 2022-07-20T02:33:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.948+0000 7f134a51d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f13440adf90 0x7f13440b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134ad1e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1344060e40 0x7f13440614b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40630/0 (socket says 172.21.15.26:40630) 2022-07-20T02:33:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134ad1e700 1 -- 172.21.15.26:0/3418388563 learned_addr learned my addr 172.21.15.26:0/3418388563 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f1349d1c700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f1344060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 -- 172.21.15.26:0/3418388563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 msgr2=0x7f1344060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f1344060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 -- 172.21.15.26:0/3418388563 >> v1:172.21.15.26:6789/0 conn(0x7f1344060e40 legacy=0x7f13440614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 -- 172.21.15.26:0/3418388563 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440b08b0 con 0x7f13440adf90 2022-07-20T02:33:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134951b700 1 -- 172.21.15.26:0/3418388563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848695312 0 0) 0x7f13440ae370 con 0x7f1344060e40 2022-07-20T02:33:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13440adf90 0x7f13440b2c60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f1340002f70 tx=0x7f1340006df0).ready entity=mon.0 client_cookie=8a6d942e1682b8d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134951b700 1 -- 172.21.15.26:0/3418388563 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1340010b00 con 0x7f13440adf90 2022-07-20T02:33:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134951b700 1 -- 172.21.15.26:0/3418388563 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340000e90 con 0x7f13440adf90 2022-07-20T02:33:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134951b700 1 -- 172.21.15.26:0/3418388563 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1340010e50 con 0x7f13440adf90 2022-07-20T02:33:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- 172.21.15.26:0/3418388563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13440adf90 msgr2=0x7f13440b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13440adf90 0x7f13440b2c60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f1340002f70 tx=0x7f1340006df0).stop 2022-07-20T02:33:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- 172.21.15.26:0/3418388563 shutdown_connections 2022-07-20T02:33:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f1344060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- 172.21.15.26:0/3418388563 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13440adf90 0x7f13440b2c60 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- 172.21.15.26:0/3418388563 >> 172.21.15.26:0/3418388563 conn(0x7f13440b66e0 msgr2=0x7f13440b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- 172.21.15.26:0/3418388563 shutdown_connections 2022-07-20T02:33:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- 172.21.15.26:0/3418388563 wait complete. 2022-07-20T02:33:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 Processor -- start 2022-07-20T02:33:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- start start 2022-07-20T02:33:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13440adf90 0x7f134414e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f13440a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13440a2010 con 0x7f1344060e40 2022-07-20T02:33:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134414e070 con 0x7f13440adf90 2022-07-20T02:33:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134c781700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13440a2240 con 0x7f13440b14b0 2022-07-20T02:33:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41352/0 (socket says 172.21.15.26:41352) 2022-07-20T02:33:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f134a51d700 1 -- 172.21.15.26:0/3407174335 learned_addr learned my addr 172.21.15.26:0/3407174335 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:26.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.952+0000 7f1349d1c700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13440adf90 0x7f134414e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134ad1e700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f13440a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 -- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 msgr2=0x7f13440a3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f13440a3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 -- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13440adf90 msgr2=0x7f134414e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13440adf90 0x7f134414e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 -- 172.21.15.26:0/3407174335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134000b180 con 0x7f1344060e40 2022-07-20T02:33:26.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134a51d700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f1340002f70 tx=0x7f1340016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1340021070 con 0x7f1344060e40 2022-07-20T02:33:26.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340014070 con 0x7f1344060e40 2022-07-20T02:33:26.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13440a24e0 con 0x7f1344060e40 2022-07-20T02:33:26.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134001c270 con 0x7f1344060e40 2022-07-20T02:33:26.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13440a0930 con 0x7f1344060e40 2022-07-20T02:33:26.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1340026460 con 0x7f1344060e40 2022-07-20T02:33:26.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13440a2670 con 0x7f1344060e40 2022-07-20T02:33:26.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 0x7f1324055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f13400709f0 con 0x7f1344060e40 2022-07-20T02:33:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f1349d1c700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 0x7f1324055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.956+0000 7f1349d1c700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 0x7f1324055e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f1338008000 tx=0x7f1338008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:25.964+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1340072e60 con 0x7f1344060e40 2022-07-20T02:33:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.128+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f13440574e0 con 0x7f1344060e40 2022-07-20T02:33:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.140+0000 7f13377fe700 1 -- 172.21.15.26:0/3407174335 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v53) v1 ==== 189+0+0 (secure 0 0 0) 0x7f13400477e0 con 0x7f1344060e40 2022-07-20T02:33:26.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 msgr2=0x7f1324055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 0x7f1324055e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f1338008000 tx=0x7f1338008040).stop 2022-07-20T02:33:26.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 msgr2=0x7f13440a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f1340002f70 tx=0x7f1340016000).stop 2022-07-20T02:33:26.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 shutdown_connections 2022-07-20T02:33:26.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1324053980 0x7f1324055e40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13440b14b0 0x7f13440a3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13440adf90 0x7f134414e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 --2- 172.21.15.26:0/3407174335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1344060e40 0x7f13440a1830 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.144+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 >> 172.21.15.26:0/3407174335 conn(0x7f13440b66e0 msgr2=0x7f13440b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.148+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 shutdown_connections 2022-07-20T02:33:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.148+0000 7f134c781700 1 -- 172.21.15.26:0/3407174335 wait complete. 2022-07-20T02:33:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 19 and data pool 20 2022-07-20T02:33:26.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:26.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 Processor -- start 2022-07-20T02:33:26.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 -- start start 2022-07-20T02:33:26.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 0x7f45740af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740b62d0 con 0x7f45740b5ef0 2022-07-20T02:33:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457414aac0 con 0x7f4574147be0 2022-07-20T02:33:26.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4573fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457414ad60 con 0x7f45740a7fc0 2022-07-20T02:33:26.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4572ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f45740b5ef0 0x7f45741474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40652/0 (socket says 172.21.15.26:40652) 2022-07-20T02:33:26.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4572ffd700 1 -- 172.21.15.26:0/1958481614 learned_addr learned my addr 172.21.15.26:0/1958481614 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:26.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f45737fe700 1 --2- 172.21.15.26:0/1958481614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 0x7f45740af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608941447 0 0) 0x7f45740b62d0 con 0x7f45740b5ef0 2022-07-20T02:33:26.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554003260 con 0x7f45740b5ef0 2022-07-20T02:33:26.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515584812 0 0) 0x7f457414aac0 con 0x7f4574147be0 2022-07-20T02:33:26.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45740b62d0 con 0x7f4574147be0 2022-07-20T02:33:26.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f45737fe700 1 -- 172.21.15.26:0/1958481614 >> v1:172.21.15.120:6789/0 conn(0x7f4574147be0 legacy=0x7f45740aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3143482804 0 0) 0x7f4554003260 con 0x7f45740b5ef0 2022-07-20T02:33:26.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f45737fe700 1 -- 172.21.15.26:0/1958481614 >> v1:172.21.15.26:6789/0 conn(0x7f45740b5ef0 legacy=0x7f45741474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f45737fe700 1 -- 172.21.15.26:0/1958481614 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457414b000 con 0x7f45740a7fc0 2022-07-20T02:33:26.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f45737fe700 1 --2- 172.21.15.26:0/1958481614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 0x7f45740af550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f4564001110 tx=0x7f455c006df0).ready entity=mon.2 client_cookie=e20a5ae7a386776a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4564001660 con 0x7f45740a7fc0 2022-07-20T02:33:26.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c000e90 con 0x7f45740a7fc0 2022-07-20T02:33:26.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.408+0000 7f4571ffb700 1 -- 172.21.15.26:0/1958481614 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f455c012460 con 0x7f45740a7fc0 2022-07-20T02:33:26.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1958481614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 msgr2=0x7f45740af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 --2- 172.21.15.26:0/1958481614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 0x7f45740af550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f4564001110 tx=0x7f455c006df0).stop 2022-07-20T02:33:26.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1958481614 shutdown_connections 2022-07-20T02:33:26.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 --2- 172.21.15.26:0/1958481614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740a7fc0 0x7f45740af550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1958481614 >> 172.21.15.26:0/1958481614 conn(0x7f4574060310 msgr2=0x7f45740b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:26.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1958481614 shutdown_connections 2022-07-20T02:33:26.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1958481614 wait complete. 2022-07-20T02:33:26.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 Processor -- start 2022-07-20T02:33:26.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- start start 2022-07-20T02:33:26.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740b5ef0 0x7f457405c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4574147be0 0x7f457411d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574067df0 con 0x7f45740a7fc0 2022-07-20T02:33:26.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574156990 con 0x7f4574147be0 2022-07-20T02:33:26.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457415a720 con 0x7f45740b5ef0 2022-07-20T02:33:26.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41378/0 (socket says 172.21.15.26:41378) 2022-07-20T02:33:26.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 -- 172.21.15.26:0/1439418439 learned_addr learned my addr 172.21.15.26:0/1439418439 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:26.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f45737fe700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4574147be0 0x7f457411d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f45727fc700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740b5ef0 0x7f457405c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 -- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740b5ef0 msgr2=0x7f457405c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740b5ef0 0x7f457405c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 -- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4574147be0 msgr2=0x7f457411d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4574147be0 0x7f457411d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 -- 172.21.15.26:0/1439418439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c00b180 con 0x7f45740a7fc0 2022-07-20T02:33:26.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4572ffd700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f4564001840 tx=0x7f4564001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f456400ce50 con 0x7f45740a7fc0 2022-07-20T02:33:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4564007170 con 0x7f45740a7fc0 2022-07-20T02:33:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4574064ba0 con 0x7f45740a7fc0 2022-07-20T02:33:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4564004730 con 0x7f45740a7fc0 2022-07-20T02:33:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.412+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457415a440 con 0x7f45740a7fc0 2022-07-20T02:33:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.416+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457415a920 con 0x7f45740a7fc0 2022-07-20T02:33:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.416+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f456401b070 con 0x7f45740a7fc0 2022-07-20T02:33:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.416+0000 7f456bfff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 0x7f4558055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.416+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f456406b680 con 0x7f45740a7fc0 2022-07-20T02:33:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.416+0000 7f45727fc700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 0x7f4558055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.420+0000 7f45727fc700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 0x7f4558055f60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f456c00b000 tx=0x7f456c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.424+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4564042440 con 0x7f45740a7fc0 2022-07-20T02:33:26.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.548+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f457414b2a0 con 0x7f45740a7fc0 2022-07-20T02:33:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.552+0000 7f456bfff700 1 -- 172.21.15.26:0/1439418439 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f4564045a60 con 0x7f45740a7fc0 2022-07-20T02:33:26.555 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:26.555 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":108,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:26.130485+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:23.691619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:24.900742+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:26.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.552+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 msgr2=0x7f4558055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.552+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 0x7f4558055f60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f456c00b000 tx=0x7f456c009040).stop 2022-07-20T02:33:26.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.552+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 msgr2=0x7f45741564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.552+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f4564001840 tx=0x7f4564001720).stop 2022-07-20T02:33:26.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 shutdown_connections 2022-07-20T02:33:26.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4558053aa0 0x7f4558055f60 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4574147be0 0x7f457411d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45740b5ef0 0x7f457405c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 --2- 172.21.15.26:0/1439418439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45740a7fc0 0x7f45741564a0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 >> 172.21.15.26:0/1439418439 conn(0x7f4574060310 msgr2=0x7f45740b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:26.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 shutdown_connections 2022-07-20T02:33:26.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.556+0000 7f4573fff700 1 -- 172.21.15.26:0/1439418439 wait complete. 2022-07-20T02:33:26.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:26.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 Processor -- start 2022-07-20T02:33:26.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 -- start start 2022-07-20T02:33:26.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fee78061860 0x7fee78067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee780980d0 0x7fee78142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7805ff70 con 0x7fee780b3260 2022-07-20T02:33:26.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee780b3640 con 0x7fee78061860 2022-07-20T02:33:26.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7dd37700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fee780b3260 0x7fee780979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40666/0 (socket says 172.21.15.26:40666) 2022-07-20T02:33:26.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7dd37700 1 -- 172.21.15.26:0/4203038538 learned_addr learned my addr 172.21.15.26:0/4203038538 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:26.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7d536700 1 --2- 172.21.15.26:0/4203038538 >> v2:172.21.15.26:3300/0 conn(0x7fee78061860 0x7fee78067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee780601a0 con 0x7fee780980d0 2022-07-20T02:33:26.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7e538700 1 --2- 172.21.15.26:0/4203038538 >> v2:172.21.15.120:3300/0 conn(0x7fee780980d0 0x7fee78142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7cd35700 1 -- 172.21.15.26:0/4203038538 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043336192 0 0) 0x7fee7805ff70 con 0x7fee780b3260 2022-07-20T02:33:26.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.696+0000 7fee7cd35700 1 -- 172.21.15.26:0/4203038538 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60003260 con 0x7fee780b3260 2022-07-20T02:33:26.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7d536700 1 -- 172.21.15.26:0/4203038538 >> v2:172.21.15.120:3300/0 conn(0x7fee780980d0 msgr2=0x7fee78142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:26.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7d536700 1 --2- 172.21.15.26:0/4203038538 >> v2:172.21.15.120:3300/0 conn(0x7fee780980d0 0x7fee78142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7d536700 1 -- 172.21.15.26:0/4203038538 >> v1:172.21.15.26:6789/0 conn(0x7fee780b3260 legacy=0x7fee780979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7d536700 1 -- 172.21.15.26:0/4203038538 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780606e0 con 0x7fee78061860 2022-07-20T02:33:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7d536700 1 --2- 172.21.15.26:0/4203038538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee78061860 0x7fee78067900 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7fee68002f70 tx=0x7fee68006df0).ready entity=mon.0 client_cookie=e333eeb4402cd738 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7cd35700 1 -- 172.21.15.26:0/4203038538 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee6800d4f0 con 0x7fee78061860 2022-07-20T02:33:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7cd35700 1 -- 172.21.15.26:0/4203038538 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68000e90 con 0x7fee78061860 2022-07-20T02:33:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7cd35700 1 -- 172.21.15.26:0/4203038538 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee6800d840 con 0x7fee78061860 2022-07-20T02:33:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee78061860 msgr2=0x7fee78067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 --2- 172.21.15.26:0/4203038538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee78061860 0x7fee78067900 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7fee68002f70 tx=0x7fee68006df0).stop 2022-07-20T02:33:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 shutdown_connections 2022-07-20T02:33:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 --2- 172.21.15.26:0/4203038538 >> v2:172.21.15.120:3300/0 conn(0x7fee780980d0 0x7fee78142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 --2- 172.21.15.26:0/4203038538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee78061860 0x7fee78067900 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 >> 172.21.15.26:0/4203038538 conn(0x7fee780b67f0 msgr2=0x7fee780b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 shutdown_connections 2022-07-20T02:33:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- 172.21.15.26:0/4203038538 wait complete. 2022-07-20T02:33:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 Processor -- start 2022-07-20T02:33:26.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.700+0000 7fee7ed39700 1 -- start start 2022-07-20T02:33:26.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee78061860 0x7fee780655e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee780b3260 0x7fee78065da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee78153850 con 0x7fee780980d0 2022-07-20T02:33:26.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee78153a80 con 0x7fee780b3260 2022-07-20T02:33:26.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee78153d20 con 0x7fee78061860 2022-07-20T02:33:26.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41394/0 (socket says 172.21.15.26:41394) 2022-07-20T02:33:26.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 -- 172.21.15.26:0/841755320 learned_addr learned my addr 172.21.15.26:0/841755320 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:26.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7dd37700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee78061860 0x7fee780655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7e538700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee780b3260 0x7fee78065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 -- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee78061860 msgr2=0x7fee780655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee78061860 0x7fee780655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 -- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee780b3260 msgr2=0x7fee78065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee780b3260 0x7fee78065da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 -- 172.21.15.26:0/841755320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6800b180 con 0x7fee780980d0 2022-07-20T02:33:26.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7d536700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fee68002f70 tx=0x7fee68014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee68013e90 con 0x7fee780980d0 2022-07-20T02:33:26.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6800de60 con 0x7fee780980d0 2022-07-20T02:33:26.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee78153fc0 con 0x7fee780980d0 2022-07-20T02:33:26.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee6801e5f0 con 0x7fee780980d0 2022-07-20T02:33:26.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee78090580 con 0x7fee780980d0 2022-07-20T02:33:26.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee780574e0 con 0x7fee780980d0 2022-07-20T02:33:26.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee68025030 con 0x7fee780980d0 2022-07-20T02:33:26.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.704+0000 7fee66ffd700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 0x7fee58055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:26.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.708+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fee6806cc30 con 0x7fee780980d0 2022-07-20T02:33:26.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.708+0000 7fee7dd37700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 0x7fee58055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:26.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.708+0000 7fee7dd37700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 0x7fee58055ed0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fee7400d000 tx=0x7fee74009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.712+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee680439f0 con 0x7fee780980d0 2022-07-20T02:33:26.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.840+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee7806afc0 con 0x7fee780980d0 2022-07-20T02:33:26.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.840+0000 7fee66ffd700 1 -- 172.21.15.26:0/841755320 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16346 (secure 0 0 0) 0x7fee68023070 con 0x7fee780980d0 2022-07-20T02:33:26.961 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:26.967 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":108,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:26.130485+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:23.691619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:24.900742+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 msgr2=0x7fee58055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 0x7fee58055ed0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fee7400d000 tx=0x7fee74009040).stop 2022-07-20T02:33:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 msgr2=0x7fee78090090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fee68002f70 tx=0x7fee68014000).stop 2022-07-20T02:33:26.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 shutdown_connections 2022-07-20T02:33:26.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee780980d0 0x7fee78090090 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fee58053a10 0x7fee58055ed0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee780b3260 0x7fee78065da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:26.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 --2- 172.21.15.26:0/841755320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee78061860 0x7fee780655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 >> 172.21.15.26:0/841755320 conn(0x7fee780b67f0 msgr2=0x7fee780b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 shutdown_connections 2022-07-20T02:33:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:26.844+0000 7fee7ed39700 1 -- 172.21.15.26:0/841755320 wait complete. 2022-07-20T02:33:27.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:33:27.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.124+0000 7f4a3b390700 1 Processor -- start 2022-07-20T02:33:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- start start 2022-07-20T02:33:27.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 0x7f4a341407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4a340b1160 0x7f4a34140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3405d3b0 con 0x7f4a34146e80 2022-07-20T02:33:27.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a341435b0 con 0x7f4a340b1160 2022-07-20T02:33:27.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a340b4bc0 con 0x7f4a3405cfd0 2022-07-20T02:33:27.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4a340b1160 0x7f4a34140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4a340b1160 0x7f4a34140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41400/0 (socket says 172.21.15.26:41400) 2022-07-20T02:33:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 -- 172.21.15.26:0/719692677 learned_addr learned my addr 172.21.15.26:0/719692677 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3912c700 1 --2- 172.21.15.26:0/719692677 >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 0x7f4a341407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 -- 172.21.15.26:0/719692677 >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 msgr2=0x7f4a341407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 --2- 172.21.15.26:0/719692677 >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 0x7f4a341407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 -- 172.21.15.26:0/719692677 >> v1:172.21.15.120:6789/0 conn(0x7f4a34146e80 legacy=0x7f4a34147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 -- 172.21.15.26:0/719692677 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a340b4e60 con 0x7f4a340b1160 2022-07-20T02:33:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3912c700 1 --2- 172.21.15.26:0/719692677 >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 0x7f4a341407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3892b700 1 --2- 172.21.15.26:0/719692677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a340b1160 0x7f4a34140cb0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f4a30002f70 tx=0x7f4a30006eb0).ready entity=mon.0 client_cookie=dbe9bedd2e3ff698 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a2bfff700 1 -- 172.21.15.26:0/719692677 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a3000d610 con 0x7f4a340b1160 2022-07-20T02:33:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a2bfff700 1 -- 172.21.15.26:0/719692677 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a30000e90 con 0x7f4a340b1160 2022-07-20T02:33:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a2bfff700 1 -- 172.21.15.26:0/719692677 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a3000d9a0 con 0x7f4a340b1160 2022-07-20T02:33:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- 172.21.15.26:0/719692677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a340b1160 msgr2=0x7f4a34140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 --2- 172.21.15.26:0/719692677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a340b1160 0x7f4a34140cb0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f4a30002f70 tx=0x7f4a30006eb0).stop 2022-07-20T02:33:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- 172.21.15.26:0/719692677 shutdown_connections 2022-07-20T02:33:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 --2- 172.21.15.26:0/719692677 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a340b1160 0x7f4a34140cb0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 --2- 172.21.15.26:0/719692677 >> v2:172.21.15.120:3300/0 conn(0x7f4a3405cfd0 0x7f4a341407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- 172.21.15.26:0/719692677 >> 172.21.15.26:0/719692677 conn(0x7f4a34066f70 msgr2=0x7f4a3405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- 172.21.15.26:0/719692677 shutdown_connections 2022-07-20T02:33:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.128+0000 7f4a3b390700 1 -- 172.21.15.26:0/719692677 wait complete. 2022-07-20T02:33:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 Processor -- start 2022-07-20T02:33:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 -- start start 2022-07-20T02:33:27.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 0x7f4a341532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 0x7f4a340af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 0x7f4a340afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a340b9690 con 0x7f4a34146e80 2022-07-20T02:33:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a34146340 con 0x7f4a3405cfd0 2022-07-20T02:33:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3b390700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a340b0190 con 0x7f4a340b1160 2022-07-20T02:33:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3992d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 0x7f4a340afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3912c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 0x7f4a341532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3892b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 0x7f4a340af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3992d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 0x7f4a340afca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41410/0 (socket says 172.21.15.26:41410) 2022-07-20T02:33:27.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.132+0000 7f4a3992d700 1 -- 172.21.15.26:0/1662866252 learned_addr learned my addr 172.21.15.26:0/1662866252 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 -- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 msgr2=0x7f4a340af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 0x7f4a340af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 -- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 msgr2=0x7f4a340afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 0x7f4a340afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 -- 172.21.15.26:0/1662866252 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3000b1e0 con 0x7f4a3405cfd0 2022-07-20T02:33:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3892b700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 0x7f4a340af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.136+0000 7f4a3912c700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 0x7f4a341532c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4a24007000 tx=0x7f4a24008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:27.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a2400ce50 con 0x7f4a3405cfd0 2022-07-20T02:33:27.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a24010580 con 0x7f4a3405cfd0 2022-07-20T02:33:27.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a34146870 con 0x7f4a3405cfd0 2022-07-20T02:33:27.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a240108e0 con 0x7f4a3405cfd0 2022-07-20T02:33:27.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a34146630 con 0x7f4a3405cfd0 2022-07-20T02:33:27.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a34145c50 con 0x7f4a3405cfd0 2022-07-20T02:33:27.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.140+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4a24010a40 con 0x7f4a3405cfd0 2022-07-20T02:33:27.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.144+0000 7f4a2a7fc700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 0x7f4a1c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:27.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.144+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4a2406bb50 con 0x7f4a3405cfd0 2022-07-20T02:33:27.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.144+0000 7f4a3892b700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 0x7f4a1c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:27.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.144+0000 7f4a3892b700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 0x7f4a1c055ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f4a3001c000 tx=0x7f4a30015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:27.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.148+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a34145c50 con 0x7f4a3405cfd0 2022-07-20T02:33:27.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:27.312+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4a340af550 con 0x7f4a3405cfd0 2022-07-20T02:33:28.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.140+0000 7f4a2a7fc700 1 -- 172.21.15.26:0/1662866252 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4a24046340 con 0x7f4a3405cfd0 2022-07-20T02:33:28.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 msgr2=0x7f4a1c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 0x7f4a1c055ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f4a3001c000 tx=0x7f4a30015000).stop 2022-07-20T02:33:28.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 msgr2=0x7f4a341532c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 0x7f4a341532c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4a24007000 tx=0x7f4a24008040).stop 2022-07-20T02:33:28.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 shutdown_connections 2022-07-20T02:33:28.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4a34146e80 0x7f4a340afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4a1c053a10 0x7f4a1c055ed0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a340b1160 0x7f4a340af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 --2- 172.21.15.26:0/1662866252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a3405cfd0 0x7f4a341532c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 >> 172.21.15.26:0/1662866252 conn(0x7f4a34066f70 msgr2=0x7f4a340b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:28.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 shutdown_connections 2022-07-20T02:33:28.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.144+0000 7f4a3b390700 1 -- 172.21.15.26:0/1662866252 wait complete. 2022-07-20T02:33:28.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:28.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.272+0000 7f131b57e700 1 Processor -- start 2022-07-20T02:33:28.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.272+0000 7f131b57e700 1 -- start start 2022-07-20T02:33:28.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 0x7f1314143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13141474e0 con 0x7f1314060f00 2022-07-20T02:33:28.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140b4e40 con 0x7f131405cfd0 2022-07-20T02:33:28.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131405d3b0 con 0x7f1314147000 2022-07-20T02:33:28.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131ad7d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1314060f00 0x7f131413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40698/0 (socket says 172.21.15.26:40698) 2022-07-20T02:33:28.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131ad7d700 1 -- 172.21.15.26:0/4259342332 learned_addr learned my addr 172.21.15.26:0/4259342332 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:28.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f1319d7b700 1 --2- 172.21.15.26:0/4259342332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 0x7f1314143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317567095 0 0) 0x7f13141474e0 con 0x7f1314060f00 2022-07-20T02:33:28.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc003260 con 0x7f1314060f00 2022-07-20T02:33:28.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f1319d7b700 1 -- 172.21.15.26:0/4259342332 >> v1:172.21.15.120:6789/0 conn(0x7f131405cfd0 legacy=0x7f13140b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487365010 0 0) 0x7f13140b4e40 con 0x7f131405cfd0 2022-07-20T02:33:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f1319d7b700 1 -- 172.21.15.26:0/4259342332 >> v1:172.21.15.26:6789/0 conn(0x7f1314060f00 legacy=0x7f131413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f1319d7b700 1 -- 172.21.15.26:0/4259342332 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13141431e0 con 0x7f1314147000 2022-07-20T02:33:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f1319d7b700 1 --2- 172.21.15.26:0/4259342332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 0x7f1314143bc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f130c001b80 tx=0x7f1310006df0).ready entity=mon.2 client_cookie=3675a78bf2bfcda1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f130c0019e0 con 0x7f1314147000 2022-07-20T02:33:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310000e90 con 0x7f1314147000 2022-07-20T02:33:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131957a700 1 -- 172.21.15.26:0/4259342332 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f131000d4a0 con 0x7f1314147000 2022-07-20T02:33:28.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- 172.21.15.26:0/4259342332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 msgr2=0x7f1314143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 --2- 172.21.15.26:0/4259342332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 0x7f1314143bc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f130c001b80 tx=0x7f1310006df0).stop 2022-07-20T02:33:28.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- 172.21.15.26:0/4259342332 shutdown_connections 2022-07-20T02:33:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 --2- 172.21.15.26:0/4259342332 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1314147000 0x7f1314143bc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- 172.21.15.26:0/4259342332 >> 172.21.15.26:0/4259342332 conn(0x7f13140620c0 msgr2=0x7f131405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:28.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- 172.21.15.26:0/4259342332 shutdown_connections 2022-07-20T02:33:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.276+0000 7f131b57e700 1 -- 172.21.15.26:0/4259342332 wait complete. 2022-07-20T02:33:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 Processor -- start 2022-07-20T02:33:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- start start 2022-07-20T02:33:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1314147000 0x7f1314137220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13141454f0 0x7f1314137710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314140f00 con 0x7f1314060f00 2022-07-20T02:33:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314141130 con 0x7f1314147000 2022-07-20T02:33:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13141413d0 con 0x7f13141454f0 2022-07-20T02:33:28.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131ad7d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13141454f0 0x7f1314137710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f1319d7b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1314147000 0x7f1314137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41420/0 (socket says 172.21.15.26:41420) 2022-07-20T02:33:28.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 -- 172.21.15.26:0/3895631505 learned_addr learned my addr 172.21.15.26:0/3895631505 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:28.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 -- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13141454f0 msgr2=0x7f1314137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13141454f0 0x7f1314137710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 -- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1314147000 msgr2=0x7f1314137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1314147000 0x7f1314137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 -- 172.21.15.26:0/3895631505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131000b180 con 0x7f1314060f00 2022-07-20T02:33:28.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131a57c700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f1304000fe0 tx=0x7f1304007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1304012400 con 0x7f1314060f00 2022-07-20T02:33:28.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304012560 con 0x7f1314060f00 2022-07-20T02:33:28.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1314141670 con 0x7f1314060f00 2022-07-20T02:33:28.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13040128c0 con 0x7f1314060f00 2022-07-20T02:33:28.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.280+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1314144740 con 0x7f1314060f00 2022-07-20T02:33:28.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.284+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1314143480 con 0x7f1314060f00 2022-07-20T02:33:28.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.284+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1304019030 con 0x7f1314060f00 2022-07-20T02:33:28.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.288+0000 7f13037fe700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 0x7f12f4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.288+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f130406ba20 con 0x7f1314060f00 2022-07-20T02:33:28.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.288+0000 7f1319d7b700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 0x7f12f4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.288+0000 7f1319d7b700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 0x7f12f4055ed0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f1310019000 tx=0x7f13140b1900).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.292+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1304042780 con 0x7f1314060f00 2022-07-20T02:33:28.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13140574e0 con 0x7f1314060f00 2022-07-20T02:33:28.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f13037fe700 1 -- 172.21.15.26:0/3895631505 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f1304045da0 con 0x7f1314060f00 2022-07-20T02:33:28.464 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:28.465 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:28.135776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7398":{"gid":7398,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3259008170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3259008170},{"type":"v1","addr":"172.21.15.26:6837","nonce":3259008170}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:28.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 msgr2=0x7f12f4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 0x7f12f4055ed0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f1310019000 tx=0x7f13140b1900).stop 2022-07-20T02:33:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 msgr2=0x7f1314145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f1304000fe0 tx=0x7f1304007b50).stop 2022-07-20T02:33:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 shutdown_connections 2022-07-20T02:33:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12f4053a10 0x7f12f4055ed0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13141454f0 0x7f1314137710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1314147000 0x7f1314137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.460+0000 7f131b57e700 1 --2- 172.21.15.26:0/3895631505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1314060f00 0x7f1314145f40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.464+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 >> 172.21.15.26:0/3895631505 conn(0x7f131405cfd0 msgr2=0x7f1314138880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.464+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 shutdown_connections 2022-07-20T02:33:28.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.464+0000 7f131b57e700 1 -- 172.21.15.26:0/3895631505 wait complete. 2022-07-20T02:33:28.532 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 55 2022-07-20T02:33:28.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:28.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.656+0000 7f01b8af7700 1 Processor -- start 2022-07-20T02:33:28.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.656+0000 7f01b8af7700 1 -- start start 2022-07-20T02:33:28.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f01b40b2920 0x7f01b4147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f01b4067190 0x7f01b4148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40b1bc0 con 0x7f01b4061050 2022-07-20T02:33:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b414b040 con 0x7f01b40b2920 2022-07-20T02:33:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b40a8860 con 0x7f01b4067190 2022-07-20T02:33:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b1d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f01b4061050 0x7f01b4066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40708/0 (socket says 172.21.15.26:40708) 2022-07-20T02:33:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b1d9b700 1 -- 172.21.15.26:0/2074767176 learned_addr learned my addr 172.21.15.26:0/2074767176 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 --2- 172.21.15.26:0/2074767176 >> v2:172.21.15.26:3300/0 conn(0x7f01b40b2920 0x7f01b4147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b2d9d700 1 --2- 172.21.15.26:0/2074767176 >> v2:172.21.15.120:3300/0 conn(0x7f01b4067190 0x7f01b4148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b159a700 1 -- 172.21.15.26:0/2074767176 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103414992 0 0) 0x7f01b40b1bc0 con 0x7f01b4061050 2022-07-20T02:33:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b159a700 1 -- 172.21.15.26:0/2074767176 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0198003260 con 0x7f01b4061050 2022-07-20T02:33:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 -- 172.21.15.26:0/2074767176 >> v2:172.21.15.120:3300/0 conn(0x7f01b4067190 msgr2=0x7f01b4148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 --2- 172.21.15.26:0/2074767176 >> v2:172.21.15.120:3300/0 conn(0x7f01b4067190 0x7f01b4148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 -- 172.21.15.26:0/2074767176 >> v1:172.21.15.26:6789/0 conn(0x7f01b4061050 legacy=0x7f01b4066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 -- 172.21.15.26:0/2074767176 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4147700 con 0x7f01b40b2920 2022-07-20T02:33:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b259c700 1 --2- 172.21.15.26:0/2074767176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b40b2920 0x7f01b4147d90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7f01a8002f70 tx=0x7f01a8006df0).ready entity=mon.0 client_cookie=ebf599c262b0bde5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b159a700 1 -- 172.21.15.26:0/2074767176 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01a800d4f0 con 0x7f01b40b2920 2022-07-20T02:33:28.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b159a700 1 -- 172.21.15.26:0/2074767176 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a8000e90 con 0x7f01b40b2920 2022-07-20T02:33:28.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b159a700 1 -- 172.21.15.26:0/2074767176 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01a800d840 con 0x7f01b40b2920 2022-07-20T02:33:28.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- 172.21.15.26:0/2074767176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b40b2920 msgr2=0x7f01b4147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 --2- 172.21.15.26:0/2074767176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b40b2920 0x7f01b4147d90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7f01a8002f70 tx=0x7f01a8006df0).stop 2022-07-20T02:33:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- 172.21.15.26:0/2074767176 shutdown_connections 2022-07-20T02:33:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 --2- 172.21.15.26:0/2074767176 >> v2:172.21.15.120:3300/0 conn(0x7f01b4067190 0x7f01b4148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 --2- 172.21.15.26:0/2074767176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b40b2920 0x7f01b4147d90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- 172.21.15.26:0/2074767176 >> 172.21.15.26:0/2074767176 conn(0x7f01b40b60a0 msgr2=0x7f01b40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:28.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- 172.21.15.26:0/2074767176 shutdown_connections 2022-07-20T02:33:28.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.660+0000 7f01b8af7700 1 -- 172.21.15.26:0/2074767176 wait complete. 2022-07-20T02:33:28.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 Processor -- start 2022-07-20T02:33:28.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- start start 2022-07-20T02:33:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01b4061050 0x7f01b40a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b40b2920 0x7f01b4156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b40643e0 con 0x7f01b4067190 2022-07-20T02:33:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4065900 con 0x7f01b40b2920 2022-07-20T02:33:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4065ba0 con 0x7f01b4061050 2022-07-20T02:33:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41434/0 (socket says 172.21.15.26:41434) 2022-07-20T02:33:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 -- 172.21.15.26:0/651322158 learned_addr learned my addr 172.21.15.26:0/651322158 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b259c700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01b4061050 0x7f01b40a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b2d9d700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b40b2920 0x7f01b4156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 -- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01b4061050 msgr2=0x7f01b40a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01b4061050 0x7f01b40a9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 -- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b40b2920 msgr2=0x7f01b4156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b40b2920 0x7f01b4156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 -- 172.21.15.26:0/651322158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a800b180 con 0x7f01b4067190 2022-07-20T02:33:28.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b1d9b700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f01a4001490 tx=0x7f01a4007ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01a4005ba0 con 0x7f01b4067190 2022-07-20T02:33:28.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a4005d00 con 0x7f01b4067190 2022-07-20T02:33:28.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b40aaf30 con 0x7f01b4067190 2022-07-20T02:33:28.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01a40072c0 con 0x7f01b4067190 2022-07-20T02:33:28.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.664+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b40a9a70 con 0x7f01b4067190 2022-07-20T02:33:28.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b414ada0 con 0x7f01b4067190 2022-07-20T02:33:28.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f01a4010070 con 0x7f01b4067190 2022-07-20T02:33:28.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01a37fe700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 0x7f0190055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:28.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01b259c700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 0x7f0190055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:28.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f01a406b920 con 0x7f01b4067190 2022-07-20T02:33:28.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.668+0000 7f01b259c700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 0x7f0190055e40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f01a801a000 tx=0x7f01a8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:28.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.676+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01a4042600 con 0x7f01b4067190 2022-07-20T02:33:28.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.800+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b40574e0 con 0x7f01b4067190 2022-07-20T02:33:28.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01a37fe700 1 -- 172.21.15.26:0/651322158 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f01a4018070 con 0x7f01b4067190 2022-07-20T02:33:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:28.808 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":109,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:27.133631+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:23.691619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:24.900742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 msgr2=0x7f0190055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 0x7f0190055e40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f01a801a000 tx=0x7f01a8014000).stop 2022-07-20T02:33:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 msgr2=0x7f01b4156920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f01a4001490 tx=0x7f01a4007ea0).stop 2022-07-20T02:33:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 shutdown_connections 2022-07-20T02:33:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01b4067190 0x7f01b4156920 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0190053980 0x7f0190055e40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b40b2920 0x7f01b4156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 --2- 172.21.15.26:0/651322158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01b4061050 0x7f01b40a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:28.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 >> 172.21.15.26:0/651322158 conn(0x7f01b40b60a0 msgr2=0x7f01b414b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:28.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 shutdown_connections 2022-07-20T02:33:28.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:28.804+0000 7f01b8af7700 1 -- 172.21.15.26:0/651322158 wait complete. 2022-07-20T02:33:28.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 Processor -- start 2022-07-20T02:33:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 -- start start 2022-07-20T02:33:29.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d0c062710 0x7f5d0c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0ba060 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c061f80 con 0x7f5d0c0b5b70 2022-07-20T02:33:29.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d14654700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c12bee0 con 0x7f5d0c062710 2022-07-20T02:33:29.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d123f0700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5d0c0b9c80 0x7f5d0c0b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40722/0 (socket says 172.21.15.26:40722) 2022-07-20T02:33:29.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d123f0700 1 -- 172.21.15.26:0/3487727334 learned_addr learned my addr 172.21.15.26:0/3487727334 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:29.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d11bef700 1 --2- 172.21.15.26:0/3487727334 >> v2:172.21.15.120:3300/0 conn(0x7f5d0c062710 0x7f5d0c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489236712 0 0) 0x7f5d0c0ba060 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003260 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252999515 0 0) 0x7f5d0c061f80 con 0x7f5d0c0b5b70 2022-07-20T02:33:29.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d0c0ba060 con 0x7f5d0c0b5b70 2022-07-20T02:33:29.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3373665425 0 0) 0x7f5cf4003260 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 >> v2:172.21.15.120:3300/0 conn(0x7f5d0c062710 msgr2=0x7f5d0c1431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:29.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 --2- 172.21.15.26:0/3487727334 >> v2:172.21.15.120:3300/0 conn(0x7f5d0c062710 0x7f5d0c1431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 >> v1:172.21.15.120:6789/0 conn(0x7f5d0c0b5b70 legacy=0x7f5d0c1403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c062220 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f5d08001680 con 0x7f5d0c0b9c80 2022-07-20T02:33:29.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 >> v1:172.21.15.26:6789/0 conn(0x7f5d0c0b9c80 legacy=0x7f5d0c0b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf40038e0 0x7f5cf4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 0x7f5cf4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004e70 0x7f5cf400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.024+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c12bee0 con 0x7f5cf4004500 2022-07-20T02:33:29.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400bcf0 con 0x7f5cf4004e70 2022-07-20T02:33:29.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400beb0 con 0x7f5cf40038e0 2022-07-20T02:33:29.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 0x7f5cf4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d123f0700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf40038e0 0x7f5cf4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d12bf1700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004e70 0x7f5cf400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 -- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf40038e0 msgr2=0x7f5cf4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:29.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf40038e0 0x7f5cf4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 -- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004e70 msgr2=0x7f5cf400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:29.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004e70 0x7f5cf400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 -- 172.21.15.26:0/3487727334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4003260 con 0x7f5cf4004500 2022-07-20T02:33:29.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d12bf1700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004e70 0x7f5cf400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:29.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d12bf1700 1 -- 172.21.15.26:0/3487727334 reap_dead start 2022-07-20T02:33:29.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d11bef700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 0x7f5cf4004980 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f5d0000c000 tx=0x7f5d00006df0).ready entity=mon.0 client_cookie=1ef3594ef90b9f3c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d0000e800 con 0x7f5cf4004500 2022-07-20T02:33:29.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0000e960 con 0x7f5cf4004500 2022-07-20T02:33:29.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d113ee700 1 -- 172.21.15.26:0/3487727334 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d0000ecc0 con 0x7f5cf4004500 2022-07-20T02:33:29.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 -- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 msgr2=0x7f5cf4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 0x7f5cf4004980 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f5d0000c000 tx=0x7f5d00006df0).stop 2022-07-20T02:33:29.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 -- 172.21.15.26:0/3487727334 shutdown_connections 2022-07-20T02:33:29.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 --2- 172.21.15.26:0/3487727334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4004500 0x7f5cf4004980 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 -- 172.21.15.26:0/3487727334 >> 172.21.15.26:0/3487727334 conn(0x7f5d0c066f70 msgr2=0x7f5d0c05faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 -- 172.21.15.26:0/3487727334 shutdown_connections 2022-07-20T02:33:29.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.028+0000 7f5d14654700 1 -- 172.21.15.26:0/3487727334 wait complete. 2022-07-20T02:33:29.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 Processor -- start 2022-07-20T02:33:29.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- start start 2022-07-20T02:33:29.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 0x7f5d0c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c152720 0x7f5d0c12b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c1438a0 con 0x7f5d0c146e80 2022-07-20T02:33:29.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c0b3890 con 0x7f5cf4004500 2022-07-20T02:33:29.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c0b2080 con 0x7f5d0c152720 2022-07-20T02:33:29.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41452/0 (socket says 172.21.15.26:41452) 2022-07-20T02:33:29.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 -- 172.21.15.26:0/2914177739 learned_addr learned my addr 172.21.15.26:0/2914177739 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:29.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d12bf1700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c152720 0x7f5d0c12b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d123f0700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 0x7f5d0c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 -- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c152720 msgr2=0x7f5d0c12b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c152720 0x7f5d0c12b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 -- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 msgr2=0x7f5d0c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 0x7f5d0c0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 -- 172.21.15.26:0/2914177739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0b22f0 con 0x7f5d0c146e80 2022-07-20T02:33:29.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d123f0700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 0x7f5d0c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:29.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d11bef700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f5d0000c000 tx=0x7f5d00019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d000245e0 con 0x7f5d0c146e80 2022-07-20T02:33:29.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00000e90 con 0x7f5d0c146e80 2022-07-20T02:33:29.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d00024930 con 0x7f5d0c146e80 2022-07-20T02:33:29.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c0b3ac0 con 0x7f5d0c146e80 2022-07-20T02:33:29.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.032+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c146740 con 0x7f5d0c146e80 2022-07-20T02:33:29.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c0b3c80 con 0x7f5d0c146e80 2022-07-20T02:33:29.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d00024a90 con 0x7f5d0c146e80 2022-07-20T02:33:29.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5cff7fe700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 0x7f5cec055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5d0006d900 con 0x7f5d0c146e80 2022-07-20T02:33:29.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5d123f0700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 0x7f5cec055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.036+0000 7f5d123f0700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 0x7f5cec055ea0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f5d08005000 tx=0x7f5d08009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.044+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d00047fe0 con 0x7f5d0c146e80 2022-07-20T02:33:29.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.212+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5d0c0b6650 con 0x7f5d0c146e80 2022-07-20T02:33:29.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.220+0000 7f5cff7fe700 1 -- 172.21.15.26:0/2914177739 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5d0000be60 con 0x7f5d0c146e80 2022-07-20T02:33:29.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 msgr2=0x7f5cec055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 0x7f5cec055ea0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f5d08005000 tx=0x7f5d08009040).stop 2022-07-20T02:33:29.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 msgr2=0x7f5d0c152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f5d0000c000 tx=0x7f5d00019000).stop 2022-07-20T02:33:29.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 shutdown_connections 2022-07-20T02:33:29.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d0c146e80 0x7f5d0c152230 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5cec0539e0 0x7f5cec055ea0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c152720 0x7f5d0c12b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 --2- 172.21.15.26:0/2914177739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf4004500 0x7f5d0c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 >> 172.21.15.26:0/2914177739 conn(0x7f5d0c066f70 msgr2=0x7f5d0c0b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 shutdown_connections 2022-07-20T02:33:29.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.224+0000 7f5d14654700 1 -- 172.21.15.26:0/2914177739 wait complete. 2022-07-20T02:33:29.229 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:33:29.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:29.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 Processor -- start 2022-07-20T02:33:29.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 -- start start 2022-07-20T02:33:29.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f55e0060f00 0x7f55e0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e01474e0 con 0x7f55e005cfd0 2022-07-20T02:33:29.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e00b4e40 con 0x7f55e0147000 2022-07-20T02:33:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e50f6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e005d3b0 con 0x7f55e0060f00 2022-07-20T02:33:29.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55dffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f55e005cfd0 0x7f55e00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40740/0 (socket says 172.21.15.26:40740) 2022-07-20T02:33:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55dffff700 1 -- 172.21.15.26:0/3599389511 learned_addr learned my addr 172.21.15.26:0/3599389511 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e48f5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f55e0060f00 0x7f55e0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e48f5700 1 -- 172.21.15.26:0/3599389511 >> v1:172.21.15.120:6789/0 conn(0x7f55e0147000 legacy=0x7f55e013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e48f5700 1 -- 172.21.15.26:0/3599389511 >> v1:172.21.15.26:6789/0 conn(0x7f55e005cfd0 legacy=0x7f55e00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e48f5700 1 -- 172.21.15.26:0/3599389511 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e01431e0 con 0x7f55e0060f00 2022-07-20T02:33:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55e48f5700 1 --2- 172.21.15.26:0/3599389511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0060f00 0x7f55e0143bc0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f55d0001110 tx=0x7f55d80091d0).ready entity=mon.0 client_cookie=74b1343068f2380 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55deffd700 1 -- 172.21.15.26:0/3599389511 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d8009ea0 con 0x7f55e0060f00 2022-07-20T02:33:29.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55deffd700 1 -- 172.21.15.26:0/3599389511 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d800b760 con 0x7f55e0060f00 2022-07-20T02:33:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.376+0000 7f55deffd700 1 -- 172.21.15.26:0/3599389511 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d800b8d0 con 0x7f55e0060f00 2022-07-20T02:33:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- 172.21.15.26:0/3599389511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0060f00 msgr2=0x7f55e0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3599389511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0060f00 0x7f55e0143bc0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f55d0001110 tx=0x7f55d80091d0).stop 2022-07-20T02:33:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- 172.21.15.26:0/3599389511 shutdown_connections 2022-07-20T02:33:29.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3599389511 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0060f00 0x7f55e0143bc0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- 172.21.15.26:0/3599389511 >> 172.21.15.26:0/3599389511 conn(0x7f55e00620c0 msgr2=0x7f55e005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- 172.21.15.26:0/3599389511 shutdown_connections 2022-07-20T02:33:29.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- 172.21.15.26:0/3599389511 wait complete. 2022-07-20T02:33:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 Processor -- start 2022-07-20T02:33:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- start start 2022-07-20T02:33:29.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e0060f00 0x7f55e01424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e0156090 0x7f55e0156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00af5f0 con 0x7f55e0147000 2022-07-20T02:33:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e0150330 con 0x7f55e0060f00 2022-07-20T02:33:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e50f6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00b97e0 con 0x7f55e0156090 2022-07-20T02:33:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55df7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55df7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41468/0 (socket says 172.21.15.26:41468) 2022-07-20T02:33:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55df7fe700 1 -- 172.21.15.26:0/3979114439 learned_addr learned my addr 172.21.15.26:0/3979114439 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55e48f5700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e0156090 0x7f55e0156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.380+0000 7f55dffff700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e0060f00 0x7f55e01424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 -- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e0156090 msgr2=0x7f55e0156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e0156090 0x7f55e0156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 -- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e0060f00 msgr2=0x7f55e01424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e0060f00 0x7f55e01424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 -- 172.21.15.26:0/3979114439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d8009470 con 0x7f55e0147000 2022-07-20T02:33:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55df7fe700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f55e0068700 tx=0x7f55d4007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d4012400 con 0x7f55e0147000 2022-07-20T02:33:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4012560 con 0x7f55e0147000 2022-07-20T02:33:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e00b9f20 con 0x7f55e0147000 2022-07-20T02:33:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d40128c0 con 0x7f55e0147000 2022-07-20T02:33:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e00b9540 con 0x7f55e0147000 2022-07-20T02:33:29.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e0143480 con 0x7f55e0147000 2022-07-20T02:33:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.384+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f55d4019030 con 0x7f55e0147000 2022-07-20T02:33:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.388+0000 7f55dd7fa700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 0x7f55c0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.388+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f55d406b3e0 con 0x7f55e0147000 2022-07-20T02:33:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.388+0000 7f55dffff700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 0x7f55c0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.388+0000 7f55dffff700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 0x7f55c0055fc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f55d000d000 tx=0x7f55d0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.392+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55d4042100 con 0x7f55e0147000 2022-07-20T02:33:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.564+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f55e00574e0 con 0x7f55e0147000 2022-07-20T02:33:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.576+0000 7f55dd7fa700 1 -- 172.21.15.26:0/3979114439 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f55d4045720 con 0x7f55e0147000 2022-07-20T02:33:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 msgr2=0x7f55c0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 0x7f55c0055fc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f55d000d000 tx=0x7f55d0009040).stop 2022-07-20T02:33:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 msgr2=0x7f55e00b9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f55e0068700 tx=0x7f55d4007c20).stop 2022-07-20T02:33:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 shutdown_connections 2022-07-20T02:33:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f55e0147000 0x7f55e00b9a30 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f55c0053b00 0x7f55c0055fc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55e0156090 0x7f55e0156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 --2- 172.21.15.26:0/3979114439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e0060f00 0x7f55e01424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 >> 172.21.15.26:0/3979114439 conn(0x7f55e005cfd0 msgr2=0x7f55e0138630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 shutdown_connections 2022-07-20T02:33:29.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.580+0000 7f55e50f6700 1 -- 172.21.15.26:0/3979114439 wait complete. 2022-07-20T02:33:29.597 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:33:29.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:29.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.720+0000 7f384a2bf700 1 Processor -- start 2022-07-20T02:33:29.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.720+0000 7f384a2bf700 1 -- start start 2022-07-20T02:33:29.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38440b2920 0x7f3844147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3844061050 0x7f3844148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440b1bc0 con 0x7f3844066a80 2022-07-20T02:33:29.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384414b040 con 0x7f3844061050 2022-07-20T02:33:29.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440a8860 con 0x7f38440b2920 2022-07-20T02:33:29.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f3843fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38440b2920 0x7f3844147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3844061050 0x7f3844148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3844061050 0x7f3844148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:32934/0 (socket says 172.21.15.26:32934) 2022-07-20T02:33:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384885c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3844066a80 0x7f3844066ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:54968/0 (socket says 172.21.15.26:54968) 2022-07-20T02:33:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 -- 172.21.15.26:0/3083099570 learned_addr learned my addr 172.21.15.26:0/3083099570 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 -- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38440b2920 msgr2=0x7f3844147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:29.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 --2- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38440b2920 0x7f3844147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 -- 172.21.15.26:0/3083099570 >> v1:172.21.15.120:6789/0 conn(0x7f3844066a80 legacy=0x7f3844066ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 -- 172.21.15.26:0/3083099570 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844147700 con 0x7f3844061050 2022-07-20T02:33:29.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f3842ffd700 1 -- 172.21.15.26:0/3083099570 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569701038 0 0) 0x7f38440b1bc0 con 0x7f3844066a80 2022-07-20T02:33:29.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f38437fe700 1 --2- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844061050 0x7f3844148280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3834002f70 tx=0x7f3834006eb0).ready entity=mon.1 client_cookie=43ee16ec788ef224 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f3842ffd700 1 -- 172.21.15.26:0/3083099570 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383400d570 con 0x7f3844061050 2022-07-20T02:33:29.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f3842ffd700 1 -- 172.21.15.26:0/3083099570 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834000e90 con 0x7f3844061050 2022-07-20T02:33:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f3842ffd700 1 -- 172.21.15.26:0/3083099570 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383400d8c0 con 0x7f3844061050 2022-07-20T02:33:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844061050 msgr2=0x7f3844148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844061050 0x7f3844148280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3834002f70 tx=0x7f3834006eb0).stop 2022-07-20T02:33:29.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- 172.21.15.26:0/3083099570 shutdown_connections 2022-07-20T02:33:29.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844061050 0x7f3844148280 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3083099570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38440b2920 0x7f3844147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- 172.21.15.26:0/3083099570 >> 172.21.15.26:0/3083099570 conn(0x7f38440b60a0 msgr2=0x7f38440b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- 172.21.15.26:0/3083099570 shutdown_connections 2022-07-20T02:33:29.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.724+0000 7f384a2bf700 1 -- 172.21.15.26:0/3083099570 wait complete. 2022-07-20T02:33:29.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 Processor -- start 2022-07-20T02:33:29.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- start start 2022-07-20T02:33:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3844061050 0x7f38440b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 0x7f384415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440aa840 con 0x7f38440b2920 2022-07-20T02:33:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440aa300 con 0x7f3844066a80 2022-07-20T02:33:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440aa5a0 con 0x7f3844061050 2022-07-20T02:33:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f3843fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3844061050 0x7f38440b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38437fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 0x7f384415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38437fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 0x7f384415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:32942/0 (socket says 172.21.15.26:32942) 2022-07-20T02:33:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38437fe700 1 -- 172.21.15.26:0/3564782375 learned_addr learned my addr 172.21.15.26:0/3564782375 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41484/0 (socket says 172.21.15.26:41484) 2022-07-20T02:33:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 -- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3844061050 msgr2=0x7f38440b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3844061050 0x7f38440b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 -- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 msgr2=0x7f384415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 0x7f384415a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 -- 172.21.15.26:0/3564782375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383400b1e0 con 0x7f38440b2920 2022-07-20T02:33:29.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384885c700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f383c000f40 tx=0x7f383c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383c005bb0 con 0x7f38440b2920 2022-07-20T02:33:29.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c005d10 con 0x7f38440b2920 2022-07-20T02:33:29.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384415a080 con 0x7f38440b2920 2022-07-20T02:33:29.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383c0072c0 con 0x7f38440b2920 2022-07-20T02:33:29.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.728+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38440a99d0 con 0x7f38440b2920 2022-07-20T02:33:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384415a210 con 0x7f38440b2920 2022-07-20T02:33:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f383c01a060 con 0x7f38440b2920 2022-07-20T02:33:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f38417fa700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 0x7f3824055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f383c06bb40 con 0x7f38440b2920 2022-07-20T02:33:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f3843fff700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 0x7f3824055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.732+0000 7f3843fff700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 0x7f3824055e10 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f383800b000 tx=0x7f3838006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.740+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f383c042790 con 0x7f38440b2920 2022-07-20T02:33:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.900+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f384414ada0 con 0x7f38440b2920 2022-07-20T02:33:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f38417fa700 1 -- 172.21.15.26:0/3564782375 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f383c018070 con 0x7f38440b2920 2022-07-20T02:33:29.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:29.921 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:28.135776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7398":{"gid":7398,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3259008170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3259008170},{"type":"v1","addr":"172.21.15.26:6837","nonce":3259008170}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 msgr2=0x7f3824055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 0x7f3824055e10 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f383800b000 tx=0x7f3838006df0).stop 2022-07-20T02:33:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 msgr2=0x7f384415aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f383c000f40 tx=0x7f383c007d70).stop 2022-07-20T02:33:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 shutdown_connections 2022-07-20T02:33:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38440b2920 0x7f384415aa10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3824053950 0x7f3824055e10 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3844066a80 0x7f384415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 --2- 172.21.15.26:0/3564782375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3844061050 0x7f38440b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 >> 172.21.15.26:0/3564782375 conn(0x7f38440b60a0 msgr2=0x7f3844145da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 shutdown_connections 2022-07-20T02:33:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:29.904+0000 7f384a2bf700 1 -- 172.21.15.26:0/3564782375 wait complete. 2022-07-20T02:33:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 55 2022-07-20T02:33:30.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:26.130844+0000', 'modified': '2022-07-20T02:33:28.135776+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7398': {'gid': 7398, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3259008170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3259008170}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3259008170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:30.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:30.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:30.404 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7398,"want_state":"up:active","state":"up:active","rank_uptime":4.2562565589999997,"mdsmap_epoch":55,"osdmap_epoch":109,"osdmap_epoch_barrier":109,"uptime":9.5829955529999999} 2022-07-20T02:33:30.406 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7398, "mdsmap_epoch": 55, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "rank_uptime": 4.256256559, "state": "up:active", "uptime": 9.582995553, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:30.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.524+0000 7f84c9398700 1 Processor -- start 2022-07-20T02:33:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 -- start start 2022-07-20T02:33:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f84c40b2ab0 0x7f84c4061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f84c40af6d0 0x7f84c40afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40aa1e0 con 0x7f84c40add30 2022-07-20T02:33:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c40ae110 con 0x7f84c40af6d0 2022-07-20T02:33:30.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c9398700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c40a9f40 con 0x7f84c40b2ab0 2022-07-20T02:33:30.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f84c40af6d0 0x7f84c40afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f84c40af6d0 0x7f84c40afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41492/0 (socket says 172.21.15.26:41492) 2022-07-20T02:33:30.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 -- 172.21.15.26:0/4178106503 learned_addr learned my addr 172.21.15.26:0/4178106503 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:30.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84bb7fe700 1 --2- 172.21.15.26:0/4178106503 >> v2:172.21.15.120:3300/0 conn(0x7f84c40b2ab0 0x7f84c4061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 -- 172.21.15.26:0/4178106503 >> v2:172.21.15.120:3300/0 conn(0x7f84c40b2ab0 msgr2=0x7f84c4061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 --2- 172.21.15.26:0/4178106503 >> v2:172.21.15.120:3300/0 conn(0x7f84c40b2ab0 0x7f84c4061060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 -- 172.21.15.26:0/4178106503 >> v1:172.21.15.120:6789/0 conn(0x7f84c40add30 legacy=0x7f84c4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 -- 172.21.15.26:0/4178106503 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40b1110 con 0x7f84c40af6d0 2022-07-20T02:33:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c8b97700 1 --2- 172.21.15.26:0/4178106503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c40afb60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f84bc0032f0 tx=0x7f84bc009290).ready entity=mon.0 client_cookie=93c048479a93de44 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c37fe700 1 -- 172.21.15.26:0/4178106503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84bc00b870 con 0x7f84c40af6d0 2022-07-20T02:33:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c37fe700 1 -- 172.21.15.26:0/4178106503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc00b9d0 con 0x7f84c40af6d0 2022-07-20T02:33:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.528+0000 7f84c37fe700 1 -- 172.21.15.26:0/4178106503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84bc00bb40 con 0x7f84c40af6d0 2022-07-20T02:33:30.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- 172.21.15.26:0/4178106503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 msgr2=0x7f84c40afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 --2- 172.21.15.26:0/4178106503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c40afb60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f84bc0032f0 tx=0x7f84bc009290).stop 2022-07-20T02:33:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- 172.21.15.26:0/4178106503 shutdown_connections 2022-07-20T02:33:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 --2- 172.21.15.26:0/4178106503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c40afb60 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 --2- 172.21.15.26:0/4178106503 >> v2:172.21.15.120:3300/0 conn(0x7f84c40b2ab0 0x7f84c4061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- 172.21.15.26:0/4178106503 >> 172.21.15.26:0/4178106503 conn(0x7f84c40b3050 msgr2=0x7f84c40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- 172.21.15.26:0/4178106503 shutdown_connections 2022-07-20T02:33:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- 172.21.15.26:0/4178106503 wait complete. 2022-07-20T02:33:30.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 Processor -- start 2022-07-20T02:33:30.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.532+0000 7f84c9398700 1 -- start start 2022-07-20T02:33:30.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 0x7f84c4145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84c40b2ab0 0x7f84c413f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40658b0 con 0x7f84c40af6d0 2022-07-20T02:33:30.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40652c0 con 0x7f84c40add30 2022-07-20T02:33:30.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40654f0 con 0x7f84c40b2ab0 2022-07-20T02:33:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84bb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84bb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41496/0 (socket says 172.21.15.26:41496) 2022-07-20T02:33:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84bb7fe700 1 -- 172.21.15.26:0/3916843249 learned_addr learned my addr 172.21.15.26:0/3916843249 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c8b97700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84c40b2ab0 0x7f84c413f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 0x7f84c4145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 -- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84c40b2ab0 msgr2=0x7f84c413f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84c40b2ab0 0x7f84c413f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 -- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 msgr2=0x7f84c413ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 -- 172.21.15.26:0/3916843249 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc009530 con 0x7f84c40add30 2022-07-20T02:33:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84bb7fe700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c3fff700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 0x7f84c4145cb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f84c414efc0 tx=0x7f84b4007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84b4012400 con 0x7f84c40add30 2022-07-20T02:33:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b4012560 con 0x7f84c40add30 2022-07-20T02:33:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c40aab60 con 0x7f84c40add30 2022-07-20T02:33:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84b40128c0 con 0x7f84c40add30 2022-07-20T02:33:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.536+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c40ab0a0 con 0x7f84c40add30 2022-07-20T02:33:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c40b13b0 con 0x7f84c40add30 2022-07-20T02:33:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84b4019030 con 0x7f84c40add30 2022-07-20T02:33:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84c1ffb700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 0x7f84a8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f84b406b470 con 0x7f84c40add30 2022-07-20T02:33:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84bb7fe700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 0x7f84a8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.540+0000 7f84bb7fe700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 0x7f84a8055ea0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f84ac00b000 tx=0x7f84ac006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.548+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84b40420a0 con 0x7f84c40add30 2022-07-20T02:33:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.672+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c40574e0 con 0x7f84c40add30 2022-07-20T02:33:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c1ffb700 1 -- 172.21.15.26:0/3916843249 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f84b40456c0 con 0x7f84c40add30 2022-07-20T02:33:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:30.706 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":109,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:27.133631+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:23.691619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:24.900742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 msgr2=0x7f84a8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 0x7f84a8055ea0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f84ac00b000 tx=0x7f84ac006df0).stop 2022-07-20T02:33:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 msgr2=0x7f84c4145cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 0x7f84c4145cb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f84c414efc0 tx=0x7f84b4007b90).stop 2022-07-20T02:33:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 shutdown_connections 2022-07-20T02:33:30.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84c40af6d0 0x7f84c413ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84a80539e0 0x7f84a8055ea0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84c40b2ab0 0x7f84c413f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 --2- 172.21.15.26:0/3916843249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84c40add30 0x7f84c4145cb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 >> 172.21.15.26:0/3916843249 conn(0x7f84c40b3050 msgr2=0x7f84c40b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.676+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 shutdown_connections 2022-07-20T02:33:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.680+0000 7f84c9398700 1 -- 172.21.15.26:0/3916843249 wait complete. 2022-07-20T02:33:30.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:33:23.691619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:33:24.900742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:30.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 Processor -- start 2022-07-20T02:33:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 -- start start 2022-07-20T02:33:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2280b5ef0 0x7fd2281474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 0x7fd2281479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280b62d0 con 0x7fd2280a8c40 2022-07-20T02:33:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd228147eb0 con 0x7fd2280b5ef0 2022-07-20T02:33:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22c966700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22814aac0 con 0x7fd2280a7fc0 2022-07-20T02:33:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 0x7fd2281479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22759e700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd2280a8c40 0x7fd2280aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:54998/0 (socket says 172.21.15.26:54998) 2022-07-20T02:33:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22759e700 1 -- 172.21.15.26:0/2918125433 learned_addr learned my addr 172.21.15.26:0/2918125433 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd226d9d700 1 --2- 172.21.15.26:0/2918125433 >> v2:172.21.15.120:3300/0 conn(0x7fd2280b5ef0 0x7fd2281474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd225d9b700 1 -- 172.21.15.26:0/2918125433 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353817787 0 0) 0x7fd2280b62d0 con 0x7fd2280a8c40 2022-07-20T02:33:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd225d9b700 1 -- 172.21.15.26:0/2918125433 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd20c003260 con 0x7fd2280a8c40 2022-07-20T02:33:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 -- 172.21.15.26:0/2918125433 >> v2:172.21.15.120:3300/0 conn(0x7fd2280b5ef0 msgr2=0x7fd2281474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 --2- 172.21.15.26:0/2918125433 >> v2:172.21.15.120:3300/0 conn(0x7fd2280b5ef0 0x7fd2281474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 -- 172.21.15.26:0/2918125433 >> v1:172.21.15.120:6789/0 conn(0x7fd2280a8c40 legacy=0x7fd2280aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 -- 172.21.15.26:0/2918125433 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22814ad60 con 0x7fd2280a7fc0 2022-07-20T02:33:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.836+0000 7fd22659c700 1 --2- 172.21.15.26:0/2918125433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 0x7fd2281479c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd2180032f0 tx=0x7fd2180091d0).ready entity=mon.2 client_cookie=126539239d800348 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd225d9b700 1 -- 172.21.15.26:0/2918125433 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21800b600 con 0x7fd2280a7fc0 2022-07-20T02:33:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd225d9b700 1 -- 172.21.15.26:0/2918125433 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21800b760 con 0x7fd2280a7fc0 2022-07-20T02:33:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd225d9b700 1 -- 172.21.15.26:0/2918125433 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21800b8d0 con 0x7fd2280a7fc0 2022-07-20T02:33:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- 172.21.15.26:0/2918125433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 msgr2=0x7fd2281479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- 172.21.15.26:0/2918125433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 0x7fd2281479c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd2180032f0 tx=0x7fd2180091d0).stop 2022-07-20T02:33:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- 172.21.15.26:0/2918125433 shutdown_connections 2022-07-20T02:33:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- 172.21.15.26:0/2918125433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a7fc0 0x7fd2281479c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- 172.21.15.26:0/2918125433 >> v2:172.21.15.120:3300/0 conn(0x7fd2280b5ef0 0x7fd2281474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- 172.21.15.26:0/2918125433 >> 172.21.15.26:0/2918125433 conn(0x7fd228060310 msgr2=0x7fd2280b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:30.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- 172.21.15.26:0/2918125433 shutdown_connections 2022-07-20T02:33:30.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- 172.21.15.26:0/2918125433 wait complete. 2022-07-20T02:33:30.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 Processor -- start 2022-07-20T02:33:30.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- start start 2022-07-20T02:33:30.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2280a7fc0 0x7fd22815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a8c40 0x7fd22811d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22815aa00 con 0x7fd2280b5ef0 2022-07-20T02:33:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22811d5e0 con 0x7fd2280a7fc0 2022-07-20T02:33:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22c966700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22811d870 con 0x7fd2280a8c40 2022-07-20T02:33:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41512/0 (socket says 172.21.15.26:41512) 2022-07-20T02:33:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22759e700 1 -- 172.21.15.26:0/1494277756 learned_addr learned my addr 172.21.15.26:0/1494277756 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd22659c700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a8c40 0x7fd22811d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.840+0000 7fd226d9d700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2280a7fc0 0x7fd22815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 -- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a8c40 msgr2=0x7fd22811d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a8c40 0x7fd22811d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 -- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2280a7fc0 msgr2=0x7fd22815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2280a7fc0 0x7fd22815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 -- 172.21.15.26:0/1494277756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd218009470 con 0x7fd2280b5ef0 2022-07-20T02:33:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22759e700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fd220001110 tx=0x7fd220007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd22000ce80 con 0x7fd2280b5ef0 2022-07-20T02:33:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd228159b30 con 0x7fd2280b5ef0 2022-07-20T02:33:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2200070d0 con 0x7fd2280b5ef0 2022-07-20T02:33:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd220007470 con 0x7fd2280b5ef0 2022-07-20T02:33:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22815a070 con 0x7fd2280b5ef0 2022-07-20T02:33:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.844+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22814b000 con 0x7fd2280b5ef0 2022-07-20T02:33:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.848+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd220010070 con 0x7fd2280b5ef0 2022-07-20T02:33:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.848+0000 7fd217fff700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 0x7fd204055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:30.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.848+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd22006b750 con 0x7fd2280b5ef0 2022-07-20T02:33:30.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.848+0000 7fd226d9d700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 0x7fd204055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.848+0000 7fd226d9d700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 0x7fd204055ea0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fd21c00b000 tx=0x7fd21c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.856+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd220042380 con 0x7fd2280b5ef0 2022-07-20T02:33:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.984+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd2280574e0 con 0x7fd2280b5ef0 2022-07-20T02:33:30.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.984+0000 7fd217fff700 1 -- 172.21.15.26:0/1494277756 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v109) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd220018070 con 0x7fd2280b5ef0 2022-07-20T02:33:30.995 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 msgr2=0x7fd204055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 0x7fd204055ea0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fd21c00b000 tx=0x7fd21c006df0).stop 2022-07-20T02:33:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 msgr2=0x7fd228156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fd220001110 tx=0x7fd220007aa0).stop 2022-07-20T02:33:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 shutdown_connections 2022-07-20T02:33:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2280b5ef0 0x7fd228156560 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd2040539e0 0x7fd204055ea0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2280a8c40 0x7fd22811d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 --2- 172.21.15.26:0/1494277756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2280a7fc0 0x7fd22815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 >> 172.21.15.26:0/1494277756 conn(0x7fd228060310 msgr2=0x7fd2280b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 shutdown_connections 2022-07-20T02:33:31.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:30.988+0000 7fd22c966700 1 -- 172.21.15.26:0/1494277756 wait complete. 2022-07-20T02:33:31.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.208+0000 7fdf31684700 1 Processor -- start 2022-07-20T02:33:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.208+0000 7fdf31684700 1 -- start start 2022-07-20T02:33:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf2c0b9c80 0x7fdf2c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0ba060 con 0x7fdf2c062710 2022-07-20T02:33:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c061f80 con 0x7fdf2c0b5b70 2022-07-20T02:33:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c12bee0 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2a7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdf2c062710 0x7fdf2c0b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40798/0 (socket says 172.21.15.26:40798) 2022-07-20T02:33:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2a7fc700 1 -- 172.21.15.26:0/2656523982 learned_addr learned my addr 172.21.15.26:0/2656523982 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2affd700 1 --2- 172.21.15.26:0/2656523982 >> v2:172.21.15.120:3300/0 conn(0x7fdf2c0b9c80 0x7fdf2c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf29ffb700 1 -- 172.21.15.26:0/2656523982 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490821577 0 0) 0x7fdf2c0ba060 con 0x7fdf2c062710 2022-07-20T02:33:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf29ffb700 1 -- 172.21.15.26:0/2656523982 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c003260 con 0x7fdf2c062710 2022-07-20T02:33:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2affd700 1 -- 172.21.15.26:0/2656523982 >> v1:172.21.15.120:6789/0 conn(0x7fdf2c0b5b70 legacy=0x7fdf2c1403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2affd700 1 -- 172.21.15.26:0/2656523982 >> v1:172.21.15.26:6789/0 conn(0x7fdf2c062710 legacy=0x7fdf2c0b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2affd700 1 -- 172.21.15.26:0/2656523982 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c062220 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf2affd700 1 --2- 172.21.15.26:0/2656523982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c1431a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fdf1c002f70 tx=0x7fdf1c006df0).ready entity=mon.1 client_cookie=57bf3947379785c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf29ffb700 1 -- 172.21.15.26:0/2656523982 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf1c00d610 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf29ffb700 1 -- 172.21.15.26:0/2656523982 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c000e90 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf29ffb700 1 -- 172.21.15.26:0/2656523982 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf1c00d960 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 -- 172.21.15.26:0/2656523982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 msgr2=0x7fdf2c1431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.212+0000 7fdf31684700 1 --2- 172.21.15.26:0/2656523982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c1431a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fdf1c002f70 tx=0x7fdf1c006df0).stop 2022-07-20T02:33:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- 172.21.15.26:0/2656523982 shutdown_connections 2022-07-20T02:33:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 --2- 172.21.15.26:0/2656523982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c1431a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- 172.21.15.26:0/2656523982 >> 172.21.15.26:0/2656523982 conn(0x7fdf2c066f70 msgr2=0x7fdf2c05faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- 172.21.15.26:0/2656523982 shutdown_connections 2022-07-20T02:33:31.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- 172.21.15.26:0/2656523982 wait complete. 2022-07-20T02:33:31.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 Processor -- start 2022-07-20T02:33:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- start start 2022-07-20T02:33:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2c0b5b70 0x7fdf2c13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c12c890 con 0x7fdf2c062710 2022-07-20T02:33:31.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c12c420 con 0x7fdf2c0b9c80 2022-07-20T02:33:31.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf31684700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c1564d0 con 0x7fdf2c0b5b70 2022-07-20T02:33:31.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41522/0 (socket says 172.21.15.26:41522) 2022-07-20T02:33:31.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 -- 172.21.15.26:0/533128911 learned_addr learned my addr 172.21.15.26:0/533128911 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:31.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2b7fe700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2a7fc700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2c0b5b70 0x7fdf2c13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 -- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2c0b5b70 msgr2=0x7fdf2c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2c0b5b70 0x7fdf2c13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 -- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 msgr2=0x7fdf2c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 -- 172.21.15.26:0/533128911 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c00b180 con 0x7fdf2c062710 2022-07-20T02:33:31.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.216+0000 7fdf2affd700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fdf1c002f70 tx=0x7fdf1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf1c00d290 con 0x7fdf2c062710 2022-07-20T02:33:31.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c01e400 con 0x7fdf2c062710 2022-07-20T02:33:31.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c156770 con 0x7fdf2c062710 2022-07-20T02:33:31.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf1c01e760 con 0x7fdf2c062710 2022-07-20T02:33:31.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c13f830 con 0x7fdf2c062710 2022-07-20T02:33:31.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c156310 con 0x7fdf2c062710 2022-07-20T02:33:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdf1c025030 con 0x7fdf2c062710 2022-07-20T02:33:31.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 0x7fdf10055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdf1c06cf60 con 0x7fdf2c062710 2022-07-20T02:33:31.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf2a7fc700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 0x7fdf10055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.220+0000 7fdf2a7fc700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 0x7fdf10055f60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fdf1400b000 tx=0x7fdf14009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.228+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdf1c043d20 con 0x7fdf2c062710 2022-07-20T02:33:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.348+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdf2c152460 con 0x7fdf2c062710 2022-07-20T02:33:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.348+0000 7fdf23fff700 1 -- 172.21.15.26:0/533128911 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v109) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdf1c023070 con 0x7fdf2c062710 2022-07-20T02:33:31.360 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 msgr2=0x7fdf10055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 0x7fdf10055f60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fdf1400b000 tx=0x7fdf14009040).stop 2022-07-20T02:33:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 msgr2=0x7fdf2c0b8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fdf1c002f70 tx=0x7fdf1c014000).stop 2022-07-20T02:33:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 shutdown_connections 2022-07-20T02:33:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf10053aa0 0x7fdf10055f60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf2c0b9c80 0x7fdf2c13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2c0b5b70 0x7fdf2c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 --2- 172.21.15.26:0/533128911 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf2c062710 0x7fdf2c0b8900 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 >> 172.21.15.26:0/533128911 conn(0x7fdf2c066f70 msgr2=0x7fdf2c142500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 shutdown_connections 2022-07-20T02:33:31.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.352+0000 7fdf31684700 1 -- 172.21.15.26:0/533128911 wait complete. 2022-07-20T02:33:31.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 Processor -- start 2022-07-20T02:33:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 -- start start 2022-07-20T02:33:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6640aebc0 0x7fc6640aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66405f840 con 0x7fc66405d050 2022-07-20T02:33:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc664147720 con 0x7fc6640af490 2022-07-20T02:33:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66b1fd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66405fa70 con 0x7fc6640aebc0 2022-07-20T02:33:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6640aebc0 0x7fc6640aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc6640aebc0 0x7fc6640aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41530/0 (socket says 172.21.15.26:41530) 2022-07-20T02:33:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 -- 172.21.15.26:0/2395042295 learned_addr learned my addr 172.21.15.26:0/2395042295 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc66979a700 1 --1- 172.21.15.26:0/2395042295 >> v1:172.21.15.26:6789/0 conn(0x7fc66405d050 0x7fc664066950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40814/0 (socket says 172.21.15.26:40814) 2022-07-20T02:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 -- 172.21.15.26:0/2395042295 >> v1:172.21.15.120:6789/0 conn(0x7fc6640af490 legacy=0x7fc6640af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc6637fe700 1 -- 172.21.15.26:0/2395042295 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462357336 0 0) 0x7fc66405f840 con 0x7fc66405d050 2022-07-20T02:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 -- 172.21.15.26:0/2395042295 >> v1:172.21.15.26:6789/0 conn(0x7fc66405d050 legacy=0x7fc664066950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 -- 172.21.15.26:0/2395042295 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66405fd10 con 0x7fc6640aebc0 2022-07-20T02:33:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc668f99700 1 --2- 172.21.15.26:0/2395042295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc6640aefa0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fc6540032f0 tx=0x7fc654009290).ready entity=mon.0 client_cookie=5106caa065a80d9b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc6637fe700 1 -- 172.21.15.26:0/2395042295 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65400b8a0 con 0x7fc6640aebc0 2022-07-20T02:33:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc6637fe700 1 -- 172.21.15.26:0/2395042295 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65400ba00 con 0x7fc6640aebc0 2022-07-20T02:33:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.488+0000 7fc6637fe700 1 -- 172.21.15.26:0/2395042295 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65400bb70 con 0x7fc6640aebc0 2022-07-20T02:33:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- 172.21.15.26:0/2395042295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 msgr2=0x7fc6640aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/2395042295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc6640aefa0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fc6540032f0 tx=0x7fc654009290).stop 2022-07-20T02:33:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- 172.21.15.26:0/2395042295 shutdown_connections 2022-07-20T02:33:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/2395042295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc6640aefa0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- 172.21.15.26:0/2395042295 >> 172.21.15.26:0/2395042295 conn(0x7fc664060950 msgr2=0x7fc664060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- 172.21.15.26:0/2395042295 shutdown_connections 2022-07-20T02:33:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- 172.21.15.26:0/2395042295 wait complete. 2022-07-20T02:33:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 Processor -- start 2022-07-20T02:33:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- start start 2022-07-20T02:33:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc66405d050 0x7fc664064de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6640af490 0x7fc6640a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640b5d10 con 0x7fc6640aebc0 2022-07-20T02:33:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640652d0 con 0x7fc66405d050 2022-07-20T02:33:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66b1fd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc664066610 con 0x7fc6640af490 2022-07-20T02:33:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc663fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc663fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41538/0 (socket says 172.21.15.26:41538) 2022-07-20T02:33:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc663fff700 1 -- 172.21.15.26:0/3163235209 learned_addr learned my addr 172.21.15.26:0/3163235209 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc668f99700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc66405d050 0x7fc664064de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc66979a700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6640af490 0x7fc6640a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc663fff700 1 -- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6640af490 msgr2=0x7fc6640a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.492+0000 7fc663fff700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6640af490 0x7fc6640a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc663fff700 1 -- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc66405d050 msgr2=0x7fc664064de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc663fff700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc66405d050 0x7fc664064de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc663fff700 1 -- 172.21.15.26:0/3163235209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc654009530 con 0x7fc6640aebc0 2022-07-20T02:33:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc663fff700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fc664065410 tx=0x7fc64c007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc64c012400 con 0x7fc6640aebc0 2022-07-20T02:33:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c012560 con 0x7fc6640aebc0 2022-07-20T02:33:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6640668b0 con 0x7fc6640aebc0 2022-07-20T02:33:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc64c0128c0 con 0x7fc6640aebc0 2022-07-20T02:33:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc664063920 con 0x7fc6640aebc0 2022-07-20T02:33:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc664066c30 con 0x7fc6640aebc0 2022-07-20T02:33:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.496+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc64c019030 con 0x7fc6640aebc0 2022-07-20T02:33:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.500+0000 7fc661ffb700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 0x7fc648055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.500+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc64c06b2a0 con 0x7fc6640aebc0 2022-07-20T02:33:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.500+0000 7fc668f99700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 0x7fc648055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.500+0000 7fc668f99700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 0x7fc648055f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fc65401b000 tx=0x7fc654016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.504+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc64c042060 con 0x7fc6640aebc0 2022-07-20T02:33:31.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.632+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc6640574e0 con 0x7fc6640aebc0 2022-07-20T02:33:31.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.632+0000 7fc661ffb700 1 -- 172.21.15.26:0/3163235209 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc64c045680 con 0x7fc6640aebc0 2022-07-20T02:33:31.634 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:31.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 msgr2=0x7fc648055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 0x7fc648055f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fc65401b000 tx=0x7fc654016000).stop 2022-07-20T02:33:31.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 msgr2=0x7fc664146160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fc664065410 tx=0x7fc64c007ca0).stop 2022-07-20T02:33:31.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 shutdown_connections 2022-07-20T02:33:31.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc6640aebc0 0x7fc664146160 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc648053a70 0x7fc648055f30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6640af490 0x7fc6640a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 --2- 172.21.15.26:0/3163235209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc66405d050 0x7fc664064de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 >> 172.21.15.26:0/3163235209 conn(0x7fc664060950 msgr2=0x7fc664144c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 shutdown_connections 2022-07-20T02:33:31.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.636+0000 7fc66b1fd700 1 -- 172.21.15.26:0/3163235209 wait complete. 2022-07-20T02:33:31.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.764+0000 7fc95b59e700 1 Processor -- start 2022-07-20T02:33:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 -- start start 2022-07-20T02:33:31.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc95c096fe0 0x7fc95c143350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 0x7fc95c0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0973c0 con 0x7fc95c05f840 2022-07-20T02:33:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95c05cc90 con 0x7fc95c096fe0 2022-07-20T02:33:31.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95c142c50 con 0x7fc95c0952f0 2022-07-20T02:33:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc95c096fe0 0x7fc95c143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95ad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc95c05f840 0x7fc95c05c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40826/0 (socket says 172.21.15.26:40826) 2022-07-20T02:33:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95ad9d700 1 -- 172.21.15.26:0/182881552 learned_addr learned my addr 172.21.15.26:0/182881552 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc951d9b700 1 --2- 172.21.15.26:0/182881552 >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 0x7fc95c0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc959d9b700 1 -- 172.21.15.26:0/182881552 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201555908 0 0) 0x7fc95c0973c0 con 0x7fc95c05f840 2022-07-20T02:33:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc959d9b700 1 -- 172.21.15.26:0/182881552 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc93c003260 con 0x7fc95c05f840 2022-07-20T02:33:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 -- 172.21.15.26:0/182881552 >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 msgr2=0x7fc95c0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 --2- 172.21.15.26:0/182881552 >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 0x7fc95c0b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 -- 172.21.15.26:0/182881552 >> v1:172.21.15.26:6789/0 conn(0x7fc95c05f840 legacy=0x7fc95c05c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 -- 172.21.15.26:0/182881552 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0b2f30 con 0x7fc95c096fe0 2022-07-20T02:33:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc951d9b700 1 --2- 172.21.15.26:0/182881552 >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 0x7fc95c0b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc95a59c700 1 --2- 172.21.15.26:0/182881552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c143350 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7fc94c002f70 tx=0x7fc94c006df0).ready entity=mon.0 client_cookie=c3e70245d6f39206 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc959d9b700 1 -- 172.21.15.26:0/182881552 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc94c00d570 con 0x7fc95c096fe0 2022-07-20T02:33:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc959d9b700 1 -- 172.21.15.26:0/182881552 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c000e90 con 0x7fc95c096fe0 2022-07-20T02:33:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.768+0000 7fc959d9b700 1 -- 172.21.15.26:0/182881552 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc94c00d8c0 con 0x7fc95c096fe0 2022-07-20T02:33:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- 172.21.15.26:0/182881552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 msgr2=0x7fc95c143350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- 172.21.15.26:0/182881552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c143350 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7fc94c002f70 tx=0x7fc94c006df0).stop 2022-07-20T02:33:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- 172.21.15.26:0/182881552 shutdown_connections 2022-07-20T02:33:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- 172.21.15.26:0/182881552 >> v2:172.21.15.120:3300/0 conn(0x7fc95c0952f0 0x7fc95c0b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- 172.21.15.26:0/182881552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c143350 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- 172.21.15.26:0/182881552 >> 172.21.15.26:0/182881552 conn(0x7fc95c063030 msgr2=0x7fc95c063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- 172.21.15.26:0/182881552 shutdown_connections 2022-07-20T02:33:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- 172.21.15.26:0/182881552 wait complete. 2022-07-20T02:33:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 Processor -- start 2022-07-20T02:33:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- start start 2022-07-20T02:33:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 0x7fc95c0947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c0952f0 0x7fc95c063cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0b5f10 con 0x7fc95c096fe0 2022-07-20T02:33:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0b61b0 con 0x7fc95c05f840 2022-07-20T02:33:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0b6450 con 0x7fc95c0952f0 2022-07-20T02:33:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41554/0 (socket says 172.21.15.26:41554) 2022-07-20T02:33:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95ad9d700 1 -- 172.21.15.26:0/428677541 learned_addr learned my addr 172.21.15.26:0/428677541 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc951d9b700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c0952f0 0x7fc95c063cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.772+0000 7fc95a59c700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 0x7fc95c0947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 -- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c0952f0 msgr2=0x7fc95c063cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c0952f0 0x7fc95c063cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 -- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 msgr2=0x7fc95c0947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 0x7fc95c0947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 -- 172.21.15.26:0/428677541 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c00b180 con 0x7fc95c096fe0 2022-07-20T02:33:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95a59c700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 0x7fc95c0947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95ad9d700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fc954001490 tx=0x7fc954007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc954005ba0 con 0x7fc95c096fe0 2022-07-20T02:33:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc954005d00 con 0x7fc95c096fe0 2022-07-20T02:33:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c0b68f0 con 0x7fc95c096fe0 2022-07-20T02:33:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc954007300 con 0x7fc95c096fe0 2022-07-20T02:33:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c0641b0 con 0x7fc95c096fe0 2022-07-20T02:33:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c0b5d70 con 0x7fc95c096fe0 2022-07-20T02:33:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc954010030 con 0x7fc95c096fe0 2022-07-20T02:33:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.776+0000 7fc953fff700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 0x7fc940055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.780+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc95406c090 con 0x7fc95c096fe0 2022-07-20T02:33:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.780+0000 7fc95a59c700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 0x7fc940055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.780+0000 7fc95a59c700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 0x7fc940055e40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fc94c018000 tx=0x7fc94c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.784+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc95c0b5d70 con 0x7fc95c096fe0 2022-07-20T02:33:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc95c0b5d70 con 0x7fc95c096fe0 2022-07-20T02:33:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc953fff700 1 -- 172.21.15.26:0/428677541 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc95c0b5d70 con 0x7fc95c096fe0 2022-07-20T02:33:31.922 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 msgr2=0x7fc940055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 0x7fc940055e40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fc94c018000 tx=0x7fc94c014000).stop 2022-07-20T02:33:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 msgr2=0x7fc95c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.912+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fc954001490 tx=0x7fc954007e40).stop 2022-07-20T02:33:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 shutdown_connections 2022-07-20T02:33:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc95c096fe0 0x7fc95c0b4e50 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc940053980 0x7fc940055e40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c0952f0 0x7fc95c063cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 --2- 172.21.15.26:0/428677541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c05f840 0x7fc95c0947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 >> 172.21.15.26:0/428677541 conn(0x7fc95c063030 msgr2=0x7fc95c0b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 shutdown_connections 2022-07-20T02:33:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:31.916+0000 7fc95b59e700 1 -- 172.21.15.26:0/428677541 wait complete. 2022-07-20T02:33:31.936 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:31.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:32.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 Processor -- start 2022-07-20T02:33:32.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 -- start start 2022-07-20T02:33:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 0x7f04940aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04940b62d0 con 0x7f04940a7fc0 2022-07-20T02:33:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049414aac0 con 0x7f04940b5ef0 2022-07-20T02:33:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f049359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049414ad60 con 0x7f0494147be0 2022-07-20T02:33:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.040+0000 7f0492d9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f04940a7fc0 0x7f04940af330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40842/0 (socket says 172.21.15.26:40842) 2022-07-20T02:33:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0492d9d700 1 -- 172.21.15.26:0/2325868916 learned_addr learned my addr 172.21.15.26:0/2325868916 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0491d9b700 1 --2- 172.21.15.26:0/2325868916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 0x7f04940aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843653647 0 0) 0x7f04940b62d0 con 0x7f04940a7fc0 2022-07-20T02:33:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474003260 con 0x7f04940a7fc0 2022-07-20T02:33:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857389087 0 0) 0x7f049414aac0 con 0x7f04940b5ef0 2022-07-20T02:33:32.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04940b62d0 con 0x7f04940b5ef0 2022-07-20T02:33:32.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0491d9b700 1 -- 172.21.15.26:0/2325868916 >> v1:172.21.15.120:6789/0 conn(0x7f04940b5ef0 legacy=0x7f04941474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0491d9b700 1 -- 172.21.15.26:0/2325868916 >> v1:172.21.15.26:6789/0 conn(0x7f04940a7fc0 legacy=0x7f04940af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0491d9b700 1 -- 172.21.15.26:0/2325868916 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049414b000 con 0x7f0494147be0 2022-07-20T02:33:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f0491d9b700 1 --2- 172.21.15.26:0/2325868916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 0x7f04940aee40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f04840032f0 tx=0x7f04840091d0).ready entity=mon.2 client_cookie=fe2b327b9355e2b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048400b640 con 0x7f0494147be0 2022-07-20T02:33:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048400b7a0 con 0x7f0494147be0 2022-07-20T02:33:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049159a700 1 -- 172.21.15.26:0/2325868916 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048400b910 con 0x7f0494147be0 2022-07-20T02:33:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 -- 172.21.15.26:0/2325868916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 msgr2=0x7f04940aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 --2- 172.21.15.26:0/2325868916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 0x7f04940aee40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f04840032f0 tx=0x7f04840091d0).stop 2022-07-20T02:33:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 -- 172.21.15.26:0/2325868916 shutdown_connections 2022-07-20T02:33:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 --2- 172.21.15.26:0/2325868916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0494147be0 0x7f04940aee40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 -- 172.21.15.26:0/2325868916 >> 172.21.15.26:0/2325868916 conn(0x7f0494060310 msgr2=0x7f04940b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 -- 172.21.15.26:0/2325868916 shutdown_connections 2022-07-20T02:33:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 -- 172.21.15.26:0/2325868916 wait complete. 2022-07-20T02:33:32.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.044+0000 7f049359e700 1 Processor -- start 2022-07-20T02:33:32.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- start start 2022-07-20T02:33:32.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 0x7f04941566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:32.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:32.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 0x7f049414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:32.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494064560 con 0x7f0494147be0 2022-07-20T02:33:32.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494156b90 con 0x7f04940b5ef0 2022-07-20T02:33:32.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0494159df0 con 0x7f04940a7fc0 2022-07-20T02:33:32.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 0x7f04941566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:32.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 0x7f049414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:32.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0491d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:32.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0491d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33026/0 (socket says 172.21.15.26:33026) 2022-07-20T02:33:32.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 0x7f04941566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:34848/0 (socket says 172.21.15.26:34848) 2022-07-20T02:33:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0491d9b700 1 -- 172.21.15.26:0/214140762 learned_addr learned my addr 172.21.15.26:0/214140762 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:32.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 -- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 msgr2=0x7f04941566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 0x7f04941566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 -- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 msgr2=0x7f049415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 -- 172.21.15.26:0/214140762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484009470 con 0x7f0494147be0 2022-07-20T02:33:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0491d9b700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f0492d9d700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 0x7f049414a140 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f048c001110 tx=0x7f048c0079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:32.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048c00ce20 con 0x7f0494147be0 2022-07-20T02:33:32.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c0070d0 con 0x7f0494147be0 2022-07-20T02:33:32.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04941479c0 con 0x7f0494147be0 2022-07-20T02:33:32.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048c007430 con 0x7f0494147be0 2022-07-20T02:33:32.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.048+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049411dd70 con 0x7f0494147be0 2022-07-20T02:33:32.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049414b2a0 con 0x7f0494147be0 2022-07-20T02:33:32.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f048c010070 con 0x7f0494147be0 2022-07-20T02:33:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f048b7fe700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 0x7f0478055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:32.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f048c06b510 con 0x7f0494147be0 2022-07-20T02:33:32.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f049259c700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 0x7f0478055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:32.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.052+0000 7f049259c700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 0x7f0478055fc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f047c00d000 tx=0x7f047c007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:32.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.056+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f048c0422d0 con 0x7f0494147be0 2022-07-20T02:33:32.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.216+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04940574e0 con 0x7f0494147be0 2022-07-20T02:33:32.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.216+0000 7f048b7fe700 1 -- 172.21.15.26:0/214140762 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f048c0458f0 con 0x7f0494147be0 2022-07-20T02:33:32.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:32.223 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:28.135776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7398":{"gid":7398,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3259008170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3259008170},{"type":"v1","addr":"172.21.15.26:6837","nonce":3259008170}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:32.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 msgr2=0x7f0478055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 0x7f0478055fc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f047c00d000 tx=0x7f047c007a60).stop 2022-07-20T02:33:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 msgr2=0x7f049414a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:32.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 0x7f049414a140 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f048c001110 tx=0x7f048c0079e0).stop 2022-07-20T02:33:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 shutdown_connections 2022-07-20T02:33:32.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0494147be0 0x7f049414a140 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0478053b00 0x7f0478055fc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04940b5ef0 0x7f049415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 --2- 172.21.15.26:0/214140762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04940a7fc0 0x7f04941566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 >> 172.21.15.26:0/214140762 conn(0x7f0494060310 msgr2=0x7f0494067870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 shutdown_connections 2022-07-20T02:33:32.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:32.220+0000 7f049359e700 1 -- 172.21.15.26:0/214140762 wait complete. 2022-07-20T02:33:32.235 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 55 2022-07-20T02:33:32.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:26.130844+0000', 'modified': '2022-07-20T02:33:28.135776+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7398': {'gid': 7398, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3259008170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3259008170}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3259008170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:32.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:32.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:32.679 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7398,"want_state":"up:active","state":"up:active","rank_uptime":6.5323025140000004,"mdsmap_epoch":55,"osdmap_epoch":109,"osdmap_epoch_barrier":109,"uptime":11.859050354000001} 2022-07-20T02:33:32.680 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7398, "mdsmap_epoch": 55, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "rank_uptime": 6.532302514, "state": "up:active", "uptime": 11.859050354, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:32.681 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:33:32.682 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:33:32.682 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:33:32.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:32.684 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:32.711 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:32.732 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:32.733 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:32.802 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:32.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:32.803 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:32.859 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:33:32.864 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:32.880 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:32.911 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:32.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:32.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:33:32.934+0000 7f288b76b080 -1 init, newargv = 0x55bfeaea7830 newargc=9ceph-fuse[23156]: starting ceph client 2022-07-20T02:33:32.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:33:32.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23156]: starting fuse 2022-07-20T02:33:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:32.970 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:32.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:32.971 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:33.033 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:33:33.034 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:33:33.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:33.037 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:33:33.042 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:33:33.043 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:33:33.044 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:33:33.045 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:33:33.045 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:33.047 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:33:33.048 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:33:33.048 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:33:33.049 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:33.050 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:33:33.051 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:33:33.051 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:33:33.052 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:33:33.053 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:33.054 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:33:33.056 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:33:33.057 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:33:33.063 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:33:33.064 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:33:33.065 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:33.065 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:33:33.066 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:33:33.114 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23156.asok 2022-07-20T02:33:33.122 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23156.asok 2022-07-20T02:33:33.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:33.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23156.asok status 2022-07-20T02:33:33.262 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:33:33.263 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:33:33.263 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:33:33.264 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:33:33.264 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:33:33.265 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:33:33.265 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:33:33.265 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23156", 2022-07-20T02:33:33.271 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:33:33.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:33.272 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:33:33.272 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:33:33.273 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 7324, 2022-07-20T02:33:33.273 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:33:33.306 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:33:33.306 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:33:33.307 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 7324 2022-07-20T02:33:33.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:33.307 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:33.307 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:33.307 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:33.308 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 954038486 2022-07-20T02:33:33.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:33:33.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:33.308 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:33.309 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:33.309 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:33.309 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 954038486 2022-07-20T02:33:33.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:33.310 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.7324 172.21.15.159:0/954038486", 2022-07-20T02:33:33.310 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/954038486", 2022-07-20T02:33:33.310 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:33:33.310 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 55, 2022-07-20T02:33:33.311 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 109, 2022-07-20T02:33:33.311 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:33:33.311 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:33:33.311 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:33:33.311 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:33:33.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:33.329 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:33.329 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:33.400 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:33:33.400 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:33.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:33.401 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:33.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 Processor -- start 2022-07-20T02:33:33.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 -- start start 2022-07-20T02:33:33.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2884147be0 0x7f28840aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f28840a7fc0 0x7f28840af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28840b62d0 con 0x7f28840b5ef0 2022-07-20T02:33:33.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288414aac0 con 0x7f28840a7fc0 2022-07-20T02:33:33.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f2883fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288414ad60 con 0x7f2884147be0 2022-07-20T02:33:33.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f28840a7fc0 0x7f28840af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f28840a7fc0 0x7f28840af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41576/0 (socket says 172.21.15.26:41576) 2022-07-20T02:33:33.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 -- 172.21.15.26:0/1692179152 learned_addr learned my addr 172.21.15.26:0/1692179152 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28827fc700 1 --2- 172.21.15.26:0/1692179152 >> v2:172.21.15.120:3300/0 conn(0x7f2884147be0 0x7f28840aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 -- 172.21.15.26:0/1692179152 >> v2:172.21.15.120:3300/0 conn(0x7f2884147be0 msgr2=0x7f28840aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:33.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 --2- 172.21.15.26:0/1692179152 >> v2:172.21.15.120:3300/0 conn(0x7f2884147be0 0x7f28840aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 -- 172.21.15.26:0/1692179152 >> v1:172.21.15.120:6789/0 conn(0x7f28840b5ef0 legacy=0x7f28841474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.600+0000 7f28837fe700 1 -- 172.21.15.26:0/1692179152 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288414b000 con 0x7f28840a7fc0 2022-07-20T02:33:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f28837fe700 1 --2- 172.21.15.26:0/1692179152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840af330 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f2884060810 tx=0x7f287c006eb0).ready entity=mon.0 client_cookie=559a99785b3a64bd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2881ffb700 1 -- 172.21.15.26:0/1692179152 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f287c00d5c0 con 0x7f28840a7fc0 2022-07-20T02:33:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2881ffb700 1 -- 172.21.15.26:0/1692179152 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c000e90 con 0x7f28840a7fc0 2022-07-20T02:33:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2881ffb700 1 -- 172.21.15.26:0/1692179152 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f287c00d910 con 0x7f28840a7fc0 2022-07-20T02:33:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 -- 172.21.15.26:0/1692179152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 msgr2=0x7f28840af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 --2- 172.21.15.26:0/1692179152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840af330 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f2884060810 tx=0x7f287c006eb0).stop 2022-07-20T02:33:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 -- 172.21.15.26:0/1692179152 shutdown_connections 2022-07-20T02:33:33.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 --2- 172.21.15.26:0/1692179152 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840af330 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 --2- 172.21.15.26:0/1692179152 >> v2:172.21.15.120:3300/0 conn(0x7f2884147be0 0x7f28840aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 -- 172.21.15.26:0/1692179152 >> 172.21.15.26:0/1692179152 conn(0x7f2884060310 msgr2=0x7f28840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 -- 172.21.15.26:0/1692179152 shutdown_connections 2022-07-20T02:33:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.604+0000 7f2883fff700 1 -- 172.21.15.26:0/1692179152 wait complete. 2022-07-20T02:33:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 Processor -- start 2022-07-20T02:33:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- start start 2022-07-20T02:33:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28840b5ef0 0x7f2884147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2884147be0 0x7f2884156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884062790 con 0x7f28840a7fc0 2022-07-20T02:33:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28840629f0 con 0x7f28840b5ef0 2022-07-20T02:33:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2884062c90 con 0x7f2884147be0 2022-07-20T02:33:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41578/0 (socket says 172.21.15.26:41578) 2022-07-20T02:33:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 -- 172.21.15.26:0/2296606750 learned_addr learned my addr 172.21.15.26:0/2296606750 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f28827fc700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28840b5ef0 0x7f2884147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f28837fe700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2884147be0 0x7f2884156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 -- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2884147be0 msgr2=0x7f2884156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2884147be0 0x7f2884156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 -- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28840b5ef0 msgr2=0x7f2884147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28840b5ef0 0x7f2884147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 -- 172.21.15.26:0/2296606750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c00b1e0 con 0x7f28840a7fc0 2022-07-20T02:33:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2882ffd700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f2878001fa0 tx=0x7f2878007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2878007030 con 0x7f28840a7fc0 2022-07-20T02:33:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878007190 con 0x7f28840a7fc0 2022-07-20T02:33:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2884062f30 con 0x7f28840a7fc0 2022-07-20T02:33:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2878006510 con 0x7f28840a7fc0 2022-07-20T02:33:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.608+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2884063470 con 0x7f28840a7fc0 2022-07-20T02:33:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288414b2a0 con 0x7f28840a7fc0 2022-07-20T02:33:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2878010030 con 0x7f28840a7fc0 2022-07-20T02:33:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f2873fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 0x7f2860055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f287806bc60 con 0x7f28840a7fc0 2022-07-20T02:33:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f28827fc700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 0x7f2860055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.612+0000 7f28827fc700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 0x7f2860055f60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f287400c000 tx=0x7f2874008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.620+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2878042a20 con 0x7f28840a7fc0 2022-07-20T02:33:33.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.748+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28840574e0 con 0x7f28840a7fc0 2022-07-20T02:33:33.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.748+0000 7f2873fff700 1 -- 172.21.15.26:0/2296606750 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f2878046040 con 0x7f28840a7fc0 2022-07-20T02:33:33.758 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:33.758 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":109,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:27.133631+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:23.691619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:24.900742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:33.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 msgr2=0x7f2860055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 0x7f2860055f60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f287400c000 tx=0x7f2874008040).stop 2022-07-20T02:33:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 msgr2=0x7f28840ac0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f2878001fa0 tx=0x7f2878007c30).stop 2022-07-20T02:33:33.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 shutdown_connections 2022-07-20T02:33:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2860053aa0 0x7f2860055f60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2884147be0 0x7f2884156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28840b5ef0 0x7f2884147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 --2- 172.21.15.26:0/2296606750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28840a7fc0 0x7f28840ac0a0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:33.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 >> 172.21.15.26:0/2296606750 conn(0x7f2884060310 msgr2=0x7f28840b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:33.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 shutdown_connections 2022-07-20T02:33:33.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.752+0000 7f2883fff700 1 -- 172.21.15.26:0/2296606750 wait complete. 2022-07-20T02:33:33.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:33:23.691619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:33:24.900742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:33.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 Processor -- start 2022-07-20T02:33:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 -- start start 2022-07-20T02:33:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe170147be0 0x7fe1700aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1700b62d0 con 0x7fe1700b5ef0 2022-07-20T02:33:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17014aac0 con 0x7fe1700a7fc0 2022-07-20T02:33:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16f59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17014ad60 con 0x7fe170147be0 2022-07-20T02:33:33.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16e59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1700b5ef0 0x7fe1701474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40866/0 (socket says 172.21.15.26:40866) 2022-07-20T02:33:33.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16e59c700 1 -- 172.21.15.26:0/3385635095 learned_addr learned my addr 172.21.15.26:0/3385635095 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:33.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16dd9b700 1 --2- 172.21.15.26:0/3385635095 >> v2:172.21.15.120:3300/0 conn(0x7fe170147be0 0x7fe1700aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:33.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300063079 0 0) 0x7fe1700b62d0 con 0x7fe1700b5ef0 2022-07-20T02:33:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe154003260 con 0x7fe1700b5ef0 2022-07-20T02:33:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16dd9b700 1 -- 172.21.15.26:0/3385635095 >> v1:172.21.15.120:6789/0 conn(0x7fe1700a7fc0 legacy=0x7fe1700af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637617262 0 0) 0x7fe17014aac0 con 0x7fe1700a7fc0 2022-07-20T02:33:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16dd9b700 1 -- 172.21.15.26:0/3385635095 >> v1:172.21.15.26:6789/0 conn(0x7fe1700b5ef0 legacy=0x7fe1701474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:33.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.896+0000 7fe16dd9b700 1 -- 172.21.15.26:0/3385635095 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17014b000 con 0x7fe170147be0 2022-07-20T02:33:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16dd9b700 1 --2- 172.21.15.26:0/3385635095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe170147be0 0x7fe1700aee40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe160002f70 tx=0x7fe160006df0).ready entity=mon.1 client_cookie=7eb3eb1631504237 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe16000d610 con 0x7fe170147be0 2022-07-20T02:33:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160000e90 con 0x7fe170147be0 2022-07-20T02:33:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16d59a700 1 -- 172.21.15.26:0/3385635095 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe16000d960 con 0x7fe170147be0 2022-07-20T02:33:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/3385635095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe170147be0 msgr2=0x7fe1700aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 --2- 172.21.15.26:0/3385635095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe170147be0 0x7fe1700aee40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe160002f70 tx=0x7fe160006df0).stop 2022-07-20T02:33:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/3385635095 shutdown_connections 2022-07-20T02:33:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 --2- 172.21.15.26:0/3385635095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe170147be0 0x7fe1700aee40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/3385635095 >> 172.21.15.26:0/3385635095 conn(0x7fe170060310 msgr2=0x7fe1700b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/3385635095 shutdown_connections 2022-07-20T02:33:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/3385635095 wait complete. 2022-07-20T02:33:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 Processor -- start 2022-07-20T02:33:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- start start 2022-07-20T02:33:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1700a7fc0 0x7fe17014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1700b5ef0 0x7fe1701470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170064740 con 0x7fe170147be0 2022-07-20T02:33:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700abcd0 con 0x7fe1700a7fc0 2022-07-20T02:33:34.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe170064970 con 0x7fe1700b5ef0 2022-07-20T02:33:34.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41594/0 (socket says 172.21.15.26:41594) 2022-07-20T02:33:34.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 -- 172.21.15.26:0/555457382 learned_addr learned my addr 172.21.15.26:0/555457382 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:34.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16e59c700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1700a7fc0 0x7fe17014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16dd9b700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1700b5ef0 0x7fe1701470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 -- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1700b5ef0 msgr2=0x7fe1701470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1700b5ef0 0x7fe1701470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 -- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1700a7fc0 msgr2=0x7fe17014df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1700a7fc0 0x7fe17014df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 -- 172.21.15.26:0/555457382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16000b180 con 0x7fe170147be0 2022-07-20T02:33:34.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16ed9d700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7fe168000f40 tx=0x7fe168007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1680105e0 con 0x7fe170147be0 2022-07-20T02:33:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe168010740 con 0x7fe170147be0 2022-07-20T02:33:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1680108b0 con 0x7fe170147be0 2022-07-20T02:33:34.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.900+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe170064c10 con 0x7fe170147be0 2022-07-20T02:33:34.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1700abf70 con 0x7fe170147be0 2022-07-20T02:33:34.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe170064da0 con 0x7fe170147be0 2022-07-20T02:33:34.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe168010070 con 0x7fe170147be0 2022-07-20T02:33:34.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe15f7fe700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 0x7fe14c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe16806bda0 con 0x7fe170147be0 2022-07-20T02:33:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe16e59c700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 0x7fe14c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.904+0000 7fe16e59c700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 0x7fe14c055ea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fe16400b000 tx=0x7fe164009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:33.912+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe168042b60 con 0x7fe170147be0 2022-07-20T02:33:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.036+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe17014b2a0 con 0x7fe170147be0 2022-07-20T02:33:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe15f7fe700 1 -- 172.21.15.26:0/555457382 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v109) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe168046180 con 0x7fe170147be0 2022-07-20T02:33:34.124 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:34.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 msgr2=0x7fe14c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 0x7fe14c055ea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fe16400b000 tx=0x7fe164009040).stop 2022-07-20T02:33:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 msgr2=0x7fe1701475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7fe168000f40 tx=0x7fe168007e00).stop 2022-07-20T02:33:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 shutdown_connections 2022-07-20T02:33:34.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe170147be0 0x7fe1701475a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe14c0539e0 0x7fe14c055ea0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1700b5ef0 0x7fe1701470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 --2- 172.21.15.26:0/555457382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1700a7fc0 0x7fe17014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 >> 172.21.15.26:0/555457382 conn(0x7fe170060310 msgr2=0x7fe17005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 shutdown_connections 2022-07-20T02:33:34.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.040+0000 7fe16f59e700 1 -- 172.21.15.26:0/555457382 wait complete. 2022-07-20T02:33:34.142 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 Processor -- start 2022-07-20T02:33:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 -- start start 2022-07-20T02:33:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 0x7fc90c064d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc90c1432e0 0x7fc90c140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c0b2cd0 con 0x7fc90c064990 2022-07-20T02:33:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c0608d0 con 0x7fc90c1432e0 2022-07-20T02:33:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc914709700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c0b3860 con 0x7fc90c0b27a0 2022-07-20T02:33:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc912ca6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc90c1432e0 0x7fc90c140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc912ca6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc90c1432e0 0x7fc90c140680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41604/0 (socket says 172.21.15.26:41604) 2022-07-20T02:33:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc912ca6700 1 -- 172.21.15.26:0/965537955 learned_addr learned my addr 172.21.15.26:0/965537955 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.260+0000 7fc911ca4700 1 --2- 172.21.15.26:0/965537955 >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 0x7fc90c064d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc912ca6700 1 -- 172.21.15.26:0/965537955 >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 msgr2=0x7fc90c064d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc912ca6700 1 --2- 172.21.15.26:0/965537955 >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 0x7fc90c064d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc912ca6700 1 -- 172.21.15.26:0/965537955 >> v1:172.21.15.120:6789/0 conn(0x7fc90c064990 legacy=0x7fc90c060d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc912ca6700 1 -- 172.21.15.26:0/965537955 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0b2f00 con 0x7fc90c1432e0 2022-07-20T02:33:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc911ca4700 1 --2- 172.21.15.26:0/965537955 >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 0x7fc90c064d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc912ca6700 1 --2- 172.21.15.26:0/965537955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c1432e0 0x7fc90c140680 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fc90c0b8920 tx=0x7fc904009290).ready entity=mon.0 client_cookie=a5b4c734e83429d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc9114a3700 1 -- 172.21.15.26:0/965537955 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc90400b860 con 0x7fc90c1432e0 2022-07-20T02:33:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc9114a3700 1 -- 172.21.15.26:0/965537955 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90400b9c0 con 0x7fc90c1432e0 2022-07-20T02:33:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc9114a3700 1 -- 172.21.15.26:0/965537955 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc90400bb30 con 0x7fc90c1432e0 2022-07-20T02:33:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- 172.21.15.26:0/965537955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c1432e0 msgr2=0x7fc90c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 --2- 172.21.15.26:0/965537955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c1432e0 0x7fc90c140680 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fc90c0b8920 tx=0x7fc904009290).stop 2022-07-20T02:33:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- 172.21.15.26:0/965537955 shutdown_connections 2022-07-20T02:33:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 --2- 172.21.15.26:0/965537955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c1432e0 0x7fc90c140680 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 --2- 172.21.15.26:0/965537955 >> v2:172.21.15.120:3300/0 conn(0x7fc90c0b27a0 0x7fc90c064d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- 172.21.15.26:0/965537955 >> 172.21.15.26:0/965537955 conn(0x7fc90c05c580 msgr2=0x7fc90c05c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- 172.21.15.26:0/965537955 shutdown_connections 2022-07-20T02:33:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- 172.21.15.26:0/965537955 wait complete. 2022-07-20T02:33:34.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 Processor -- start 2022-07-20T02:33:34.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.264+0000 7fc914709700 1 -- start start 2022-07-20T02:33:34.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 0x7fc90c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 0x7fc90c14fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0ad150 con 0x7fc90c0b27a0 2022-07-20T02:33:34.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0abf80 con 0x7fc90c1432e0 2022-07-20T02:33:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c150310 con 0x7fc90c064990 2022-07-20T02:33:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41608/0 (socket says 172.21.15.26:41608) 2022-07-20T02:33:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 -- 172.21.15.26:0/2149274613 learned_addr learned my addr 172.21.15.26:0/2149274613 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc912ca6700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 0x7fc90c14fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc9124a5700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 0x7fc90c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 -- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 msgr2=0x7fc90c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 0x7fc90c145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 -- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 msgr2=0x7fc90c14fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 0x7fc90c14fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 -- 172.21.15.26:0/2149274613 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904009530 con 0x7fc90c0b27a0 2022-07-20T02:33:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc912ca6700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 0x7fc90c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc9124a5700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 0x7fc90c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc911ca4700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fc908006940 tx=0x7fc908008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc908011780 con 0x7fc90c0b27a0 2022-07-20T02:33:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9080118e0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c0ac1e0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc908011c40 con 0x7fc90c0b27a0 2022-07-20T02:33:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.268+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c0ad330 con 0x7fc90c0b27a0 2022-07-20T02:33:34.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc90801c460 con 0x7fc90c0b27a0 2022-07-20T02:33:34.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c0574e0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc8fb7fe700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 0x7fc8ec055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc90806c360 con 0x7fc90c0b27a0 2022-07-20T02:33:34.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc9124a5700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 0x7fc8ec055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.272+0000 7fc9124a5700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 0x7fc8ec055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fc8fc00b000 tx=0x7fc8fc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.276+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc9080430c0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc90c0b31e0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc8fb7fe700 1 -- 172.21.15.26:0/2149274613 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v109) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc9080466e0 con 0x7fc90c0b27a0 2022-07-20T02:33:34.416 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 msgr2=0x7fc8ec055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 0x7fc8ec055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fc8fc00b000 tx=0x7fc8fc009040).stop 2022-07-20T02:33:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 msgr2=0x7fc90c14f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fc908006940 tx=0x7fc908008040).stop 2022-07-20T02:33:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 shutdown_connections 2022-07-20T02:33:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc90c0b27a0 0x7fc90c14f930 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc8ec053b00 0x7fc8ec055fc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc90c1432e0 0x7fc90c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 --2- 172.21.15.26:0/2149274613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc90c064990 0x7fc90c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 >> 172.21.15.26:0/2149274613 conn(0x7fc90c05c580 msgr2=0x7fc90c142960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 shutdown_connections 2022-07-20T02:33:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.408+0000 7fc914709700 1 -- 172.21.15.26:0/2149274613 wait complete. 2022-07-20T02:33:34.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.532+0000 7f051d4e4700 1 Processor -- start 2022-07-20T02:33:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.532+0000 7f051d4e4700 1 -- start start 2022-07-20T02:33:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0518147000 0x7f0518143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05181474e0 con 0x7f051805cfd0 2022-07-20T02:33:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180b4e40 con 0x7f0518060f00 2022-07-20T02:33:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051805d3b0 con 0x7f0518147000 2022-07-20T02:33:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f05177fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0518147000 0x7f0518143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0517fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f051805cfd0 0x7f05180b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40894/0 (socket says 172.21.15.26:40894) 2022-07-20T02:33:34.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0517fff700 1 -- 172.21.15.26:0/1592741413 learned_addr learned my addr 172.21.15.26:0/1592741413 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:34.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909224357 0 0) 0x7f05181474e0 con 0x7f051805cfd0 2022-07-20T02:33:34.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504003260 con 0x7f051805cfd0 2022-07-20T02:33:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944110700 0 0) 0x7f05180b4e40 con 0x7f0518060f00 2022-07-20T02:33:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f05177fe700 1 -- 172.21.15.26:0/1592741413 >> v1:172.21.15.120:6789/0 conn(0x7f0518060f00 legacy=0x7f051813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f05177fe700 1 -- 172.21.15.26:0/1592741413 >> v1:172.21.15.26:6789/0 conn(0x7f051805cfd0 legacy=0x7f05180b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f05177fe700 1 -- 172.21.15.26:0/1592741413 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05181431e0 con 0x7f0518147000 2022-07-20T02:33:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f05177fe700 1 --2- 172.21.15.26:0/1592741413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0518147000 0x7f0518143bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f05080032f0 tx=0x7f05080091d0).ready entity=mon.0 client_cookie=22461174c3304fa1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f050800b6c0 con 0x7f0518147000 2022-07-20T02:33:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050800b820 con 0x7f0518147000 2022-07-20T02:33:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f0516ffd700 1 -- 172.21.15.26:0/1592741413 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f050800b990 con 0x7f0518147000 2022-07-20T02:33:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- 172.21.15.26:0/1592741413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0518147000 msgr2=0x7f0518143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 --2- 172.21.15.26:0/1592741413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0518147000 0x7f0518143bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f05080032f0 tx=0x7f05080091d0).stop 2022-07-20T02:33:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- 172.21.15.26:0/1592741413 shutdown_connections 2022-07-20T02:33:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 --2- 172.21.15.26:0/1592741413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0518147000 0x7f0518143bc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.536+0000 7f051d4e4700 1 -- 172.21.15.26:0/1592741413 >> 172.21.15.26:0/1592741413 conn(0x7f05180620c0 msgr2=0x7f051805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- 172.21.15.26:0/1592741413 shutdown_connections 2022-07-20T02:33:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- 172.21.15.26:0/1592741413 wait complete. 2022-07-20T02:33:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 Processor -- start 2022-07-20T02:33:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- start start 2022-07-20T02:33:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 0x7f05181429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0518147000 0x7f05181380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 0x7f05181385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518152220 con 0x7f05180b91f0 2022-07-20T02:33:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518155ba0 con 0x7f0518060f00 2022-07-20T02:33:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051d4e4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05180b04e0 con 0x7f0518147000 2022-07-20T02:33:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051cce3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 0x7f05181385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 0x7f05181429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051cce3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 0x7f05181385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41622/0 (socket says 172.21.15.26:41622) 2022-07-20T02:33:34.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f051cce3700 1 -- 172.21.15.26:0/3961846877 learned_addr learned my addr 172.21.15.26:0/3961846877 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:34.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f05177fe700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0518147000 0x7f05181380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 -- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0518147000 msgr2=0x7f05181380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0518147000 0x7f05181380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 -- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 msgr2=0x7f05181385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 0x7f05181385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.540+0000 7f0517fff700 1 -- 172.21.15.26:0/3961846877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0508009470 con 0x7f0518060f00 2022-07-20T02:33:34.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f0517fff700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 0x7f05181429c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f0518155820 tx=0x7f050c0079e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f050c00ccf0 con 0x7f0518060f00 2022-07-20T02:33:34.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05180b0240 con 0x7f0518060f00 2022-07-20T02:33:34.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c00ce50 con 0x7f0518060f00 2022-07-20T02:33:34.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518138a90 con 0x7f0518060f00 2022-07-20T02:33:34.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f050c0072c0 con 0x7f0518060f00 2022-07-20T02:33:34.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518152370 con 0x7f0518060f00 2022-07-20T02:33:34.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f050c010070 con 0x7f0518060f00 2022-07-20T02:33:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 0x7f04f8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.544+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f050c06c0e0 con 0x7f0518060f00 2022-07-20T02:33:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.548+0000 7f05177fe700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 0x7f04f8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.548+0000 7f05177fe700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 0x7f04f8055f60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f050801b000 tx=0x7f0508016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.552+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f050c042e40 con 0x7f0518060f00 2022-07-20T02:33:34.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.676+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0518143480 con 0x7f0518060f00 2022-07-20T02:33:34.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f05157fa700 1 -- 172.21.15.26:0/3961846877 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+10 (secure 0 0 0) 0x7f050c046460 con 0x7f0518060f00 2022-07-20T02:33:34.684 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:34.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 msgr2=0x7f04f8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 0x7f04f8055f60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f050801b000 tx=0x7f0508016000).stop 2022-07-20T02:33:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 msgr2=0x7f05181429c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 0x7f05181429c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f0518155820 tx=0x7f050c0079e0).stop 2022-07-20T02:33:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 shutdown_connections 2022-07-20T02:33:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05180b91f0 0x7f05181385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f04f8053aa0 0x7f04f8055f60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0518147000 0x7f05181380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 --2- 172.21.15.26:0/3961846877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0518060f00 0x7f05181429c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 >> 172.21.15.26:0/3961846877 conn(0x7f051805cfd0 msgr2=0x7f051813bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 shutdown_connections 2022-07-20T02:33:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.680+0000 7f051d4e4700 1 -- 172.21.15.26:0/3961846877 wait complete. 2022-07-20T02:33:34.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 Processor -- start 2022-07-20T02:33:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 -- start start 2022-07-20T02:33:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fdc143f80 0x7f6fdc147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc061c80 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0b2810 con 0x7f6fdc0618a0 2022-07-20T02:33:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe1fd5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fdc0b2ab0 con 0x7f6fdc143f80 2022-07-20T02:33:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe17d4700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6fdc0b22d0 0x7f6fdc0b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40910/0 (socket says 172.21.15.26:40910) 2022-07-20T02:33:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fe17d4700 1 -- 172.21.15.26:0/3874829740 learned_addr learned my addr 172.21.15.26:0/3874829740 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.872+0000 7f6fdbfff700 1 --2- 172.21.15.26:0/3874829740 >> v2:172.21.15.120:3300/0 conn(0x7f6fdc143f80 0x7f6fdc147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872723941 0 0) 0x7f6fdc061c80 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc003260 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdbfff700 1 -- 172.21.15.26:0/3874829740 >> v1:172.21.15.120:6789/0 conn(0x7f6fdc0618a0 legacy=0x7f6fdc143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdbfff700 1 -- 172.21.15.26:0/3874829740 >> v1:172.21.15.26:6789/0 conn(0x7f6fdc0b22d0 legacy=0x7f6fdc0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720485075 0 0) 0x7f6fbc003260 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdbfff700 1 -- 172.21.15.26:0/3874829740 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc0b2d50 con 0x7f6fdc143f80 2022-07-20T02:33:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdbfff700 1 --2- 172.21.15.26:0/3874829740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc143f80 0x7f6fdc147250 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f6fd4001ba0 tx=0x7f6fcc006df0).ready entity=mon.1 client_cookie=7e423ea06536963 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fd4001a00 con 0x7f6fdc143f80 2022-07-20T02:33:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc000e90 con 0x7f6fdc143f80 2022-07-20T02:33:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fdb7fe700 1 -- 172.21.15.26:0/3874829740 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc00d4f0 con 0x7f6fdc143f80 2022-07-20T02:33:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/3874829740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc143f80 msgr2=0x7f6fdc147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/3874829740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc143f80 0x7f6fdc147250 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f6fd4001ba0 tx=0x7f6fcc006df0).stop 2022-07-20T02:33:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/3874829740 shutdown_connections 2022-07-20T02:33:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/3874829740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc143f80 0x7f6fdc147250 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/3874829740 >> 172.21.15.26:0/3874829740 conn(0x7f6fdc066f70 msgr2=0x7f6fdc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:34.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/3874829740 shutdown_connections 2022-07-20T02:33:34.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/3874829740 wait complete. 2022-07-20T02:33:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 Processor -- start 2022-07-20T02:33:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.876+0000 7f6fe1fd5700 1 -- start start 2022-07-20T02:33:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc0618a0 0x7f6fdc146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fdc143f80 0x7f6fdc0adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc13f5a0 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1529e0 con 0x7f6fdc0618a0 2022-07-20T02:33:34.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc152c80 con 0x7f6fdc143f80 2022-07-20T02:33:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41636/0 (socket says 172.21.15.26:41636) 2022-07-20T02:33:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 -- 172.21.15.26:0/2311407438 learned_addr learned my addr 172.21.15.26:0/2311407438 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe0fd3700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc0618a0 0x7f6fdc146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe17d4700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fdc143f80 0x7f6fdc0adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 -- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fdc143f80 msgr2=0x7f6fdc0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fdc143f80 0x7f6fdc0adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 -- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc0618a0 msgr2=0x7f6fdc146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc0618a0 0x7f6fdc146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 -- 172.21.15.26:0/2311407438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc00b180 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fdbfff700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f6fd4001ba0 tx=0x7f6fcc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc016070 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc002f50 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc152f20 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc00d4f0 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc147190 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.880+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc0574e0 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.884+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6fcc00f030 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.884+0000 7f6fd9ffb700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 0x7f6fc0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.884+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6fcc06cc80 con 0x7f6fdc0b22d0 2022-07-20T02:33:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.884+0000 7f6fe0fd3700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 0x7f6fc0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.884+0000 7f6fe0fd3700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 0x7f6fc0055f30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6fc400d000 tx=0x7f6fc4007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:34.892+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6fcc043960 con 0x7f6fdc0b22d0 2022-07-20T02:33:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fdc1533c0 con 0x7f6fdc0b22d0 2022-07-20T02:33:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fd9ffb700 1 -- 172.21.15.26:0/2311407438 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6fcc0230a0 con 0x7f6fdc0b22d0 2022-07-20T02:33:35.023 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:35.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 msgr2=0x7f6fc0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 0x7f6fc0055f30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6fc400d000 tx=0x7f6fc4007b90).stop 2022-07-20T02:33:35.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 msgr2=0x7f6fdc0ad760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f6fd4001ba0 tx=0x7f6fcc014000).stop 2022-07-20T02:33:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 shutdown_connections 2022-07-20T02:33:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6fdc0b22d0 0x7f6fdc0ad760 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6fc0053a70 0x7f6fc0055f30 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fdc143f80 0x7f6fdc0adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 --2- 172.21.15.26:0/2311407438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fdc0618a0 0x7f6fdc146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.020+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 >> 172.21.15.26:0/2311407438 conn(0x7f6fdc066f70 msgr2=0x7f6fdc141dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:35.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.024+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 shutdown_connections 2022-07-20T02:33:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.024+0000 7f6fe1fd5700 1 -- 172.21.15.26:0/2311407438 wait complete. 2022-07-20T02:33:35.043 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:35.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.164+0000 7fa7b7f7c700 1 Processor -- start 2022-07-20T02:33:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.164+0000 7fa7b7f7c700 1 -- start start 2022-07-20T02:33:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa7b00b2920 0x7fa7b0147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b0067190 0x7fa7b0148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b00b1bc0 con 0x7fa7b0061050 2022-07-20T02:33:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b014b040 con 0x7fa7b00b2920 2022-07-20T02:33:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00a8860 con 0x7fa7b0067190 2022-07-20T02:33:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5517700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa7b0061050 0x7fa7b0066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40922/0 (socket says 172.21.15.26:40922) 2022-07-20T02:33:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5517700 1 -- 172.21.15.26:0/3269452053 learned_addr learned my addr 172.21.15.26:0/3269452053 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:35.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3269452053 >> v2:172.21.15.26:3300/0 conn(0x7fa7b00b2920 0x7fa7b0147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b6519700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b0067190 0x7fa7b0148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023629180 0 0) 0x7fa7b00b1bc0 con 0x7fa7b0061050 2022-07-20T02:33:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798003260 con 0x7fa7b0061050 2022-07-20T02:33:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986240411 0 0) 0x7fa798003260 con 0x7fa7b0061050 2022-07-20T02:33:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3269452053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b0067190 msgr2=0x7fa7b0148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b0067190 0x7fa7b0148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3269452053 >> v1:172.21.15.26:6789/0 conn(0x7fa7b0061050 legacy=0x7fa7b0066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3269452053 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0147700 con 0x7fa7b00b2920 2022-07-20T02:33:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b00b2920 0x7fa7b0147d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fa7ac001820 tx=0x7fa7a0006df0).ready entity=mon.0 client_cookie=2dd1b929906074cb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:35.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7ac001680 con 0x7fa7b00b2920 2022-07-20T02:33:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0000e90 con 0x7fa7b00b2920 2022-07-20T02:33:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b4d16700 1 -- 172.21.15.26:0/3269452053 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7a0012460 con 0x7fa7b00b2920 2022-07-20T02:33:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3269452053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b00b2920 msgr2=0x7fa7b0147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b00b2920 0x7fa7b0147d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fa7ac001820 tx=0x7fa7a0006df0).stop 2022-07-20T02:33:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3269452053 shutdown_connections 2022-07-20T02:33:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b0067190 0x7fa7b0148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3269452053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b00b2920 0x7fa7b0147d90 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3269452053 >> 172.21.15.26:0/3269452053 conn(0x7fa7b00b60a0 msgr2=0x7fa7b00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:35.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3269452053 shutdown_connections 2022-07-20T02:33:35.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.168+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3269452053 wait complete. 2022-07-20T02:33:35.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 Processor -- start 2022-07-20T02:33:35.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- start start 2022-07-20T02:33:35.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7b0067190 0x7fa7b0062810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b00b2920 0x7fa7b0062d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0064fb0 con 0x7fa7b0061050 2022-07-20T02:33:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0065250 con 0x7fa7b0067190 2022-07-20T02:33:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b015a950 con 0x7fa7b00b2920 2022-07-20T02:33:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41648/0 (socket says 172.21.15.26:41648) 2022-07-20T02:33:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3953547433 learned_addr learned my addr 172.21.15.26:0/3953547433 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b6519700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b00b2920 0x7fa7b0062d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5517700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7b0067190 0x7fa7b0062810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b00b2920 msgr2=0x7fa7b0062d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b00b2920 0x7fa7b0062d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7b0067190 msgr2=0x7fa7b0062810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7b0067190 0x7fa7b0062810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 -- 172.21.15.26:0/3953547433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a000b180 con 0x7fa7b0061050 2022-07-20T02:33:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b5d18700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fa7ac001820 tx=0x7fa7a0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7a000da20 con 0x7fa7b0061050 2022-07-20T02:33:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a000db80 con 0x7fa7b0061050 2022-07-20T02:33:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b015abf0 con 0x7fa7b0061050 2022-07-20T02:33:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7a001e400 con 0x7fa7b0061050 2022-07-20T02:33:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b0065c20 con 0x7fa7b0061050 2022-07-20T02:33:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.172+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b0066fb0 con 0x7fa7b0061050 2022-07-20T02:33:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.176+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa7a0025030 con 0x7fa7b0061050 2022-07-20T02:33:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.176+0000 7fa79effd700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 0x7fa790055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.176+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa7a006cca0 con 0x7fa7b0061050 2022-07-20T02:33:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.176+0000 7fa7b5517700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 0x7fa790055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.176+0000 7fa7b5517700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 0x7fa790055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fa7ac00b000 tx=0x7fa7ac001ca0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.184+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa7b0066fb0 con 0x7fa7b0061050 2022-07-20T02:33:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.344+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7b0066fb0 con 0x7fa7b0061050 2022-07-20T02:33:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa79effd700 1 -- 172.21.15.26:0/3953547433 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2493 (secure 0 0 0) 0x7fa7b0066fb0 con 0x7fa7b0061050 2022-07-20T02:33:35.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:35.360 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:28.135776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7398":{"gid":7398,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3259008170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3259008170},{"type":"v1","addr":"172.21.15.26:6837","nonce":3259008170}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:35.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 msgr2=0x7fa790055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 0x7fa790055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fa7ac00b000 tx=0x7fa7ac001ca0).stop 2022-07-20T02:33:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 msgr2=0x7fa7b0066780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fa7ac001820 tx=0x7fa7a0014000).stop 2022-07-20T02:33:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 shutdown_connections 2022-07-20T02:33:35.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa7900539e0 0x7fa790055ea0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7b00b2920 0x7fa7b0062d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7b0067190 0x7fa7b0062810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 --2- 172.21.15.26:0/3953547433 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7b0061050 0x7fa7b0066780 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 >> 172.21.15.26:0/3953547433 conn(0x7fa7b00b60a0 msgr2=0x7fa7b0146a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 shutdown_connections 2022-07-20T02:33:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:35.348+0000 7fa7b7f7c700 1 -- 172.21.15.26:0/3953547433 wait complete. 2022-07-20T02:33:35.427 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 55 2022-07-20T02:33:35.732 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:26.130844+0000', 'modified': '2022-07-20T02:33:28.135776+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7398': {'gid': 7398, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/3259008170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 3259008170}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 3259008170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:35.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:35.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:35.878 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7398,"want_state":"up:active","state":"up:active","rank_uptime":9.7346403919999993,"mdsmap_epoch":55,"osdmap_epoch":109,"osdmap_epoch_barrier":109,"uptime":15.061388681} 2022-07-20T02:33:35.879 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7398, "mdsmap_epoch": 55, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "rank_uptime": 9.734640392, "state": "up:active", "uptime": 15.061388681, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:35.881 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:33:35.882 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:33:35.883 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:33:35.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:35.884 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:35.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:33:35.969 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:35.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:36.039 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:36.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.040 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:33:36.059 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:33:36.060 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:36.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:33:36.154 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:36.166 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:36.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:33:36.175+0000 7f4667bc3080 -1 init, newargv = 0x55726f6d2c50 newargc=9 2022-07-20T02:33:36.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22778]: starting ceph client 2022-07-20T02:33:36.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22778]: starting fuse 2022-07-20T02:33:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:36.221 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:36.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.234 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:33:36.288 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:33:36.289 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:33:36.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.311 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:33:36.319 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:33:36.322 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:33:36.322 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:33:36.323 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:33:36.330 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:36.334 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:33:36.334 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:33:36.335 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:33:36.335 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:36.336 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:33:36.336 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:33:36.337 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:33:36.337 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:33:36.346 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:36.347 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:33:36.347 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:33:36.348 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:33:36.369 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:33:36.370 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:33:36.371 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:33:36.371 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:33:36.372 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:33:36.462 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22778.asok 2022-07-20T02:33:36.475 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22778.asok 2022-07-20T02:33:36.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22778.asok status 2022-07-20T02:33:36.599 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:33:36.599 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:33:36.600 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:33:36.600 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:33:36.601 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:33:36.601 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:33:36.602 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:33:36.602 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22778", 2022-07-20T02:33:36.602 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:33:36.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:36.603 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:33:36.604 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:33:36.604 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 7369, 2022-07-20T02:33:36.605 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:33:36.605 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:33:36.622 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:33:36.623 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 7369 2022-07-20T02:33:36.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:36.625 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:33:36.625 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:33:36.626 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:33:36.626 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3400844641 2022-07-20T02:33:36.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:33:36.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:36.628 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:33:36.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:33:36.646 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:33:36.647 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3400844641 2022-07-20T02:33:36.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:33:36.648 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.7369 172.21.15.176:0/3400844641", 2022-07-20T02:33:36.648 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/3400844641", 2022-07-20T02:33:36.648 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:33:36.648 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 55, 2022-07-20T02:33:36.649 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 109, 2022-07-20T02:33:36.649 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:33:36.655 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:33:36.655 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:33:36.655 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:33:36.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.658 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:36.658 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:36.669 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:33:36.671 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:36.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:36.672 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:36.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T02:33:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 Processor -- start 2022-07-20T02:33:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 -- start start 2022-07-20T02:33:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fefe80b3260 0x7fefe8067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe805ff70 con 0x7fefe80980d0 2022-07-20T02:33:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80606e0 con 0x7fefe8061860 2022-07-20T02:33:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.856+0000 7fefece6d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe80b3640 con 0x7fefe80b3260 2022-07-20T02:33:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe7fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fefe80980d0 0x7fefe8142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40940/0 (socket says 172.21.15.26:40940) 2022-07-20T02:33:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe7fff700 1 -- 172.21.15.26:0/3123840259 learned_addr learned my addr 172.21.15.26:0/3123840259 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 --2- 172.21.15.26:0/3123840259 >> v2:172.21.15.120:3300/0 conn(0x7fefe80b3260 0x7fefe8067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123702273 0 0) 0x7fefe805ff70 con 0x7fefe80980d0 2022-07-20T02:33:36.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8004370 con 0x7fefe80980d0 2022-07-20T02:33:36.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655891655 0 0) 0x7fefe80606e0 con 0x7fefe8061860 2022-07-20T02:33:36.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe805ff70 con 0x7fefe8061860 2022-07-20T02:33:36.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489949002 0 0) 0x7fefc8004370 con 0x7fefe80980d0 2022-07-20T02:33:36.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 >> v2:172.21.15.120:3300/0 conn(0x7fefe80b3260 msgr2=0x7fefe8067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:36.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 --2- 172.21.15.26:0/3123840259 >> v2:172.21.15.120:3300/0 conn(0x7fefe80b3260 0x7fefe8067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 >> v1:172.21.15.120:6789/0 conn(0x7fefe8061860 legacy=0x7fefe80979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:36.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80601a0 con 0x7fefe80980d0 2022-07-20T02:33:36.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fefe00019e0 con 0x7fefe80980d0 2022-07-20T02:33:36.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 >> v1:172.21.15.26:6789/0 conn(0x7fefe80980d0 legacy=0x7fefe8142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefc8004df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefc8005610 0x7fefc8005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc8005f80 0x7fefc800c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe80b3640 con 0x7fefc80049f0 2022-07-20T02:33:36.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc800ce00 con 0x7fefc8005f80 2022-07-20T02:33:36.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc800cfc0 con 0x7fefc8005610 2022-07-20T02:33:36.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefc8004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe7fff700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc8005f80 0x7fefc800c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefdeffd700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefc8005610 0x7fefc8005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 -- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefc8005610 msgr2=0x7fefc8005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:36.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefc8005610 0x7fefc8005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 -- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc8005f80 msgr2=0x7fefc800c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:36.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc8005f80 0x7fefc800c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 -- 172.21.15.26:0/3123840259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc8004370 con 0x7fefc80049f0 2022-07-20T02:33:36.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe7fff700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc8005f80 0x7fefc800c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:36.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe7fff700 1 -- 172.21.15.26:0/3123840259 reap_dead start 2022-07-20T02:33:36.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe77fe700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefc8004df0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fefd800c000 tx=0x7fefd8006df0).ready entity=mon.0 client_cookie=972c18354324c693 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:36.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd800e800 con 0x7fefc80049f0 2022-07-20T02:33:36.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd800e960 con 0x7fefc80049f0 2022-07-20T02:33:36.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefe6ffd700 1 -- 172.21.15.26:0/3123840259 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd800ecc0 con 0x7fefc80049f0 2022-07-20T02:33:36.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 -- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 msgr2=0x7fefc8004df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:36.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefc8004df0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fefd800c000 tx=0x7fefd8006df0).stop 2022-07-20T02:33:36.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 -- 172.21.15.26:0/3123840259 shutdown_connections 2022-07-20T02:33:36.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 --2- 172.21.15.26:0/3123840259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefc8004df0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 -- 172.21.15.26:0/3123840259 >> 172.21.15.26:0/3123840259 conn(0x7fefe80b67f0 msgr2=0x7fefe80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:36.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.860+0000 7fefece6d700 1 -- 172.21.15.26:0/3123840259 shutdown_connections 2022-07-20T02:33:36.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 -- 172.21.15.26:0/3123840259 wait complete. 2022-07-20T02:33:36.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 Processor -- start 2022-07-20T02:33:36.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 -- start start 2022-07-20T02:33:36.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe80b5020 0x7fefe80b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8152ad0 0x7fefe8152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe80b0ec0 con 0x7fefc80049f0 2022-07-20T02:33:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe81526a0 con 0x7fefe80b5020 2022-07-20T02:33:36.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefece6d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8098f60 con 0x7fefe8152ad0 2022-07-20T02:33:36.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41668/0 (socket says 172.21.15.26:41668) 2022-07-20T02:33:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 -- 172.21.15.26:0/2523238583 learned_addr learned my addr 172.21.15.26:0/2523238583 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe7fff700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8152ad0 0x7fefe8152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefdeffd700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe80b5020 0x7fefe80b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 -- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8152ad0 msgr2=0x7fefe8152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8152ad0 0x7fefe8152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 -- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe80b5020 msgr2=0x7fefe80b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:36.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe80b5020 0x7fefe80b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:36.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 -- 172.21.15.26:0/2523238583 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8099200 con 0x7fefc80049f0 2022-07-20T02:33:36.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.864+0000 7fefe77fe700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7fefd800c000 tx=0x7fefd8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:36.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd801cdc0 con 0x7fefc80049f0 2022-07-20T02:33:36.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd8000e90 con 0x7fefc80049f0 2022-07-20T02:33:36.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe80994a0 con 0x7fefc80049f0 2022-07-20T02:33:36.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd80245e0 con 0x7fefc80049f0 2022-07-20T02:33:36.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe8095f20 con 0x7fefc80049f0 2022-07-20T02:33:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fefd8024740 con 0x7fefc80049f0 2022-07-20T02:33:36.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe80574e0 con 0x7fefc80049f0 2022-07-20T02:33:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 0x7fefcc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:36.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fefd806d7b0 con 0x7fefc80049f0 2022-07-20T02:33:36.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefdeffd700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 0x7fefcc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:36.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.868+0000 7fefdeffd700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 0x7fefcc055fc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fefd0003000 tx=0x7fefd0007a00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:36.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:36.876+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fefd8047f40 con 0x7fefc80049f0 2022-07-20T02:33:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.040+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fefe806afc0 con 0x7fefc80049f0 2022-07-20T02:33:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.040+0000 7fefe57fa700 1 -- 172.21.15.26:0/2523238583 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1936 (secure 0 0 0) 0x7fefd800be60 con 0x7fefc80049f0 2022-07-20T02:33:37.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:37.055 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:37.055 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:33:37.056 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:33:37.056 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764493135872, 2022-07-20T02:33:37.056 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 11042816, 2022-07-20T02:33:37.057 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8600977408, 2022-07-20T02:33:37.057 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011125395074486732, 2022-07-20T02:33:37.057 INFO:teuthology.orchestra.run.smithi026.stdout: "num_osds": 8, 2022-07-20T02:33:37.058 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_osds": 8, 2022-07-20T02:33:37.058 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_omap_osds": 8 2022-07-20T02:33:37.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:37.059 INFO:teuthology.orchestra.run.smithi026.stdout: "stats_by_class": { 2022-07-20T02:33:37.060 INFO:teuthology.orchestra.run.smithi026.stdout: "ssd": { 2022-07-20T02:33:37.060 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:33:37.060 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764493135872, 2022-07-20T02:33:37.061 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 11042816, 2022-07-20T02:33:37.061 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8600977408, 2022-07-20T02:33:37.061 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011125395074486732 2022-07-20T02:33:37.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:37.062 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:33:37.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:37.064 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_health_metrics", 2022-07-20T02:33:37.064 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 1, 2022-07-20T02:33:37.065 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:33:37.065 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:33:37.065 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 2, 2022-07-20T02:33:37.066 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:33:37.066 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:33:37.067 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:33:37.067 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362919165952 2022-07-20T02:33:37.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:37.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:37.069 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd", 2022-07-20T02:33:37.069 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 2, 2022-07-20T02:33:37.070 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:33:37.070 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:33:37.070 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:33:37.071 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:33:37.071 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:33:37.072 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:33:37.072 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362919165952 2022-07-20T02:33:37.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:37.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:37.077 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephfs_metadata", 2022-07-20T02:33:37.079 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 19, 2022-07-20T02:33:37.085 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:33:37.086 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 2742, 2022-07-20T02:33:37.086 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 22, 2022-07-20T02:33:37.087 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 64, 2022-07-20T02:33:37.087 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 65536, 2022-07-20T02:33:37.087 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 9.0290072307652736e-08, 2022-07-20T02:33:37.088 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362919165952 2022-07-20T02:33:37.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:37.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:37.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephfs_data", 2022-07-20T02:33:37.091 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 20, 2022-07-20T02:33:37.091 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:33:37.092 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:33:37.092 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:33:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:33:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:33:37.094 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:33:37.095 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362919165952 2022-07-20T02:33:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:37.097 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:37.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 msgr2=0x7fefcc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 0x7fefcc055fc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fefd0003000 tx=0x7fefd0007a00).stop 2022-07-20T02:33:37.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 msgr2=0x7fefe8095770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7fefd800c000 tx=0x7fefd8019000).stop 2022-07-20T02:33:37.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 shutdown_connections 2022-07-20T02:33:37.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fefcc053b00 0x7fefcc055fc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fefe8152ad0 0x7fefe8152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefe80b5020 0x7fefe80b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 --2- 172.21.15.26:0/2523238583 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fefc80049f0 0x7fefe8095770 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 >> 172.21.15.26:0/2523238583 conn(0x7fefe80b67f0 msgr2=0x7fefe805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:37.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 shutdown_connections 2022-07-20T02:33:37.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.044+0000 7fefece6d700 1 -- 172.21.15.26:0/2523238583 wait complete. 2022-07-20T02:33:37.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data size -f json-pretty 2022-07-20T02:33:37.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.220+0000 7f788959d700 1 Processor -- start 2022-07-20T02:33:37.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.220+0000 7f788959d700 1 -- start start 2022-07-20T02:33:37.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f78840b9c80 0x7f78841431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:37.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78840ba060 con 0x7f7884062710 2022-07-20T02:33:37.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7884061f80 con 0x7f78840b5b70 2022-07-20T02:33:37.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788412bee0 con 0x7f78840b9c80 2022-07-20T02:33:37.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f78840b9c80 0x7f78841431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:37.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f78827fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7884062710 0x7f78840b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40958/0 (socket says 172.21.15.26:40958) 2022-07-20T02:33:37.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f78840b9c80 0x7f78841431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41676/0 (socket says 172.21.15.26:41676) 2022-07-20T02:33:37.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f78827fc700 1 -- 172.21.15.26:0/2624031601 learned_addr learned my addr 172.21.15.26:0/2624031601 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:37.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 -- 172.21.15.26:0/2624031601 >> v1:172.21.15.120:6789/0 conn(0x7f78840b5b70 legacy=0x7f78841403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 -- 172.21.15.26:0/2624031601 >> v1:172.21.15.26:6789/0 conn(0x7f7884062710 legacy=0x7f78840b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 -- 172.21.15.26:0/2624031601 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884062220 con 0x7f78840b9c80 2022-07-20T02:33:37.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7882ffd700 1 --2- 172.21.15.26:0/2624031601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78840b9c80 0x7f78841431a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f786c002f70 tx=0x7f786c006df0).ready entity=mon.0 client_cookie=e48b9b5d00df3c73 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:37.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7881ffb700 1 -- 172.21.15.26:0/2624031601 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c00d610 con 0x7f78840b9c80 2022-07-20T02:33:37.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7881ffb700 1 -- 172.21.15.26:0/2624031601 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c000e90 con 0x7f78840b9c80 2022-07-20T02:33:37.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f7881ffb700 1 -- 172.21.15.26:0/2624031601 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c00d960 con 0x7f78840b9c80 2022-07-20T02:33:37.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- 172.21.15.26:0/2624031601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78840b9c80 msgr2=0x7f78841431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 --2- 172.21.15.26:0/2624031601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78840b9c80 0x7f78841431a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f786c002f70 tx=0x7f786c006df0).stop 2022-07-20T02:33:37.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- 172.21.15.26:0/2624031601 shutdown_connections 2022-07-20T02:33:37.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 --2- 172.21.15.26:0/2624031601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78840b9c80 0x7f78841431a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- 172.21.15.26:0/2624031601 >> 172.21.15.26:0/2624031601 conn(0x7f7884066f70 msgr2=0x7f788405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:37.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- 172.21.15.26:0/2624031601 shutdown_connections 2022-07-20T02:33:37.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.224+0000 7f788959d700 1 -- 172.21.15.26:0/2624031601 wait complete. 2022-07-20T02:33:37.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 Processor -- start 2022-07-20T02:33:37.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- start start 2022-07-20T02:33:37.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:37.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 0x7f78841564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:37.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 0x7f78840b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:37.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7884155b70 con 0x7f7884062710 2022-07-20T02:33:37.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788412c1a0 con 0x7f78840b9c80 2022-07-20T02:33:37.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78840b1ca0 con 0x7f78840b5b70 2022-07-20T02:33:37.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:37.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41682/0 (socket says 172.21.15.26:41682) 2022-07-20T02:33:37.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 -- 172.21.15.26:0/4272362972 learned_addr learned my addr 172.21.15.26:0/4272362972 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f78827fc700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 0x7f78841564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f78837fe700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 0x7f78840b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 -- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 msgr2=0x7f78841564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 0x7f78841564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 -- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 msgr2=0x7f78840b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 0x7f78840b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 -- 172.21.15.26:0/4272362972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c00b180 con 0x7f7884062710 2022-07-20T02:33:37.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f78837fe700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 0x7f78840b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:37.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f78827fc700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 0x7f78841564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:37.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f7882ffd700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f786c002f70 tx=0x7f786c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:37.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c00d990 con 0x7f7884062710 2022-07-20T02:33:37.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c011ce0 con 0x7f7884062710 2022-07-20T02:33:37.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78840b30a0 con 0x7f7884062710 2022-07-20T02:33:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c01e650 con 0x7f7884062710 2022-07-20T02:33:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.228+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7884155d60 con 0x7f7884062710 2022-07-20T02:33:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f786c025030 con 0x7f7884062710 2022-07-20T02:33:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78840b6650 con 0x7f7884062710 2022-07-20T02:33:37.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f787bfff700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 0x7f7868055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:37.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f786c06d4c0 con 0x7f7884062710 2022-07-20T02:33:37.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f78827fc700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 0x7f7868055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:37.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.232+0000 7f78827fc700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 0x7f7868055f30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f787400b000 tx=0x7f7874009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:37.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.240+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f786c06ae90 con 0x7f7884062710 2022-07-20T02:33:37.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.364+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"} v 0) v1 -- 0x7f78840574e0 con 0x7f7884062710 2022-07-20T02:33:37.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.364+0000 7f787bfff700 1 -- 172.21.15.26:0/4272362972 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"}]=0 v109) v1 ==== 123+0+64 (secure 0 0 0) 0x7f786c0230c0 con 0x7f7884062710 2022-07-20T02:33:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:37.368 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": "cephfs_data", 2022-07-20T02:33:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_id": 20, 2022-07-20T02:33:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2 2022-07-20T02:33:37.368 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:37.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 msgr2=0x7f7868055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 0x7f7868055f30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f787400b000 tx=0x7f7874009040).stop 2022-07-20T02:33:37.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 msgr2=0x7f78841415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:37.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f786c002f70 tx=0x7f786c015000).stop 2022-07-20T02:33:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 shutdown_connections 2022-07-20T02:33:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7868053a70 0x7f7868055f30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78840b9c80 0x7f78840b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78840b5b70 0x7f78841564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 --2- 172.21.15.26:0/4272362972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7884062710 0x7f78841415c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 >> 172.21.15.26:0/4272362972 conn(0x7f7884066f70 msgr2=0x7f78840b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 shutdown_connections 2022-07-20T02:33:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:37.368+0000 7f788959d700 1 -- 172.21.15.26:0/4272362972 wait complete. 2022-07-20T02:33:37.395 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T02:33:37.396 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest/mnt.0 && sudo df . 2022-07-20T02:33:37.426 INFO:teuthology.orchestra.run.smithi159.stdout:Filesystem 1K-blocks Used Available Use% Mounted on 2022-07-20T02:33:37.427 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-fuse 354410496 0 354410496 0% /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:37.427 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:33:37.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:37.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:37.483 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:37.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:37.563 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:33:37.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:37.589 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:37.589 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:37.632 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:33:37.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:37.654 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:33:37.687 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:33:37.687 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:37.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23156]: fuse finished with error 0 and tester_r 0 2022-07-20T02:33:37.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:37.745 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:37.746 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:37.805 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:33:37.809 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:33:37.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:33:37.817 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:37.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:37.838 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:37.841 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:37.866 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:33:37.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:37.868 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:33:37.869 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:33:37.872 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:37.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22778]: fuse finished with error 0 and tester_r 0 2022-07-20T02:33:37.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:37.933 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:37.933 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:37.956 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:33:37.957 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:33:37.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:33:37.975 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:38.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:33:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 Processor -- start 2022-07-20T02:33:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- start start 2022-07-20T02:33:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f286c0b2920 0x7f286c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 0x7f286c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c0b1bc0 con 0x7f286c066a80 2022-07-20T02:33:38.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c14b040 con 0x7f286c0b2920 2022-07-20T02:33:38.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c0a8860 con 0x7f286c061050 2022-07-20T02:33:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286b59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f286c066a80 0x7f286c066ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40970/0 (socket says 172.21.15.26:40970) 2022-07-20T02:33:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286b59e700 1 -- 172.21.15.26:0/1416560473 learned_addr learned my addr 172.21.15.26:0/1416560473 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 --2- 172.21.15.26:0/1416560473 >> v2:172.21.15.26:3300/0 conn(0x7f286c0b2920 0x7f286c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286a59c700 1 --2- 172.21.15.26:0/1416560473 >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 0x7f286c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226592632 0 0) 0x7f286c0b1bc0 con 0x7f286c066a80 2022-07-20T02:33:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850003260 con 0x7f286c066a80 2022-07-20T02:33:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 -- 172.21.15.26:0/1416560473 >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 msgr2=0x7f286c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 --2- 172.21.15.26:0/1416560473 >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 0x7f286c148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 288059714 0 0) 0x7f2850003260 con 0x7f286c066a80 2022-07-20T02:33:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 -- 172.21.15.26:0/1416560473 >> v1:172.21.15.26:6789/0 conn(0x7f286c066a80 legacy=0x7f286c066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 -- 172.21.15.26:0/1416560473 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c147700 con 0x7f286c0b2920 2022-07-20T02:33:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286a59c700 1 --2- 172.21.15.26:0/1416560473 >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 0x7f286c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f286ad9d700 1 --2- 172.21.15.26:0/1416560473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c0b2920 0x7f286c147d90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f2864001b80 tx=0x7f285c006df0).ready entity=mon.0 client_cookie=289db1686f5e6eab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28640019e0 con 0x7f286c0b2920 2022-07-20T02:33:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c000e90 con 0x7f286c0b2920 2022-07-20T02:33:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f2869d9b700 1 -- 172.21.15.26:0/1416560473 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c012460 con 0x7f286c0b2920 2022-07-20T02:33:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- 172.21.15.26:0/1416560473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c0b2920 msgr2=0x7f286c147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 --2- 172.21.15.26:0/1416560473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c0b2920 0x7f286c147d90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f2864001b80 tx=0x7f285c006df0).stop 2022-07-20T02:33:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- 172.21.15.26:0/1416560473 shutdown_connections 2022-07-20T02:33:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 --2- 172.21.15.26:0/1416560473 >> v2:172.21.15.120:3300/0 conn(0x7f286c061050 0x7f286c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 --2- 172.21.15.26:0/1416560473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c0b2920 0x7f286c147d90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- 172.21.15.26:0/1416560473 >> 172.21.15.26:0/1416560473 conn(0x7f286c0b60a0 msgr2=0x7f286c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- 172.21.15.26:0/1416560473 shutdown_connections 2022-07-20T02:33:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.148+0000 7f28711c3700 1 -- 172.21.15.26:0/1416560473 wait complete. 2022-07-20T02:33:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 Processor -- start 2022-07-20T02:33:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 -- start start 2022-07-20T02:33:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286c066a80 0x7f286c065e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f286c0b2920 0x7f286c066380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0a98a0 con 0x7f286c061050 2022-07-20T02:33:38.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c065a60 con 0x7f286c0b2920 2022-07-20T02:33:38.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f28711c3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c065010 con 0x7f286c066a80 2022-07-20T02:33:38.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41698/0 (socket says 172.21.15.26:41698) 2022-07-20T02:33:38.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 -- 172.21.15.26:0/211963132 learned_addr learned my addr 172.21.15.26:0/211963132 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286a59c700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286c066a80 0x7f286c065e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286b59e700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f286c0b2920 0x7f286c066380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 -- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286c066a80 msgr2=0x7f286c065e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286c066a80 0x7f286c065e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:38.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 -- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f286c0b2920 msgr2=0x7f286c066380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:38.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f286c0b2920 0x7f286c066380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:38.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 -- 172.21.15.26:0/211963132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286000b180 con 0x7f286c061050 2022-07-20T02:33:38.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.152+0000 7f286ad9d700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f2864001b80 tx=0x7f285c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:38.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c016070 con 0x7f286c061050 2022-07-20T02:33:38.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c002f50 con 0x7f286c061050 2022-07-20T02:33:38.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c012460 con 0x7f286c061050 2022-07-20T02:33:38.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c00b180 con 0x7f286c061050 2022-07-20T02:33:38.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c0ab6c0 con 0x7f286c061050 2022-07-20T02:33:38.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c065bd0 con 0x7f286c061050 2022-07-20T02:33:38.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f285c00f030 con 0x7f286c061050 2022-07-20T02:33:38.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 0x7f2848055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:38.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f285c06cb20 con 0x7f286c061050 2022-07-20T02:33:38.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.156+0000 7f286a59c700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 0x7f2848055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:38.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.160+0000 7f286a59c700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 0x7f2848055f30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f286000b000 tx=0x7f2860006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:38.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.164+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f286c065bd0 con 0x7f286c061050 2022-07-20T02:33:38.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.292+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f286c065bd0 con 0x7f286c061050 2022-07-20T02:33:38.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.292+0000 7f285bfff700 1 -- 172.21.15.26:0/211963132 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v109) v1 ==== 81+0+34099 (secure 0 0 0) 0x7f286c065bd0 con 0x7f286c061050 2022-07-20T02:33:38.299 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:38.300 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:38.300 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 109, 2022-07-20T02:33:38.300 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:33:38.301 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:33:38.301 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:33:27.133631+0000", 2022-07-20T02:33:38.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:38.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:38.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:33:38.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:33:38.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:33:38.302 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:33:38.303 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:33:38.303 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:33:38.303 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:33:38.303 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:38.304 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:33:38.304 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:33:38.304 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:33:38.304 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:33:38.305 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:33:38.305 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 20, 2022-07-20T02:33:38.305 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:33:38.305 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:33:38.306 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:33:38.306 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:33:38.307 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:33:38.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.307 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:33:38.307 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:33:38.308 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:33:38.308 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:38.308 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:38.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:38.309 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:33:38.310 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:33:38.311 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:33:38.311 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:33:38.311 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:33:38.311 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:38.312 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:38.312 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:38.312 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:38.312 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:38.312 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:38.313 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:38.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.313 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:33:38.313 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:38.314 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:38.314 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:38.314 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:38.314 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:38.315 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:38.315 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:38.315 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:38.315 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:38.315 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:38.316 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:38.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:38.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:38.316 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:38.317 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:38.317 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:38.318 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:38.318 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:38.318 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:38.318 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:38.319 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:38.319 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:38.319 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:38.319 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:38.320 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:38.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:38.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.320 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:38.320 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:38.321 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:38.321 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:38.321 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:38.321 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:38.322 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:38.322 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:38.322 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:38.322 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:38.323 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:38.323 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:38.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:33:38.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.324 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:38.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:33:38.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.325 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:33:38.325 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:33:38.325 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:33:38.325 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:38.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:38.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:38.326 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:38.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:38.327 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:38.327 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:38.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:38.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:38.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:38.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:38.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:38.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:38.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:38.330 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:38.330 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:38.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:38.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:38.331 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:38.332 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:38.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.332 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:33:38.332 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:38.333 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:38.333 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:38.333 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:38.333 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:38.334 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:38.334 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:38.334 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:38.334 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:38.335 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:38.335 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:38.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:38.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:38.335 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:38.336 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:38.336 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:38.336 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:38.337 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:38.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:38.339 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:38.339 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:38.340 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:38.340 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:38.341 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:38.341 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:38.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:38.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.342 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:38.343 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:38.343 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:38.344 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:38.344 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:38.344 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:38.345 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:38.345 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:38.346 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:38.346 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:38.347 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:38.347 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:38.347 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:38.348 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:33:38.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.350 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 19, 2022-07-20T02:33:38.351 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:33:38.351 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:33:23.691619+0000", 2022-07-20T02:33:38.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:38.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:38.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:38.353 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:38.354 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:38.355 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:38.355 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:38.356 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:38.356 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:38.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:38.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:38.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:38.358 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:38.359 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:38.360 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:38.360 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:38.361 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:38.361 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:38.362 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:38.362 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:38.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.363 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "108", 2022-07-20T02:33:38.364 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:38.364 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:38.365 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:38.366 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:38.390 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:38.390 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:38.391 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:38.391 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:38.391 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:38.401 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:38.402 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:38.403 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:38.404 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.405 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:38.406 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:38.406 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:38.406 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:38.406 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:38.407 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:38.407 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:38.407 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:38.407 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:38.407 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:38.408 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:38.408 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:38.409 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:33:38.410 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:33:38.410 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:33:38.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.411 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:38.411 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:33:38.412 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:33:38.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.413 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 20, 2022-07-20T02:33:38.413 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:33:38.414 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:33:24.900742+0000", 2022-07-20T02:33:38.414 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:38.414 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:38.414 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:38.415 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:38.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:38.415 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:38.415 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:38.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:38.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:38.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:38.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:38.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:38.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:38.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:38.417 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:38.417 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:38.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:38.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:38.418 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:38.418 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:38.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.419 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "109", 2022-07-20T02:33:38.421 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:38.421 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:38.421 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:38.422 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:38.423 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:38.423 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:38.423 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:38.424 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:38.424 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:38.424 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:38.424 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:38.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:38.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:38.425 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:38.426 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:38.426 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:38.426 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:38.426 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:38.427 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:38.427 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:38.427 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:38.427 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:38.428 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:38.428 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:38.428 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:38.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:38.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.429 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:38.429 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:38.432 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:38.432 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:38.432 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:38.432 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:38.433 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:38.433 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:38.434 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:38.434 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:38.435 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:38.435 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:38.435 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:38.435 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:33:38.436 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:33:38.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.437 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:38.437 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:33:38.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.437 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:38.448 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:33:38.448 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.448 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.449 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.449 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.450 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.450 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:38.450 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.450 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:33:38.454 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:33:38.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.457 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.457 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:33:38.458 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:33:38.459 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.460 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:33:38.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:33:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.478 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:33:38.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:33:38.481 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:38.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.482 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:33:38.482 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:33:38.482 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:33:38.482 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:33:38.483 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.483 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.483 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.484 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:38.484 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:33:38.484 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.485 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.485 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.485 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.485 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.487 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.487 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.496 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:38.497 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.497 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:33:38.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:33:38.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.505 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:33:38.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:33:38.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.512 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:33:38.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:33:38.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.518 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:33:38.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:33:38.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:38.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:33:38.522 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:33:38.522 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:33:38.522 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:33:38.523 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.523 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.523 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.524 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:33:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.525 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.526 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.527 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:38.528 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.528 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.529 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:33:38.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:33:38.534 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.536 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:33:38.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:33:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:33:38.552 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:33:38.554 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.556 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:33:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:33:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:33:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:33:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:33:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:33:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:33:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:33:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:33:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:33:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:33:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:33:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:33:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:33:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:33:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:33:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:33:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:33:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:33:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:33:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:38.639 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:33:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:33:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.652 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:33:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:33:38.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.661 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:33:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:33:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.665 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:33:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:33:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:38.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.694 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:33:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:33:38.695 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:33:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:33:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.696 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.697 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.702 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:38.703 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:33:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.704 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.705 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.706 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.708 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:38.709 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:33:38.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:33:38.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:33:38.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:33:38.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.782 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.783 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:33:38.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:33:38.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.790 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:33:38.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:33:38.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:38.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:33:38.794 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:33:38.794 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:33:38.794 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:33:38.794 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.795 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.795 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.796 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:38.796 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:33:38.796 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.797 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.797 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.797 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.809 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.809 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.810 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.810 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:38.811 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.812 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:33:38.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:33:38.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.818 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:33:38.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:33:38.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.856 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:33:38.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:33:38.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.863 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:38.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:33:38.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:33:38.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:38.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.871 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:33:38.871 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:33:38.875 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:33:38.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:33:38.888 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:38.892 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:38.893 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:38.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.896 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:38.898 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:33:38.898 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:38.899 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:38.904 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:38.904 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:38.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:38.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:38.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:38.906 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:38.906 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:38.907 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:38.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:38.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:33:38.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:38.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:33:38.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:38.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.913 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:38.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:33:38.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:38.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:38.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:33:38.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:38.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:38.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:38.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:38.975 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:38.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:38.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:38.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:38.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:33:38.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:39.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:39.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:33:39.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:39.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:39.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:39.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.021 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:39.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:39.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:39.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:33:39.037 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:39.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:39.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:33:39.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:39.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:39.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:39.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.044 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:33:39.044 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:33:39.045 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:33:39.046 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:33:39.047 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:39.058 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:39.060 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:39.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:39.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:39.061 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:39.061 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:33:39.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.062 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:39.063 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.063 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.064 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.064 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.065 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.065 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:33:39.067 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.071 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:39.071 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.072 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.072 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.073 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.073 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.074 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:33:39.074 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.093 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:39.095 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.096 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.096 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.097 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.097 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.098 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:33:39.098 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.099 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:39.100 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.100 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.101 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.101 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.102 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.102 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:33:39.123 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:39.124 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.125 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.125 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.125 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.125 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:33:39.129 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:39.130 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.143 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.143 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.143 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.144 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:33:39.144 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.145 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:39.145 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.145 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.145 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.145 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.146 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.146 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:33:39.146 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:39.147 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:39.147 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:39.148 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:39.149 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:39.150 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:39.160 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:39.161 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:33:39.166 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:39.167 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:39.168 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:33:39.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:33:39.170 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:33:39.170 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:33:39.171 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:33:39.172 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:33:39.175 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:33:39.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:33:39.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:39.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:33:39.177 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:33:39.178 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:39.178 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:33:39.179 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:39.180 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:39.180 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:33:39.181 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:33:39.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.182 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:33:39.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:39.183 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:39.183 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:33:39.184 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:39.186 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:39.187 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:33:39.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:39.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:39.188 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:33:39.189 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:33:39.189 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:33:39.190 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:33:39.191 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:33:39.191 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:39.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 msgr2=0x7f2848055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 0x7f2848055f30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f286000b000 tx=0x7f2860006df0).stop 2022-07-20T02:33:39.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 msgr2=0x7f286c14e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f2864001b80 tx=0x7f285c014000).stop 2022-07-20T02:33:39.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 shutdown_connections 2022-07-20T02:33:39.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2848053a70 0x7f2848055f30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f286c0b2920 0x7f286c066380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286c066a80 0x7f286c065e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 --2- 172.21.15.26:0/211963132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f286c061050 0x7f286c14e400 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 >> 172.21.15.26:0/211963132 conn(0x7f286c0b60a0 msgr2=0x7f286c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:39.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 shutdown_connections 2022-07-20T02:33:39.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:38.296+0000 7f28711c3700 1 -- 172.21.15.26:0/211963132 wait complete. 2022-07-20T02:33:39.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 Processor -- start 2022-07-20T02:33:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 -- start start 2022-07-20T02:33:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062c0ae370 con 0x7f062c0614b0 2022-07-20T02:33:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062c062070 con 0x7f062c0adf90 2022-07-20T02:33:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062c0b28b0 con 0x7f062c0b14b0 2022-07-20T02:33:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062bfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f062c0614b0 0x7f062c061960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40988/0 (socket says 172.21.15.26:40988) 2022-07-20T02:33:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062bfff700 1 -- 172.21.15.26:0/2526068719 learned_addr learned my addr 172.21.15.26:0/2526068719 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:39.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062affd700 1 --2- 172.21.15.26:0/2526068719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979039214 0 0) 0x7f062c0ae370 con 0x7f062c0614b0 2022-07-20T02:33:39.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0610003260 con 0x7f062c0614b0 2022-07-20T02:33:39.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042269916 0 0) 0x7f062c062070 con 0x7f062c0adf90 2022-07-20T02:33:39.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f062c0ae370 con 0x7f062c0adf90 2022-07-20T02:33:39.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062affd700 1 -- 172.21.15.26:0/2526068719 >> v1:172.21.15.120:6789/0 conn(0x7f062c0adf90 legacy=0x7f062c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062affd700 1 -- 172.21.15.26:0/2526068719 >> v1:172.21.15.26:6789/0 conn(0x7f062c0614b0 legacy=0x7f062c061960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062affd700 1 -- 172.21.15.26:0/2526068719 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c0b0610 con 0x7f062c0b14b0 2022-07-20T02:33:39.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062affd700 1 --2- 172.21.15.26:0/2526068719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c0b2c60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f06240032f0 tx=0x7f06240091d0).ready entity=mon.2 client_cookie=cc9de7b9f33c8c24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400b640 con 0x7f062c0b14b0 2022-07-20T02:33:39.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062400b7a0 con 0x7f062c0b14b0 2022-07-20T02:33:39.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f062a7fc700 1 -- 172.21.15.26:0/2526068719 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400b910 con 0x7f062c0b14b0 2022-07-20T02:33:39.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 -- 172.21.15.26:0/2526068719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 msgr2=0x7f062c0b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.332+0000 7f0631ed4700 1 --2- 172.21.15.26:0/2526068719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c0b2c60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f06240032f0 tx=0x7f06240091d0).stop 2022-07-20T02:33:39.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- 172.21.15.26:0/2526068719 shutdown_connections 2022-07-20T02:33:39.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 --2- 172.21.15.26:0/2526068719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c0b2c60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- 172.21.15.26:0/2526068719 >> 172.21.15.26:0/2526068719 conn(0x7f062c0b66e0 msgr2=0x7f062c0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:39.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- 172.21.15.26:0/2526068719 shutdown_connections 2022-07-20T02:33:39.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- 172.21.15.26:0/2526068719 wait complete. 2022-07-20T02:33:39.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 Processor -- start 2022-07-20T02:33:39.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- start start 2022-07-20T02:33:39.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f062c0614b0 0x7f062c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c065e00 con 0x7f062c0adf90 2022-07-20T02:33:39.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c0650a0 con 0x7f062c0614b0 2022-07-20T02:33:39.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0631ed4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062c064e00 con 0x7f062c0b14b0 2022-07-20T02:33:39.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41712/0 (socket says 172.21.15.26:41712) 2022-07-20T02:33:39.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 -- 172.21.15.26:0/4284511862 learned_addr learned my addr 172.21.15.26:0/4284511862 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:39.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062b7fe700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f062c0614b0 0x7f062c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062bfff700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 -- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 msgr2=0x7f062c150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 -- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f062c0614b0 msgr2=0x7f062c148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f062c0614b0 0x7f062c148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 -- 172.21.15.26:0/4284511862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624009470 con 0x7f062c0adf90 2022-07-20T02:33:39.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f062affd700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f06240032f0 tx=0x7f0624016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.336+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400b910 con 0x7f062c0adf90 2022-07-20T02:33:39.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06240205e0 con 0x7f062c0adf90 2022-07-20T02:33:39.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062c066060 con 0x7f062c0adf90 2022-07-20T02:33:39.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400b640 con 0x7f062c0adf90 2022-07-20T02:33:39.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062c063c10 con 0x7f062c0adf90 2022-07-20T02:33:39.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062c150210 con 0x7f062c0adf90 2022-07-20T02:33:39.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f062401e070 con 0x7f062c0adf90 2022-07-20T02:33:39.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0628ff9700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 0x7f0604055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f062406cdf0 con 0x7f062c0adf90 2022-07-20T02:33:39.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f062b7fe700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 0x7f0604055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.340+0000 7f062b7fe700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 0x7f0604055fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f061400d000 tx=0x7f0614000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.348+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0624043bc0 con 0x7f062c0adf90 2022-07-20T02:33:39.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.516+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f062c0574e0 con 0x7f062c0adf90 2022-07-20T02:33:39.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.516+0000 7f0628ff9700 1 -- 172.21.15.26:0/4284511862 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2493 (secure 0 0 0) 0x7f062401b090 con 0x7f062c0adf90 2022-07-20T02:33:39.525 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:39.525 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:28.135776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7398":{"gid":7398,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/3259008170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3259008170},{"type":"v1","addr":"172.21.15.26:6837","nonce":3259008170}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 msgr2=0x7f0604055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 0x7f0604055fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f061400d000 tx=0x7f0614000bc0).stop 2022-07-20T02:33:39.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 msgr2=0x7f062c150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f06240032f0 tx=0x7f0624016000).stop 2022-07-20T02:33:39.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 shutdown_connections 2022-07-20T02:33:39.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f062c0adf90 0x7f062c150430 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0604053b00 0x7f0604055fc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f062c0b14b0 0x7f062c150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 --2- 172.21.15.26:0/4284511862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f062c0614b0 0x7f062c148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 >> 172.21.15.26:0/4284511862 conn(0x7f062c0b66e0 msgr2=0x7f062c061960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 shutdown_connections 2022-07-20T02:33:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.520+0000 7f0631ed4700 1 -- 172.21.15.26:0/4284511862 wait complete. 2022-07-20T02:33:39.530 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 55 2022-07-20T02:33:39.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:33:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 Processor -- start 2022-07-20T02:33:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 -- start start 2022-07-20T02:33:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 0x7f76f0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f76f0060f00 0x7f76f013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f01474e0 con 0x7f76f005cfd0 2022-07-20T02:33:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f00b4e40 con 0x7f76f0060f00 2022-07-20T02:33:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76effff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f005d3b0 con 0x7f76f0147000 2022-07-20T02:33:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76eeffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f76f005cfd0 0x7f76f00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40998/0 (socket says 172.21.15.26:40998) 2022-07-20T02:33:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76eeffd700 1 -- 172.21.15.26:0/2788950244 learned_addr learned my addr 172.21.15.26:0/2788950244 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76ef7fe700 1 --2- 172.21.15.26:0/2788950244 >> v2:172.21.15.120:3300/0 conn(0x7f76f0060f00 0x7f76f013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.640+0000 7f76e7fff700 1 --2- 172.21.15.26:0/2788950244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 0x7f76f0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76ee7fc700 1 -- 172.21.15.26:0/2788950244 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541692475 0 0) 0x7f76f01474e0 con 0x7f76f005cfd0 2022-07-20T02:33:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76ee7fc700 1 -- 172.21.15.26:0/2788950244 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d0003260 con 0x7f76f005cfd0 2022-07-20T02:33:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76e7fff700 1 -- 172.21.15.26:0/2788950244 >> v2:172.21.15.120:3300/0 conn(0x7f76f0060f00 msgr2=0x7f76f013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76e7fff700 1 --2- 172.21.15.26:0/2788950244 >> v2:172.21.15.120:3300/0 conn(0x7f76f0060f00 0x7f76f013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76e7fff700 1 -- 172.21.15.26:0/2788950244 >> v1:172.21.15.26:6789/0 conn(0x7f76f005cfd0 legacy=0x7f76f00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76e7fff700 1 -- 172.21.15.26:0/2788950244 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f01431e0 con 0x7f76f0147000 2022-07-20T02:33:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76e7fff700 1 --2- 172.21.15.26:0/2788950244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 0x7f76f0143bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f76d8002f70 tx=0x7f76d8006df0).ready entity=mon.2 client_cookie=4fac777d49011b1a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76ee7fc700 1 -- 172.21.15.26:0/2788950244 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76d800d4a0 con 0x7f76f0147000 2022-07-20T02:33:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76ee7fc700 1 -- 172.21.15.26:0/2788950244 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8000e90 con 0x7f76f0147000 2022-07-20T02:33:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76ee7fc700 1 -- 172.21.15.26:0/2788950244 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76d800d7f0 con 0x7f76f0147000 2022-07-20T02:33:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- 172.21.15.26:0/2788950244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 msgr2=0x7f76f0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 --2- 172.21.15.26:0/2788950244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 0x7f76f0143bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f76d8002f70 tx=0x7f76d8006df0).stop 2022-07-20T02:33:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- 172.21.15.26:0/2788950244 shutdown_connections 2022-07-20T02:33:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 --2- 172.21.15.26:0/2788950244 >> v2:172.21.15.120:3300/0 conn(0x7f76f0060f00 0x7f76f013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 --2- 172.21.15.26:0/2788950244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f0147000 0x7f76f0143bc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- 172.21.15.26:0/2788950244 >> 172.21.15.26:0/2788950244 conn(0x7f76f00620c0 msgr2=0x7f76f005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:39.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- 172.21.15.26:0/2788950244 shutdown_connections 2022-07-20T02:33:39.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- 172.21.15.26:0/2788950244 wait complete. 2022-07-20T02:33:39.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 Processor -- start 2022-07-20T02:33:39.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.644+0000 7f76effff700 1 -- start start 2022-07-20T02:33:39.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f0060f00 0x7f76f00b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f00b00f0 0x7f76f0145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00ae280 con 0x7f76f0147000 2022-07-20T02:33:39.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0146720 con 0x7f76f0060f00 2022-07-20T02:33:39.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f01469c0 con 0x7f76f00b00f0 2022-07-20T02:33:39.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76eeffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f0060f00 0x7f76f00b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76ef7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f00b00f0 0x7f76f0145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41728/0 (socket says 172.21.15.26:41728) 2022-07-20T02:33:39.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 -- 172.21.15.26:0/1666851480 learned_addr learned my addr 172.21.15.26:0/1666851480 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:39.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 -- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f00b00f0 msgr2=0x7f76f0145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f00b00f0 0x7f76f0145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 -- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f0060f00 msgr2=0x7f76f00b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:39.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f0060f00 0x7f76f00b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:39.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 -- 172.21.15.26:0/1666851480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d800b180 con 0x7f76f0147000 2022-07-20T02:33:39.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76e7fff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f76d8002f70 tx=0x7f76d8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76d8012930 con 0x7f76f0147000 2022-07-20T02:33:39.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8012a90 con 0x7f76f0147000 2022-07-20T02:33:39.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f0144f10 con 0x7f76f0147000 2022-07-20T02:33:39.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76d8012df0 con 0x7f76f0147000 2022-07-20T02:33:39.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f0145ef0 con 0x7f76f0147000 2022-07-20T02:33:39.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.648+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f00aeda0 con 0x7f76f0147000 2022-07-20T02:33:39.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.652+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f76d801c070 con 0x7f76f0147000 2022-07-20T02:33:39.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.652+0000 7f76ecff9700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 0x7f76d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:39.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.652+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f76d806d040 con 0x7f76f0147000 2022-07-20T02:33:39.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.652+0000 7f76eeffd700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 0x7f76d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:39.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.652+0000 7f76eeffd700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 0x7f76d4055f30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f76e000d000 tx=0x7f76e0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:39.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.656+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76d8043eb0 con 0x7f76f0147000 2022-07-20T02:33:39.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:39.820+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f76f01463d0 con 0x7f76f0147000 2022-07-20T02:33:40.621 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:33:40.620+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:33:40.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.628+0000 7f76ecff9700 1 -- 172.21.15.26:0/1666851480 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) v1 ==== 161+0+0 (secure 0 0 0) 0x7f76d8019030 con 0x7f76f0147000 2022-07-20T02:33:40.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 msgr2=0x7f76d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 0x7f76d4055f30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f76e000d000 tx=0x7f76e0009040).stop 2022-07-20T02:33:40.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 msgr2=0x7f76f00add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f76d8002f70 tx=0x7f76d8014000).stop 2022-07-20T02:33:40.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 shutdown_connections 2022-07-20T02:33:40.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f0147000 0x7f76f00add90 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f76d4053a70 0x7f76d4055f30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76f00b00f0 0x7f76f0145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 --2- 172.21.15.26:0/1666851480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76f0060f00 0x7f76f00b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 >> 172.21.15.26:0/1666851480 conn(0x7f76f005cfd0 msgr2=0x7f76f00b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:40.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 shutdown_connections 2022-07-20T02:33:40.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.632+0000 7f76effff700 1 -- 172.21.15.26:0/1666851480 wait complete. 2022-07-20T02:33:40.648 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:33:40.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:40.660 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:33:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 Processor -- start 2022-07-20T02:33:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 -- start start 2022-07-20T02:33:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d9814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:40.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d98064880 con 0x7f9d980670c0 2022-07-20T02:33:40.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d980b35a0 con 0x7f9d98064450 2022-07-20T02:33:40.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e8a0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980b85f0 con 0x7f9d981444a0 2022-07-20T02:33:40.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9d89e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9d980670c0 0x7f9d980b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41024/0 (socket says 172.21.15.26:41024) 2022-07-20T02:33:40.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9d89e700 1 -- 172.21.15.26:0/3534948032 learned_addr learned my addr 172.21.15.26:0/3534948032 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:40.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e09f700 1 --2- 172.21.15.26:0/3534948032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d9814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:40.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384080214 0 0) 0x7f9d98064880 con 0x7f9d980670c0 2022-07-20T02:33:40.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d80003260 con 0x7f9d980670c0 2022-07-20T02:33:40.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874234440 0 0) 0x7f9d980b35a0 con 0x7f9d98064450 2022-07-20T02:33:40.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98064880 con 0x7f9d98064450 2022-07-20T02:33:40.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3534948032 >> v1:172.21.15.120:6789/0 conn(0x7f9d98064450 legacy=0x7f9d98064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3534948032 >> v1:172.21.15.26:6789/0 conn(0x7f9d980670c0 legacy=0x7f9d980b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3534948032 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980b80b0 con 0x7f9d981444a0 2022-07-20T02:33:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9e09f700 1 --2- 172.21.15.26:0/3534948032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d9814c850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9d90002f70 tx=0x7f9d90006df0).ready entity=mon.2 client_cookie=c55a46b03e47d9cc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d9000d5c0 con 0x7f9d981444a0 2022-07-20T02:33:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90000e90 con 0x7f9d981444a0 2022-07-20T02:33:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.764+0000 7f9d9c89c700 1 -- 172.21.15.26:0/3534948032 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d9000d910 con 0x7f9d981444a0 2022-07-20T02:33:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3534948032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 msgr2=0x7f9d9814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3534948032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d9814c850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9d90002f70 tx=0x7f9d90006df0).stop 2022-07-20T02:33:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3534948032 shutdown_connections 2022-07-20T02:33:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3534948032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d9814c850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3534948032 >> 172.21.15.26:0/3534948032 conn(0x7f9d9805f840 msgr2=0x7f9d9805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:40.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3534948032 shutdown_connections 2022-07-20T02:33:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3534948032 wait complete. 2022-07-20T02:33:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 Processor -- start 2022-07-20T02:33:40.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- start start 2022-07-20T02:33:40.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d980670c0 0x7f9d98141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d980ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980ae050 con 0x7f9d9814ff70 2022-07-20T02:33:40.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980ae2f0 con 0x7f9d980670c0 2022-07-20T02:33:40.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e8a0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98146610 con 0x7f9d981444a0 2022-07-20T02:33:40.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e09f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:40.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e09f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41754/0 (socket says 172.21.15.26:41754) 2022-07-20T02:33:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.768+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3315252515 learned_addr learned my addr 172.21.15.26:0/3315252515 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9d89e700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d980670c0 0x7f9d98141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9d09d700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d980ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 msgr2=0x7f9d980ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d980ad1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d980670c0 msgr2=0x7f9d98141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d980670c0 0x7f9d98141a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 -- 172.21.15.26:0/3315252515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9000b180 con 0x7f9d9814ff70 2022-07-20T02:33:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e09f700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f9d90002f70 tx=0x7f9d90014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d90013de0 con 0x7f9d9814ff70 2022-07-20T02:33:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9000de70 con 0x7f9d9814ff70 2022-07-20T02:33:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d9001e5f0 con 0x7f9d9814ff70 2022-07-20T02:33:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d981468b0 con 0x7f9d9814ff70 2022-07-20T02:33:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d980aeb30 con 0x7f9d9814ff70 2022-07-20T02:33:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d98146a40 con 0x7f9d9814ff70 2022-07-20T02:33:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.772+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9d9001c070 con 0x7f9d9814ff70 2022-07-20T02:33:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.776+0000 7f9d8affd700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 0x7f9d78055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.776+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f9d9006cb10 con 0x7f9d9814ff70 2022-07-20T02:33:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.776+0000 7f9d9d89e700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 0x7f9d78055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:40.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.776+0000 7f9d9d89e700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 0x7f9d78055ff0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f9d9400d000 tx=0x7f9d94009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:40.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.784+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d90043770 con 0x7f9d9814ff70 2022-07-20T02:33:40.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.944+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d980b8350 con 0x7f9d9814ff70 2022-07-20T02:33:40.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d8affd700 1 -- 172.21.15.26:0/3315252515 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2133 (secure 0 0 0) 0x7f9d90023070 con 0x7f9d9814ff70 2022-07-20T02:33:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:40.951 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:26.130844+0000","modified":"2022-07-20T02:33:40.621639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T02:33:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 msgr2=0x7f9d78055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 0x7f9d78055ff0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f9d9400d000 tx=0x7f9d94009040).stop 2022-07-20T02:33:40.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 msgr2=0x7f9d980b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:40.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f9d90002f70 tx=0x7f9d90014000).stop 2022-07-20T02:33:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 shutdown_connections 2022-07-20T02:33:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9d9814ff70 0x7f9d980b0f90 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9d78053b30 0x7f9d78055ff0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d981444a0 0x7f9d980ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 --2- 172.21.15.26:0/3315252515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d980670c0 0x7f9d98141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 >> 172.21.15.26:0/3315252515 conn(0x7f9d9805f840 msgr2=0x7f9d980b6170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 shutdown_connections 2022-07-20T02:33:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:40.948+0000 7f9d9e8a0700 1 -- 172.21.15.26:0/3315252515 wait complete. 2022-07-20T02:33:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 56 2022-07-20T02:33:40.962 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:33:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.068+0000 7eff03213700 1 Processor -- start 2022-07-20T02:33:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.068+0000 7eff03213700 1 -- start start 2022-07-20T02:33:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 0x7efefc1445b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efefc097c50 0x7efefc144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc0b3640 con 0x7efefc0b3260 2022-07-20T02:33:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc095300 con 0x7efefc097c50 2022-07-20T02:33:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0955a0 con 0x7efefc060c00 2022-07-20T02:33:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efefc097c50 0x7efefc144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02211700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efefc0b3260 0x7efefc0604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41038/0 (socket says 172.21.15.26:41038) 2022-07-20T02:33:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efefc097c50 0x7efefc144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41762/0 (socket says 172.21.15.26:41762) 2022-07-20T02:33:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02211700 1 -- 172.21.15.26:0/2310553231 learned_addr learned my addr 172.21.15.26:0/2310553231 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff01a10700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 0x7efefc1445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff0120f700 1 -- 172.21.15.26:0/2310553231 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203895234 0 0) 0x7efefc0b3640 con 0x7efefc0b3260 2022-07-20T02:33:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 -- 172.21.15.26:0/2310553231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 msgr2=0x7efefc1445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 0x7efefc1445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 -- 172.21.15.26:0/2310553231 >> v1:172.21.15.26:6789/0 conn(0x7efefc0b3260 legacy=0x7efefc0604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 -- 172.21.15.26:0/2310553231 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc095840 con 0x7efefc097c50 2022-07-20T02:33:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff01a10700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 0x7efefc1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff02a12700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc097c50 0x7efefc144aa0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4006df0).ready entity=mon.0 client_cookie=395b162b30ad6935 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff0120f700 1 -- 172.21.15.26:0/2310553231 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef400d570 con 0x7efefc097c50 2022-07-20T02:33:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff0120f700 1 -- 172.21.15.26:0/2310553231 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4000e90 con 0x7efefc097c50 2022-07-20T02:33:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff0120f700 1 -- 172.21.15.26:0/2310553231 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef400d8c0 con 0x7efefc097c50 2022-07-20T02:33:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- 172.21.15.26:0/2310553231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc097c50 msgr2=0x7efefc144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc097c50 0x7efefc144aa0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4006df0).stop 2022-07-20T02:33:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- 172.21.15.26:0/2310553231 shutdown_connections 2022-07-20T02:33:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc097c50 0x7efefc144aa0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 --2- 172.21.15.26:0/2310553231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc060c00 0x7efefc1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- 172.21.15.26:0/2310553231 >> 172.21.15.26:0/2310553231 conn(0x7efefc0b67f0 msgr2=0x7efefc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- 172.21.15.26:0/2310553231 shutdown_connections 2022-07-20T02:33:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.072+0000 7eff03213700 1 -- 172.21.15.26:0/2310553231 wait complete. 2022-07-20T02:33:41.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 Processor -- start 2022-07-20T02:33:41.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- start start 2022-07-20T02:33:41.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efefc097c50 0x7efefc156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc0b3260 0x7efefc152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc156840 con 0x7efefc060c00 2022-07-20T02:33:41.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0a9e20 con 0x7efefc097c50 2022-07-20T02:33:41.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0aa0c0 con 0x7efefc0b3260 2022-07-20T02:33:41.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41764/0 (socket says 172.21.15.26:41764) 2022-07-20T02:33:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 -- 172.21.15.26:0/2104653227 learned_addr learned my addr 172.21.15.26:0/2104653227 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02a12700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc0b3260 0x7efefc152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff01a10700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efefc097c50 0x7efefc156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 -- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc0b3260 msgr2=0x7efefc152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc0b3260 0x7efefc152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 -- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efefc097c50 msgr2=0x7efefc156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efefc097c50 0x7efefc156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 -- 172.21.15.26:0/2104653227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef8009470 con 0x7efefc060c00 2022-07-20T02:33:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff02211700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7efeec001000 tx=0x7efeec007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeec0070d0 con 0x7efefc060c00 2022-07-20T02:33:41.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec007230 con 0x7efefc060c00 2022-07-20T02:33:41.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef400b180 con 0x7efefc060c00 2022-07-20T02:33:41.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeec0055d0 con 0x7efefc060c00 2022-07-20T02:33:41.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.076+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc0aa600 con 0x7efefc060c00 2022-07-20T02:33:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc152440 con 0x7efefc060c00 2022-07-20T02:33:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efeec01a060 con 0x7efefc060c00 2022-07-20T02:33:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7efef37fe700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 0x7efedc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7eff01a10700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 0x7efedc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6506+0+0 (secure 0 0 0) 0x7efeec06b8c0 con 0x7efefc060c00 2022-07-20T02:33:41.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.080+0000 7eff01a10700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 0x7efedc055fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7efef800b000 tx=0x7efef80091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.088+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efeec042520 con 0x7efefc060c00 2022-07-20T02:33:41.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.244+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efefc0993f0 con 0x7efefc060c00 2022-07-20T02:33:41.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.636+0000 7efef37fe700 1 -- 172.21.15.26:0/2104653227 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7efeec018020 con 0x7efefc060c00 2022-07-20T02:33:41.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 msgr2=0x7efedc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 0x7efedc055fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7efef800b000 tx=0x7efef80091d0).stop 2022-07-20T02:33:41.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 msgr2=0x7efefc0b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7efeec001000 tx=0x7efeec007c30).stop 2022-07-20T02:33:41.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 shutdown_connections 2022-07-20T02:33:41.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efedc053b00 0x7efedc055fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efefc0b3260 0x7efefc152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efefc097c50 0x7efefc156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 --2- 172.21.15.26:0/2104653227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efefc060c00 0x7efefc0b08c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 >> 172.21.15.26:0/2104653227 conn(0x7efefc0b67f0 msgr2=0x7efefc144430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 shutdown_connections 2022-07-20T02:33:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.640+0000 7eff03213700 1 -- 172.21.15.26:0/2104653227 wait complete. 2022-07-20T02:33:41.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:33:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 Processor -- start 2022-07-20T02:33:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 -- start start 2022-07-20T02:33:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 0x7fbafc061780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbafc0aa070 0x7fbafc061c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc067030 con 0x7fbafc0af4a0 2022-07-20T02:33:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc0aeec0 con 0x7fbafc0aa070 2022-07-20T02:33:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb03491700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc066af0 con 0x7fbafc0aeae0 2022-07-20T02:33:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb01a2e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbafc0aa070 0x7fbafc061c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb0122d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 0x7fbafc061780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb00a2c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbafc0af4a0 0x7fbafc14ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:55264/0 (socket says 172.21.15.26:55264) 2022-07-20T02:33:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbb00a2c700 1 -- 172.21.15.26:0/1749228325 learned_addr learned my addr 172.21.15.26:0/1749228325 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbafbfff700 1 -- 172.21.15.26:0/1749228325 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855921491 0 0) 0x7fbafc067030 con 0x7fbafc0af4a0 2022-07-20T02:33:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.775+0000 7fbafbfff700 1 -- 172.21.15.26:0/1749228325 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8003260 con 0x7fbafc0af4a0 2022-07-20T02:33:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb01a2e700 1 -- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 msgr2=0x7fbafc061780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb01a2e700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 0x7fbafc061780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb01a2e700 1 -- 172.21.15.26:0/1749228325 >> v1:172.21.15.120:6789/0 conn(0x7fbafc0af4a0 legacy=0x7fbafc14ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb01a2e700 1 -- 172.21.15.26:0/1749228325 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc066d90 con 0x7fbafc0aa070 2022-07-20T02:33:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb0122d700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 0x7fbafc061780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb01a2e700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc061c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fbaf40032f0 tx=0x7fbaf40091d0).ready entity=mon.1 client_cookie=70204af4efb2c514 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbafbfff700 1 -- 172.21.15.26:0/1749228325 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf400b600 con 0x7fbafc0aa070 2022-07-20T02:33:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbafbfff700 1 -- 172.21.15.26:0/1749228325 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf400b760 con 0x7fbafc0aa070 2022-07-20T02:33:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbafbfff700 1 -- 172.21.15.26:0/1749228325 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf400b8d0 con 0x7fbafc0aa070 2022-07-20T02:33:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 -- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 msgr2=0x7fbafc061c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc061c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fbaf40032f0 tx=0x7fbaf40091d0).stop 2022-07-20T02:33:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 -- 172.21.15.26:0/1749228325 shutdown_connections 2022-07-20T02:33:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc061c70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 --2- 172.21.15.26:0/1749228325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0aeae0 0x7fbafc061780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 -- 172.21.15.26:0/1749228325 >> 172.21.15.26:0/1749228325 conn(0x7fbafc0b66e0 msgr2=0x7fbafc0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 -- 172.21.15.26:0/1749228325 shutdown_connections 2022-07-20T02:33:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.779+0000 7fbb03491700 1 -- 172.21.15.26:0/1749228325 wait complete. 2022-07-20T02:33:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 Processor -- start 2022-07-20T02:33:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 -- start start 2022-07-20T02:33:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0af4a0 0x7fbafc15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc156770 con 0x7fbafc0aeae0 2022-07-20T02:33:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0647a0 con 0x7fbafc0aa070 2022-07-20T02:33:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb03491700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc063ca0 con 0x7fbafc0af4a0 2022-07-20T02:33:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41780/0 (socket says 172.21.15.26:41780) 2022-07-20T02:33:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 -- 172.21.15.26:0/3801525940 learned_addr learned my addr 172.21.15.26:0/3801525940 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb0122d700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb01a2e700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0af4a0 0x7fbafc15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 -- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0af4a0 msgr2=0x7fbafc15a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0af4a0 0x7fbafc15a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 -- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 msgr2=0x7fbafc14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc14e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb00a2c700 1 -- 172.21.15.26:0/3801525940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec00b180 con 0x7fbafc0aeae0 2022-07-20T02:33:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.783+0000 7fbb0122d700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbb00a2c700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fbafc066610 tx=0x7fbaf0007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf00070d0 con 0x7fbafc0aeae0 2022-07-20T02:33:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0007230 con 0x7fbafc0aeae0 2022-07-20T02:33:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4009470 con 0x7fbafc0aeae0 2022-07-20T02:33:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf0005610 con 0x7fbafc0aeae0 2022-07-20T02:33:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc148a90 con 0x7fbafc0aeae0 2022-07-20T02:33:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc064370 con 0x7fbafc0aeae0 2022-07-20T02:33:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbaf001a060 con 0x7fbafc0aeae0 2022-07-20T02:33:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 0x7fbadc055ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fbaf006b990 con 0x7fbafc0aeae0 2022-07-20T02:33:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.787+0000 7fbb0122d700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 0x7fbadc055ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.791+0000 7fbb0122d700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 0x7fbadc055ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fbaec00b000 tx=0x7fbaec006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.795+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbaf00425f0 con 0x7fbafc0aeae0 2022-07-20T02:33:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:41.923+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbafc0ad440 con 0x7fbafc0aeae0 2022-07-20T02:33:42.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.651+0000 7fbafa7fc700 1 -- 172.21.15.26:0/3801525940 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) v1 ==== 185+0+0 (secure 0 0 0) 0x7fbaf0018070 con 0x7fbafc0aeae0 2022-07-20T02:33:42.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 msgr2=0x7fbadc055ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 0x7fbadc055ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fbaec00b000 tx=0x7fbaec006df0).stop 2022-07-20T02:33:42.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 msgr2=0x7fbafc15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fbafc066610 tx=0x7fbaf0007b10).stop 2022-07-20T02:33:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 shutdown_connections 2022-07-20T02:33:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbafc0aeae0 0x7fbafc15a000 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbadc053a30 0x7fbadc055ef0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbafc0af4a0 0x7fbafc15a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 --2- 172.21.15.26:0/3801525940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbafc0aa070 0x7fbafc14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 >> 172.21.15.26:0/3801525940 conn(0x7fbafc0b66e0 msgr2=0x7fbafc0b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:42.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 shutdown_connections 2022-07-20T02:33:42.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.655+0000 7fbb03491700 1 -- 172.21.15.26:0/3801525940 wait complete. 2022-07-20T02:33:42.666 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:33:42.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:33:42.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.787+0000 7f57f14f8700 1 Processor -- start 2022-07-20T02:33:42.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.787+0000 7f57f14f8700 1 -- start start 2022-07-20T02:33:42.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57ec05cfd0 0x7f57ec143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57ec060f00 0x7f57ec13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec1474e0 con 0x7f57ec147000 2022-07-20T02:33:42.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec0b4e40 con 0x7f57ec060f00 2022-07-20T02:33:42.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec05d3b0 con 0x7f57ec05cfd0 2022-07-20T02:33:42.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57ec060f00 0x7f57ec13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57e37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f57ec147000 0x7f57ec0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41070/0 (socket says 172.21.15.26:41070) 2022-07-20T02:33:42.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57ec060f00 0x7f57ec13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41788/0 (socket says 172.21.15.26:41788) 2022-07-20T02:33:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57e37fe700 1 -- 172.21.15.26:0/2322772279 learned_addr learned my addr 172.21.15.26:0/2322772279 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:42.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57ebfff700 1 --2- 172.21.15.26:0/2322772279 >> v2:172.21.15.120:3300/0 conn(0x7f57ec05cfd0 0x7f57ec143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 -- 172.21.15.26:0/2322772279 >> v2:172.21.15.120:3300/0 conn(0x7f57ec05cfd0 msgr2=0x7f57ec143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:42.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 --2- 172.21.15.26:0/2322772279 >> v2:172.21.15.120:3300/0 conn(0x7f57ec05cfd0 0x7f57ec143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 -- 172.21.15.26:0/2322772279 >> v1:172.21.15.26:6789/0 conn(0x7f57ec147000 legacy=0x7f57ec0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57eb7fe700 1 -- 172.21.15.26:0/2322772279 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53344012 0 0) 0x7f57ec1474e0 con 0x7f57ec147000 2022-07-20T02:33:42.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 -- 172.21.15.26:0/2322772279 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec1431e0 con 0x7f57ec060f00 2022-07-20T02:33:42.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f0cf7700 1 --2- 172.21.15.26:0/2322772279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec060f00 0x7f57ec13c9b0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f57e40032f0 tx=0x7f57e40091d0).ready entity=mon.0 client_cookie=9a706b0c2b48c12 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:42.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57eb7fe700 1 -- 172.21.15.26:0/2322772279 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57e400b680 con 0x7f57ec060f00 2022-07-20T02:33:42.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57eb7fe700 1 -- 172.21.15.26:0/2322772279 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e400b7e0 con 0x7f57ec060f00 2022-07-20T02:33:42.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57eb7fe700 1 -- 172.21.15.26:0/2322772279 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57e400b950 con 0x7f57ec060f00 2022-07-20T02:33:42.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- 172.21.15.26:0/2322772279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec060f00 msgr2=0x7f57ec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 --2- 172.21.15.26:0/2322772279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec060f00 0x7f57ec13c9b0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f57e40032f0 tx=0x7f57e40091d0).stop 2022-07-20T02:33:42.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- 172.21.15.26:0/2322772279 shutdown_connections 2022-07-20T02:33:42.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 --2- 172.21.15.26:0/2322772279 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec060f00 0x7f57ec13c9b0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 --2- 172.21.15.26:0/2322772279 >> v2:172.21.15.120:3300/0 conn(0x7f57ec05cfd0 0x7f57ec143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- 172.21.15.26:0/2322772279 >> 172.21.15.26:0/2322772279 conn(0x7f57ec0620c0 msgr2=0x7f57ec05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:42.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- 172.21.15.26:0/2322772279 shutdown_connections 2022-07-20T02:33:42.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- 172.21.15.26:0/2322772279 wait complete. 2022-07-20T02:33:42.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 Processor -- start 2022-07-20T02:33:42.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.791+0000 7f57f14f8700 1 -- start start 2022-07-20T02:33:42.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57ec060f00 0x7f57ec13d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57ec147000 0x7f57ec0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec13caf0 con 0x7f57ec0b7480 2022-07-20T02:33:42.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec153050 con 0x7f57ec147000 2022-07-20T02:33:42.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec13cd20 con 0x7f57ec060f00 2022-07-20T02:33:42.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41796/0 (socket says 172.21.15.26:41796) 2022-07-20T02:33:42.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 -- 172.21.15.26:0/3642395484 learned_addr learned my addr 172.21.15.26:0/3642395484 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:42.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57e37fe700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57ec147000 0x7f57ec0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57ebfff700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57ec060f00 0x7f57ec13d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 -- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57ec060f00 msgr2=0x7f57ec13d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57ec060f00 0x7f57ec13d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 -- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57ec147000 msgr2=0x7f57ec0ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:42.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57ec147000 0x7f57ec0ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:42.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 -- 172.21.15.26:0/3642395484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4009470 con 0x7f57ec0b7480 2022-07-20T02:33:42.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f0cf7700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f57e40032f0 tx=0x7f57e4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:42.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57e400b950 con 0x7f57ec0b7480 2022-07-20T02:33:42.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e4021630 con 0x7f57ec0b7480 2022-07-20T02:33:42.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec150090 con 0x7f57ec0b7480 2022-07-20T02:33:42.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57e400b680 con 0x7f57ec0b7480 2022-07-20T02:33:42.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.795+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec0b0580 con 0x7f57ec0b7480 2022-07-20T02:33:42.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec143480 con 0x7f57ec0b7480 2022-07-20T02:33:42.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f57e400f070 con 0x7f57ec0b7480 2022-07-20T02:33:42.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57e9ffb700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 0x7f57d0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:42.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f57e406d340 con 0x7f57ec0b7480 2022-07-20T02:33:42.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57ebfff700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 0x7f57d0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.799+0000 7f57ebfff700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 0x7f57d0055f30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f57dc00b000 tx=0x7f57dc006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.803+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57e40441e0 con 0x7f57ec0b7480 2022-07-20T02:33:42.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:42.931+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57ec0574e0 con 0x7f57ec0b7480 2022-07-20T02:33:43.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.671+0000 7f57e9ffb700 1 -- 172.21.15.26:0/3642395484 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) v1 ==== 173+0+0 (secure 0 0 0) 0x7f57e4015020 con 0x7f57ec0b7480 2022-07-20T02:33:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 msgr2=0x7f57d0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 0x7f57d0055f30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f57dc00b000 tx=0x7f57dc006df0).stop 2022-07-20T02:33:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 msgr2=0x7f57ec0ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f57e40032f0 tx=0x7f57e4016000).stop 2022-07-20T02:33:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 shutdown_connections 2022-07-20T02:33:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57ec0b7480 0x7f57ec0ae930 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57d0053a70 0x7f57d0055f30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57ec147000 0x7f57ec0ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 --2- 172.21.15.26:0/3642395484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57ec060f00 0x7f57ec13d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 >> 172.21.15.26:0/3642395484 conn(0x7f57ec05cfd0 msgr2=0x7f57ec0b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 shutdown_connections 2022-07-20T02:33:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.675+0000 7f57f14f8700 1 -- 172.21.15.26:0/3642395484 wait complete. 2022-07-20T02:33:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:33:43.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:33:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 Processor -- start 2022-07-20T02:33:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 -- start start 2022-07-20T02:33:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57a4147be0 0x7f57a40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57a40a7fc0 0x7f57a40af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40b62d0 con 0x7f57a40b5ef0 2022-07-20T02:33:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a414aac0 con 0x7f57a40a7fc0 2022-07-20T02:33:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.823+0000 7f57a958d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a414ad60 con 0x7f57a4147be0 2022-07-20T02:33:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57a40a7fc0 0x7f57a40af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f57a40a7fc0 0x7f57a40af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41804/0 (socket says 172.21.15.26:41804) 2022-07-20T02:33:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 -- 172.21.15.26:0/334629478 learned_addr learned my addr 172.21.15.26:0/334629478 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a37fe700 1 --2- 172.21.15.26:0/334629478 >> v2:172.21.15.120:3300/0 conn(0x7f57a4147be0 0x7f57a40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 -- 172.21.15.26:0/334629478 >> v2:172.21.15.120:3300/0 conn(0x7f57a4147be0 msgr2=0x7f57a40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 --2- 172.21.15.26:0/334629478 >> v2:172.21.15.120:3300/0 conn(0x7f57a4147be0 0x7f57a40aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 -- 172.21.15.26:0/334629478 >> v1:172.21.15.120:6789/0 conn(0x7f57a40b5ef0 legacy=0x7f57a41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 -- 172.21.15.26:0/334629478 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a414b000 con 0x7f57a40a7fc0 2022-07-20T02:33:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a8d8c700 1 --2- 172.21.15.26:0/334629478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a40af330 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f579c0032f0 tx=0x7f579c009290).ready entity=mon.0 client_cookie=478bb2642aac0ce4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a2ffd700 1 -- 172.21.15.26:0/334629478 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f579c00b870 con 0x7f57a40a7fc0 2022-07-20T02:33:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a2ffd700 1 -- 172.21.15.26:0/334629478 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c00b9d0 con 0x7f57a40a7fc0 2022-07-20T02:33:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a2ffd700 1 -- 172.21.15.26:0/334629478 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f579c00bb40 con 0x7f57a40a7fc0 2022-07-20T02:33:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- 172.21.15.26:0/334629478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 msgr2=0x7f57a40af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 --2- 172.21.15.26:0/334629478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a40af330 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f579c0032f0 tx=0x7f579c009290).stop 2022-07-20T02:33:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- 172.21.15.26:0/334629478 shutdown_connections 2022-07-20T02:33:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 --2- 172.21.15.26:0/334629478 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a40af330 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 --2- 172.21.15.26:0/334629478 >> v2:172.21.15.120:3300/0 conn(0x7f57a4147be0 0x7f57a40aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- 172.21.15.26:0/334629478 >> 172.21.15.26:0/334629478 conn(0x7f57a4060310 msgr2=0x7f57a40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- 172.21.15.26:0/334629478 shutdown_connections 2022-07-20T02:33:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- 172.21.15.26:0/334629478 wait complete. 2022-07-20T02:33:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 Processor -- start 2022-07-20T02:33:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.827+0000 7f57a958d700 1 -- start start 2022-07-20T02:33:43.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 0x7f57a41568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 0x7f57a4147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a40a9190 con 0x7f57a40a7fc0 2022-07-20T02:33:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4156db0 con 0x7f57a4147be0 2022-07-20T02:33:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4157040 con 0x7f57a40b5ef0 2022-07-20T02:33:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 0x7f57a41568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41806/0 (socket says 172.21.15.26:41806) 2022-07-20T02:33:43.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 -- 172.21.15.26:0/3735972765 learned_addr learned my addr 172.21.15.26:0/3735972765 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a8d8c700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 0x7f57a4147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 -- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 msgr2=0x7f57a41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 0x7f57a41568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 -- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 msgr2=0x7f57a4147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 0x7f57a4147010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 -- 172.21.15.26:0/3735972765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c009530 con 0x7f57a40a7fc0 2022-07-20T02:33:43.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a8d8c700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 0x7f57a4147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:43.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a37fe700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 0x7f57a41568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a3fff700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f57a4060810 tx=0x7f5798007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5798007030 con 0x7f57a40a7fc0 2022-07-20T02:33:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798007190 con 0x7f57a40a7fc0 2022-07-20T02:33:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a41572e0 con 0x7f57a40a7fc0 2022-07-20T02:33:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5798006510 con 0x7f57a40a7fc0 2022-07-20T02:33:43.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.831+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a40aacc0 con 0x7f57a40a7fc0 2022-07-20T02:33:43.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f579801b070 con 0x7f57a40a7fc0 2022-07-20T02:33:43.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a40a9300 con 0x7f57a40a7fc0 2022-07-20T02:33:43.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a17fa700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 0x7f5784055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:43.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f579806b320 con 0x7f57a40a7fc0 2022-07-20T02:33:43.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a37fe700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 0x7f5784055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:43.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.835+0000 7f57a37fe700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 0x7f5784055fc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f579400b000 tx=0x7f5794006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:43.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.843+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57a40a9300 con 0x7f57a40a7fc0 2022-07-20T02:33:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.971+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f57a40a92d0 con 0x7f57a40a7fc0 2022-07-20T02:33:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.979+0000 7f57a17fa700 1 -- 172.21.15.26:0/3735972765 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5798042320 con 0x7f57a40a7fc0 2022-07-20T02:33:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 msgr2=0x7f5784055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 0x7f5784055fc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f579400b000 tx=0x7f5794006df0).stop 2022-07-20T02:33:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 msgr2=0x7f57a414e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f57a4060810 tx=0x7f5798007c00).stop 2022-07-20T02:33:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 shutdown_connections 2022-07-20T02:33:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5784053b00 0x7f5784055fc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57a4147be0 0x7f57a4147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57a40b5ef0 0x7f57a41568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 --2- 172.21.15.26:0/3735972765 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57a40a7fc0 0x7f57a414e1c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:43.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 >> 172.21.15.26:0/3735972765 conn(0x7f57a4060310 msgr2=0x7f57a40aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:43.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 shutdown_connections 2022-07-20T02:33:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:43.983+0000 7f57a958d700 1 -- 172.21.15.26:0/3735972765 wait complete. 2022-07-20T02:33:44.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-20T02:33:44.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 Processor -- start 2022-07-20T02:33:44.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 -- start start 2022-07-20T02:33:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 0x7f5814143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58141474e0 con 0x7f5814060f00 2022-07-20T02:33:44.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58140b4e40 con 0x7f581405cfd0 2022-07-20T02:33:44.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5819fcd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581405d3b0 con 0x7f5814147000 2022-07-20T02:33:44.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58197cc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5814060f00 0x7f581413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41098/0 (socket says 172.21.15.26:41098) 2022-07-20T02:33:44.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58197cc700 1 -- 172.21.15.26:0/4220475048 learned_addr learned my addr 172.21.15.26:0/4220475048 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:44.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5813fff700 1 --2- 172.21.15.26:0/4220475048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 0x7f5814143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528169285 0 0) 0x7f58141474e0 con 0x7f5814060f00 2022-07-20T02:33:44.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f4003260 con 0x7f5814060f00 2022-07-20T02:33:44.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481686021 0 0) 0x7f58140b4e40 con 0x7f581405cfd0 2022-07-20T02:33:44.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58141474e0 con 0x7f581405cfd0 2022-07-20T02:33:44.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5813fff700 1 -- 172.21.15.26:0/4220475048 >> v1:172.21.15.120:6789/0 conn(0x7f581405cfd0 legacy=0x7f58140b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5813fff700 1 -- 172.21.15.26:0/4220475048 >> v1:172.21.15.26:6789/0 conn(0x7f5814060f00 legacy=0x7f581413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5813fff700 1 -- 172.21.15.26:0/4220475048 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58141431e0 con 0x7f5814147000 2022-07-20T02:33:44.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f5813fff700 1 --2- 172.21.15.26:0/4220475048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 0x7f5814143bc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57fc002f70 tx=0x7f57fc006df0).ready entity=mon.2 client_cookie=57e44b433a82acc4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc00d5c0 con 0x7f5814147000 2022-07-20T02:33:44.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc000e90 con 0x7f5814147000 2022-07-20T02:33:44.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.139+0000 7f58137fe700 1 -- 172.21.15.26:0/4220475048 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc00d910 con 0x7f5814147000 2022-07-20T02:33:44.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- 172.21.15.26:0/4220475048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 msgr2=0x7f5814143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 --2- 172.21.15.26:0/4220475048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 0x7f5814143bc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57fc002f70 tx=0x7f57fc006df0).stop 2022-07-20T02:33:44.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- 172.21.15.26:0/4220475048 shutdown_connections 2022-07-20T02:33:44.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 --2- 172.21.15.26:0/4220475048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814147000 0x7f5814143bc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- 172.21.15.26:0/4220475048 >> 172.21.15.26:0/4220475048 conn(0x7f58140620c0 msgr2=0x7f581405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:44.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- 172.21.15.26:0/4220475048 shutdown_connections 2022-07-20T02:33:44.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- 172.21.15.26:0/4220475048 wait complete. 2022-07-20T02:33:44.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 Processor -- start 2022-07-20T02:33:44.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- start start 2022-07-20T02:33:44.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5814147000 0x7f5814145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5814146470 0x7f58141448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140ad670 con 0x7f5814147000 2022-07-20T02:33:44.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5814156260 con 0x7f5814146470 2022-07-20T02:33:44.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5819fcd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5814156500 con 0x7f5814060f00 2022-07-20T02:33:44.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5818fcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5818fcb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35104/0 (socket says 172.21.15.26:35104) 2022-07-20T02:33:44.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5818fcb700 1 -- 172.21.15.26:0/270708560 learned_addr learned my addr 172.21.15.26:0/270708560 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:44.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f5813fff700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5814147000 0x7f5814145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.143+0000 7f58197cc700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5814146470 0x7f58141448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 -- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5814146470 msgr2=0x7f58141448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5814146470 0x7f58141448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 -- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5814147000 msgr2=0x7f5814145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5814147000 0x7f5814145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 -- 172.21.15.26:0/270708560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc00b180 con 0x7f5814060f00 2022-07-20T02:33:44.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5818fcb700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f58140b91e0 tx=0x7f5804007ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f580400e4a0 con 0x7f5814060f00 2022-07-20T02:33:44.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580400e600 con 0x7f5814060f00 2022-07-20T02:33:44.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f580400e960 con 0x7f5814060f00 2022-07-20T02:33:44.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5814144460 con 0x7f5814060f00 2022-07-20T02:33:44.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581414a490 con 0x7f5814060f00 2022-07-20T02:33:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58141560a0 con 0x7f5814060f00 2022-07-20T02:33:44.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5804010070 con 0x7f5814060f00 2022-07-20T02:33:44.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 0x7f57f8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f580406b5b0 con 0x7f5814060f00 2022-07-20T02:33:44.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.147+0000 7f5813fff700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 0x7f57f8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.151+0000 7f5813fff700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 0x7f57f8055ea0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f57fc01a000 tx=0x7f57fc014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.159+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58040424c0 con 0x7f5814060f00 2022-07-20T02:33:44.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.327+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f58140ad7b0 con 0x7f5814060f00 2022-07-20T02:33:44.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.675+0000 7f5811ffb700 1 -- 172.21.15.26:0/270708560 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v376) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5804069e60 con 0x7f5814060f00 2022-07-20T02:33:44.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 msgr2=0x7f57f8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 0x7f57f8055ea0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f57fc01a000 tx=0x7f57fc014000).stop 2022-07-20T02:33:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 msgr2=0x7f58140b8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f58140b91e0 tx=0x7f5804007ee0).stop 2022-07-20T02:33:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 shutdown_connections 2022-07-20T02:33:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5814147000 0x7f5814145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f57f80539e0 0x7f57f8055ea0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5814146470 0x7f58141448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 --2- 172.21.15.26:0/270708560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5814060f00 0x7f58140b8750 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 >> 172.21.15.26:0/270708560 conn(0x7f581405cfd0 msgr2=0x7f58140b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:44.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 shutdown_connections 2022-07-20T02:33:44.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.679+0000 7f5819fcd700 1 -- 172.21.15.26:0/270708560 wait complete. 2022-07-20T02:33:44.704 INFO:tasks.cephfs_test_runner:test_filtered_df (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:33:44.705 INFO:tasks.cephfs_test_runner:Starting test: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:33:44.705 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-20T02:33:44.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 Processor -- start 2022-07-20T02:33:44.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- start start 2022-07-20T02:33:44.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f900af730 0x7f0f90142b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900adc90 con 0x7f0f90143020 2022-07-20T02:33:44.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f900628b0 con 0x7f0f900af730 2022-07-20T02:33:44.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900b22b0 con 0x7f0f900683d0 2022-07-20T02:33:44.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35112/0 (socket says 172.21.15.26:35112) 2022-07-20T02:33:44.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 -- 172.21.15.26:0/1292781775 learned_addr learned my addr 172.21.15.26:0/1292781775 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:44.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8f7fe700 1 --2- 172.21.15.26:0/1292781775 >> v2:172.21.15.120:3300/0 conn(0x7f0f900af730 0x7f0f90142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 -- 172.21.15.26:0/1292781775 >> v2:172.21.15.120:3300/0 conn(0x7f0f900af730 msgr2=0x7f0f90142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 --2- 172.21.15.26:0/1292781775 >> v2:172.21.15.120:3300/0 conn(0x7f0f900af730 0x7f0f90142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 -- 172.21.15.26:0/1292781775 >> v1:172.21.15.120:6789/0 conn(0x7f0f90143020 legacy=0x7f0f90149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 -- 172.21.15.26:0/1292781775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900b2c00 con 0x7f0f900683d0 2022-07-20T02:33:44.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8effd700 1 -- 172.21.15.26:0/1292781775 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121231271 0 0) 0x7f0f900adc90 con 0x7f0f90143020 2022-07-20T02:33:44.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8ffff700 1 --2- 172.21.15.26:0/1292781775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f0f800032f0 tx=0x7f0f80009290).ready entity=mon.2 client_cookie=f2a80c4b22e1b9b8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8effd700 1 -- 172.21.15.26:0/1292781775 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f8000b8a0 con 0x7f0f900683d0 2022-07-20T02:33:44.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8effd700 1 -- 172.21.15.26:0/1292781775 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000ba00 con 0x7f0f900683d0 2022-07-20T02:33:44.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f8effd700 1 -- 172.21.15.26:0/1292781775 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f8000bb70 con 0x7f0f900683d0 2022-07-20T02:33:44.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- 172.21.15.26:0/1292781775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 msgr2=0x7f0f900b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 --2- 172.21.15.26:0/1292781775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f0f800032f0 tx=0x7f0f80009290).stop 2022-07-20T02:33:44.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.819+0000 7f0f9543e700 1 -- 172.21.15.26:0/1292781775 shutdown_connections 2022-07-20T02:33:44.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 --2- 172.21.15.26:0/1292781775 >> v2:172.21.15.120:3300/0 conn(0x7f0f900af730 0x7f0f90142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 --2- 172.21.15.26:0/1292781775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f900b2710 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- 172.21.15.26:0/1292781775 >> 172.21.15.26:0/1292781775 conn(0x7f0f9005f840 msgr2=0x7f0f9005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:44.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- 172.21.15.26:0/1292781775 shutdown_connections 2022-07-20T02:33:44.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- 172.21.15.26:0/1292781775 wait complete. 2022-07-20T02:33:44.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 Processor -- start 2022-07-20T02:33:44.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- start start 2022-07-20T02:33:44.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f90153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f90143020 0x7f0f900b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9013a8b0 con 0x7f0f900af730 2022-07-20T02:33:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90152640 con 0x7f0f90143020 2022-07-20T02:33:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f901528e0 con 0x7f0f900683d0 2022-07-20T02:33:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f90153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f94c3d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f90143020 0x7f0f900b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41838/0 (socket says 172.21.15.26:41838) 2022-07-20T02:33:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 -- 172.21.15.26:0/826371495 learned_addr learned my addr 172.21.15.26:0/826371495 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 -- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 msgr2=0x7f0f90153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f90153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 -- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f90143020 msgr2=0x7f0f900b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f90143020 0x7f0f900b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 -- 172.21.15.26:0/826371495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80009530 con 0x7f0f900af730 2022-07-20T02:33:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8f7fe700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f0f7800c000 tx=0x7f0f78006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f78010780 con 0x7f0f900af730 2022-07-20T02:33:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f780108e0 con 0x7f0f900af730 2022-07-20T02:33:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f90152b80 con 0x7f0f900af730 2022-07-20T02:33:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f78010c40 con 0x7f0f900af730 2022-07-20T02:33:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.823+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f900b3c20 con 0x7f0f900af730 2022-07-20T02:33:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f900b3fa0 con 0x7f0f900af730 2022-07-20T02:33:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0f7801c460 con 0x7f0f900af730 2022-07-20T02:33:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f8d7fa700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 0x7f0f68055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0f7806c150 con 0x7f0f900af730 2022-07-20T02:33:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f8ffff700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 0x7f0f68055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.827+0000 7f0f8ffff700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 0x7f0f68055fc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f0f8001b000 tx=0x7f0f80016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:44.835+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0f900b3fa0 con 0x7f0f900af730 2022-07-20T02:33:45.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.003+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f0f900b3fa0 con 0x7f0f900af730 2022-07-20T02:33:45.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.675+0000 7f0f8d7fa700 1 -- 172.21.15.26:0/826371495 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v377) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0f900b3fa0 con 0x7f0f900af730 2022-07-20T02:33:45.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.679+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 msgr2=0x7f0f68055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.679+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 0x7f0f68055fc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f0f8001b000 tx=0x7f0f80016000).stop 2022-07-20T02:33:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.679+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 msgr2=0x7f0f900b5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.679+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f0f7800c000 tx=0x7f0f78006df0).stop 2022-07-20T02:33:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 shutdown_connections 2022-07-20T02:33:45.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0f900af730 0x7f0f900b5150 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0f68053b00 0x7f0f68055fc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f90143020 0x7f0f900b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 --2- 172.21.15.26:0/826371495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f900683d0 0x7f0f90153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 >> 172.21.15.26:0/826371495 conn(0x7f0f9005f840 msgr2=0x7f0f900a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:45.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 shutdown_connections 2022-07-20T02:33:45.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.683+0000 7f0f9543e700 1 -- 172.21.15.26:0/826371495 wait complete. 2022-07-20T02:33:45.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:33:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.815+0000 7fbef9074700 1 Processor -- start 2022-07-20T02:33:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.815+0000 7fbef9074700 1 -- start start 2022-07-20T02:33:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 0x7fbef4147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbef40b22d0 0x7fbef4143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef4061c80 con 0x7fbef4143d60 2022-07-20T02:33:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef41441f0 con 0x7fbef40b22d0 2022-07-20T02:33:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40b2810 con 0x7fbef40618a0 2022-07-20T02:33:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbef40b22d0 0x7fbef4143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbef40b22d0 0x7fbef4143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41848/0 (socket says 172.21.15.26:41848) 2022-07-20T02:33:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 -- 172.21.15.26:0/2700821027 learned_addr learned my addr 172.21.15.26:0/2700821027 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef3fff700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 0x7fbef4147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 -- 172.21.15.26:0/2700821027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 msgr2=0x7fbef4147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 0x7fbef4147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 -- 172.21.15.26:0/2700821027 >> v1:172.21.15.120:6789/0 conn(0x7fbef4143d60 legacy=0x7fbef40b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 -- 172.21.15.26:0/2700821027 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40b2ab0 con 0x7fbef40b22d0 2022-07-20T02:33:45.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef3fff700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 0x7fbef4147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:45.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef37fe700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40b22d0 0x7fbef4143870 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7fbee40032f0 tx=0x7fbee4009290).ready entity=mon.0 client_cookie=672f36b2fca6632 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:45.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef2ffd700 1 -- 172.21.15.26:0/2700821027 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee400b8a0 con 0x7fbef40b22d0 2022-07-20T02:33:45.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef2ffd700 1 -- 172.21.15.26:0/2700821027 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee400ba00 con 0x7fbef40b22d0 2022-07-20T02:33:45.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef2ffd700 1 -- 172.21.15.26:0/2700821027 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee400bb70 con 0x7fbef40b22d0 2022-07-20T02:33:45.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- 172.21.15.26:0/2700821027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40b22d0 msgr2=0x7fbef4143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40b22d0 0x7fbef4143870 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7fbee40032f0 tx=0x7fbee4009290).stop 2022-07-20T02:33:45.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- 172.21.15.26:0/2700821027 shutdown_connections 2022-07-20T02:33:45.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40b22d0 0x7fbef4143870 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 --2- 172.21.15.26:0/2700821027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40618a0 0x7fbef4147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- 172.21.15.26:0/2700821027 >> 172.21.15.26:0/2700821027 conn(0x7fbef4066f70 msgr2=0x7fbef4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- 172.21.15.26:0/2700821027 shutdown_connections 2022-07-20T02:33:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.819+0000 7fbef9074700 1 -- 172.21.15.26:0/2700821027 wait complete. 2022-07-20T02:33:45.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 Processor -- start 2022-07-20T02:33:45.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 -- start start 2022-07-20T02:33:45.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40b22d0 0x7fbef40afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbef4143d60 0x7fbef40b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4144ef0 con 0x7fbef40618a0 2022-07-20T02:33:45.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40b0380 con 0x7fbef4143d60 2022-07-20T02:33:45.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef9074700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40b3490 con 0x7fbef40b22d0 2022-07-20T02:33:45.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41852/0 (socket says 172.21.15.26:41852) 2022-07-20T02:33:45.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef3fff700 1 -- 172.21.15.26:0/3787782345 learned_addr learned my addr 172.21.15.26:0/3787782345 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:45.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef37fe700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40b22d0 0x7fbef40afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.823+0000 7fbef8873700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbef4143d60 0x7fbef40b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 -- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40b22d0 msgr2=0x7fbef40afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40b22d0 0x7fbef40afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 -- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbef4143d60 msgr2=0x7fbef40b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbef4143d60 0x7fbef40b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 -- 172.21.15.26:0/3787782345 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4009530 con 0x7fbef40618a0 2022-07-20T02:33:45.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef3fff700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fbedc006940 tx=0x7fbedc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:45.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbedc011780 con 0x7fbef40618a0 2022-07-20T02:33:45.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0118e0 con 0x7fbef40618a0 2022-07-20T02:33:45.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef40b3730 con 0x7fbef40618a0 2022-07-20T02:33:45.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbedc011c40 con 0x7fbef40618a0 2022-07-20T02:33:45.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef40b46c0 con 0x7fbef40618a0 2022-07-20T02:33:45.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef40574e0 con 0x7fbef40618a0 2022-07-20T02:33:45.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.827+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbedc012420 con 0x7fbef40618a0 2022-07-20T02:33:45.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.831+0000 7fbef17fa700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 0x7fbed8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:45.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.831+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fbedc06b9d0 con 0x7fbef40618a0 2022-07-20T02:33:45.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.831+0000 7fbef37fe700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 0x7fbed8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:45.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.831+0000 7fbef37fe700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 0x7fbed8055f30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fbee401b000 tx=0x7fbee4016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:45.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.835+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbedc042a10 con 0x7fbef40618a0 2022-07-20T02:33:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.963+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbef40b3bd0 con 0x7fbef40618a0 2022-07-20T02:33:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.971+0000 7fbef17fa700 1 -- 172.21.15.26:0/3787782345 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbedc046030 con 0x7fbef40618a0 2022-07-20T02:33:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 msgr2=0x7fbed8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 0x7fbed8055f30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fbee401b000 tx=0x7fbee4016000).stop 2022-07-20T02:33:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 msgr2=0x7fbef40af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fbedc006940 tx=0x7fbedc008040).stop 2022-07-20T02:33:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 shutdown_connections 2022-07-20T02:33:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbed8053a70 0x7fbed8055f30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbef4143d60 0x7fbef40b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbef40b22d0 0x7fbef40afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 --2- 172.21.15.26:0/3787782345 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbef40618a0 0x7fbef40af490 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:45.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 >> 172.21.15.26:0/3787782345 conn(0x7fbef4066f70 msgr2=0x7fbef405cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:45.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 shutdown_connections 2022-07-20T02:33:45.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:45.975+0000 7fbef9074700 1 -- 172.21.15.26:0/3787782345 wait complete. 2022-07-20T02:33:45.990 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:33:45.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:46.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:46.051 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:46.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:33:46.117 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:33:46.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:46.119 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:46.120 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:33:46.176 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:33:46.178 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:46.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:46.180 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:46.181 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:33:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:33:46.213 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:33:46.214 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:33:46.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:33:46.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 Processor -- start 2022-07-20T02:33:46.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 -- start start 2022-07-20T02:33:46.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 0x7f4008143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40081474e0 con 0x7f400805cfd0 2022-07-20T02:33:46.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080b4e40 con 0x7f4008147000 2022-07-20T02:33:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400ed11700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400805d3b0 con 0x7f4008060f00 2022-07-20T02:33:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35146/0 (socket says 172.21.15.26:35146) 2022-07-20T02:33:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 -- 172.21.15.26:0/2045189453 learned_addr learned my addr 172.21.15.26:0/2045189453 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400d50e700 1 --2- 172.21.15.26:0/2045189453 >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 0x7f4008143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 -- 172.21.15.26:0/2045189453 >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 msgr2=0x7f4008143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 --2- 172.21.15.26:0/2045189453 >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 0x7f4008143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 -- 172.21.15.26:0/2045189453 >> v1:172.21.15.120:6789/0 conn(0x7f400805cfd0 legacy=0x7f40080b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 -- 172.21.15.26:0/2045189453 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40081431e0 con 0x7f4008060f00 2022-07-20T02:33:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400d50e700 1 --2- 172.21.15.26:0/2045189453 >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 0x7f4008143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:46.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400e510700 1 --2- 172.21.15.26:0/2045189453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f40000032f0 tx=0x7f4000009290).ready entity=mon.2 client_cookie=d07d50136436867c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:46.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400cd0d700 1 -- 172.21.15.26:0/2045189453 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400000b8a0 con 0x7f4008060f00 2022-07-20T02:33:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400cd0d700 1 -- 172.21.15.26:0/2045189453 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400000ba00 con 0x7f4008060f00 2022-07-20T02:33:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.339+0000 7f400cd0d700 1 -- 172.21.15.26:0/2045189453 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f400000bb70 con 0x7f4008060f00 2022-07-20T02:33:46.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- 172.21.15.26:0/2045189453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 msgr2=0x7f400813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 --2- 172.21.15.26:0/2045189453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f40000032f0 tx=0x7f4000009290).stop 2022-07-20T02:33:46.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- 172.21.15.26:0/2045189453 shutdown_connections 2022-07-20T02:33:46.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 --2- 172.21.15.26:0/2045189453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008060f00 0x7f400813c9b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:46.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 --2- 172.21.15.26:0/2045189453 >> v2:172.21.15.120:3300/0 conn(0x7f4008147000 0x7f4008143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:46.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- 172.21.15.26:0/2045189453 >> 172.21.15.26:0/2045189453 conn(0x7f40080620c0 msgr2=0x7f400805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:46.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- 172.21.15.26:0/2045189453 shutdown_connections 2022-07-20T02:33:46.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- 172.21.15.26:0/2045189453 wait complete. 2022-07-20T02:33:46.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 Processor -- start 2022-07-20T02:33:46.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.343+0000 7f400ed11700 1 -- start start 2022-07-20T02:33:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008147000 0x7f400813ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 0x7f400813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400813d3e0 con 0x7f4008060f00 2022-07-20T02:33:46.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4008153430 con 0x7f40080af380 2022-07-20T02:33:46.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4008152ef0 con 0x7f4008147000 2022-07-20T02:33:46.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41864/0 (socket says 172.21.15.26:41864) 2022-07-20T02:33:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 -- 172.21.15.26:0/2780371797 learned_addr learned my addr 172.21.15.26:0/2780371797 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400e510700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 0x7f400813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400d50e700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008147000 0x7f400813ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 -- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008147000 msgr2=0x7f400813ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008147000 0x7f400813ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 -- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 msgr2=0x7f400813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 0x7f400813cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:46.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 -- 172.21.15.26:0/2780371797 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff800b180 con 0x7f4008060f00 2022-07-20T02:33:46.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400e510700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 0x7f400813cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:46.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400dd0f700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7f4008068660 tx=0x7f4004007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:46.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4004012400 con 0x7f4008060f00 2022-07-20T02:33:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4004012560 con 0x7f4008060f00 2022-07-20T02:33:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4000009530 con 0x7f4008060f00 2022-07-20T02:33:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40040128c0 con 0x7f4008060f00 2022-07-20T02:33:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.347+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40080ad7a0 con 0x7f4008060f00 2022-07-20T02:33:46.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4008143480 con 0x7f4008060f00 2022-07-20T02:33:46.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4004019030 con 0x7f4008060f00 2022-07-20T02:33:46.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f3ff6ffd700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 0x7f3fe8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:46.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f400406afa0 con 0x7f4008060f00 2022-07-20T02:33:46.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f400d50e700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 0x7f3fe8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:46.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.351+0000 7f400d50e700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 0x7f3fe8055f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f3ff800b000 tx=0x7f3ff8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:46.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.359+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4004042040 con 0x7f4008060f00 2022-07-20T02:33:46.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.491+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f40080574e0 con 0x7f4008060f00 2022-07-20T02:33:46.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.491+0000 7f3ff6ffd700 1 -- 172.21.15.26:0/2780371797 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v112) v1 ==== 81+0+29277 (secure 0 0 0) 0x7f4004045660 con 0x7f4008060f00 2022-07-20T02:33:46.499 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:46.500 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:46.500 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 112, 2022-07-20T02:33:46.501 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:33:46.501 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:33:46.506 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:33:43.657842+0000", 2022-07-20T02:33:46.506 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:46.506 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:33:46.507 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:33:46.507 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:33:46.507 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:33:46.507 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:33:46.508 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:33:46.512 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:33:46.512 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:33:46.515 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:46.517 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:33:46.518 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:33:46.518 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:33:46.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:33:46.518 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:33:46.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 20, 2022-07-20T02:33:46.519 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:33:46.520 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:33:46.520 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:33:46.520 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:33:46.520 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:33:46.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:33:46.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:33:46.521 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:33:46.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:46.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:46.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:46.523 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:46.524 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:46.524 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:46.525 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:46.525 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:46.526 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:33:46.527 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:33:46.527 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:33:46.527 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:33:46.528 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:33:46.528 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:46.528 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:46.529 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:46.529 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:46.529 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:46.529 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:46.530 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:46.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.530 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:33:46.531 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:46.531 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:46.531 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:46.531 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:46.531 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:46.532 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:46.532 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:46.532 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:46.532 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:46.533 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:46.533 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:46.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:46.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:46.534 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:46.534 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:46.545 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:46.545 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:46.545 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:46.546 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:46.546 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:46.546 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:46.546 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:46.547 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:46.547 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:46.547 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:46.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:46.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.549 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:46.549 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:46.549 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:46.550 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:46.550 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:46.550 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:46.551 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:46.551 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:46.551 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:46.552 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:46.552 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:46.552 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:33:46.553 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:33:46.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.560 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:46.560 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:33:46.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.563 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:33:46.563 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:33:46.563 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:33:46.564 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:33:46.564 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:33:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:33:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:33:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:33:46.578 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:33:46.579 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:33:46.581 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:33:46.581 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:33:46.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:33:46.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:33:46.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:33:46.583 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:33:46.583 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:33:46.584 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:33:46.584 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:33:46.585 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:33:46.585 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:33:46.586 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:33:46.586 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:33:46.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.587 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:33:46.587 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:33:46.588 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:33:46.588 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:33:46.588 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:33:46.589 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:33:46.589 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:33:46.589 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:33:46.590 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:33:46.591 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:33:46.591 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:33:46.591 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:33:46.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:33:46.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:33:46.592 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:33:46.592 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:33:46.593 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:33:46.593 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:33:46.593 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:33:46.593 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:33:46.593 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:33:46.594 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:33:46.594 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:33:46.594 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:33:46.594 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:33:46.595 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:33:46.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:33:46.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.595 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:33:46.596 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:33:46.596 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:33:46.596 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:33:46.596 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:33:46.596 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:33:46.597 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:33:46.597 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:33:46.597 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:33:46.597 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:33:46.598 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:33:46.598 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:33:46.598 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:33:46.599 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:33:46.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:46.600 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:33:46.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.608 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:46.608 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:33:46.608 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.609 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.609 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.609 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.610 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.610 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.611 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:46.611 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.612 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:33:46.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:33:46.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.616 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:33:46.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:33:46.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.623 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:33:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:33:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.626 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:33:46.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:33:46.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:33:46.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:33:46.630 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:33:46.630 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:33:46.630 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:33:46.631 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.631 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.631 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.633 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:46.633 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:33:46.633 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.633 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.634 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.634 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.634 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.635 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:46.635 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.635 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.636 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:33:46.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:33:46.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.639 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:33:46.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:33:46.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.643 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:33:46.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:33:46.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:33:46.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:33:46.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:33:46.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:33:46.651 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:33:46.651 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:33:46.652 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:33:46.652 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.652 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.652 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.654 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:46.654 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:33:46.654 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.655 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.655 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.655 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.655 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.656 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.656 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.656 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:46.656 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.657 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:33:46.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:33:46.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.662 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:33:46.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:33:46.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.666 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:33:46.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:33:46.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.684 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:33:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:33:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:33:46.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:33:46.709 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:33:46.710 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:33:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:33:46.711 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.736 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:46.736 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:33:46.737 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.737 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.749 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.749 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.750 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.750 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.751 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.751 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:46.752 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.752 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:33:46.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:33:46.762 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.763 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:33:46.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:33:46.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.767 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:33:46.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:33:46.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:33:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:33:46.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:33:46.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:33:46.775 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:33:46.775 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:33:46.775 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:33:46.776 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.776 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.776 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.778 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:46.778 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:33:46.778 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.778 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.779 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.779 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.779 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.779 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.789 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.790 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:46.790 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.790 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:33:46.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:33:46.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:33:46.804 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:33:46.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.806 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:33:46.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:33:46.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.810 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:33:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:33:46.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:33:46.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:33:46.814 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:33:46.814 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:33:46.815 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:33:46.815 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.815 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.815 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.816 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:46.816 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:33:46.817 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.817 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.820 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.820 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.822 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.822 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.823 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 105, 2022-07-20T02:33:46.823 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.823 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:33:46.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:33:46.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.827 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:33:46.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:33:46.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.830 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:33:46.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:33:46.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.834 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:33:46.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:33:46.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:33:46.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:33:46.837 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:33:46.838 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:33:46.838 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:33:46.838 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.838 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.839 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.840 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:46.840 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:33:46.840 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.840 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.840 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.841 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.841 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.842 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:46.842 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.842 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:33:46.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:33:46.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:33:46.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:33:46.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.849 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:33:46.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:33:46.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.852 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:33:46.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:33:46.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:33:46.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:33:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:33:46.856 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:33:46.857 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:33:46.857 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.857 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.858 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.860 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:46.860 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:33:46.860 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:33:46.861 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:33:46.862 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:33:46.863 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:33:46.864 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:33:46.864 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:33:46.864 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:33:46.865 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 107, 2022-07-20T02:33:46.866 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:33:46.866 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:33:46.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:33:46.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:33:46.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:33:46.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.872 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.874 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:33:46.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:33:46.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:33:46.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:33:46.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:33:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:33:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:33:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:33:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:33:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:33:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:33:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:33:46.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:33:46.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:33:46.895 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:33:46.895 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:33:46.896 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:33:46.896 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:33:46.897 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:33:46.897 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:33:46.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:46.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.898 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:46.899 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:33:46.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:33:46.900 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.901 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.901 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.902 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.903 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:33:46.903 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:33:46.905 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.905 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.907 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.908 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.908 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.909 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:33:46.909 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:33:46.911 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.912 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.912 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.913 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.913 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.914 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:33:46.914 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.923 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:33:46.931 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.932 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.932 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.933 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.933 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:33:46.934 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:33:46.936 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.936 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.937 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.937 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.938 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:33:46.938 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.940 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:33:46.940 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.941 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.941 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.942 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.942 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.942 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:33:46.943 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:33:46.944 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.945 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.945 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.946 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.946 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.947 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:33:46.947 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:46.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:33:46.949 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:33:46.949 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:33:46.958 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:33:46.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:33:46.959 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:33:46.961 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:33:46.962 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:33:46.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.963 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:33:46.964 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:33:46.964 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:33:46.965 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:33:46.965 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:33:46.966 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:33:46.966 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/3259008170": "2022-07-21T02:33:39.824985+0000", 2022-07-20T02:33:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/3259008170": "2022-07-21T02:33:39.824985+0000" 2022-07-20T02:33:46.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:33:46.968 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:33:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:33:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:46.969 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:33:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:33:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:46.971 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:33:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:46.972 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:33:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:33:46.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.974 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:33:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:33:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:33:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:33:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:33:46.976 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:33:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:33:46.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:46.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:33:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:33:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:33:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:33:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:33:46.998 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:47.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 msgr2=0x7f3fe8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 0x7f3fe8055f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f3ff800b000 tx=0x7f3ff8006df0).stop 2022-07-20T02:33:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 msgr2=0x7f4008146010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7f4008068660 tx=0x7f4004007c50).stop 2022-07-20T02:33:47.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 shutdown_connections 2022-07-20T02:33:47.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3fe8053a70 0x7f3fe8055f30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40080af380 0x7f400813cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4008147000 0x7f400813ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 --2- 172.21.15.26:0/2780371797 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4008060f00 0x7f4008146010 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 >> 172.21.15.26:0/2780371797 conn(0x7f400805cfd0 msgr2=0x7f400813b080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 shutdown_connections 2022-07-20T02:33:47.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:46.495+0000 7f400ed11700 1 -- 172.21.15.26:0/2780371797 wait complete. 2022-07-20T02:33:47.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:47.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 Processor -- start 2022-07-20T02:33:47.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 -- start start 2022-07-20T02:33:47.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10280b3640 con 0x7f10280b3260 2022-07-20T02:33:47.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1028095300 con 0x7f1028060c00 2022-07-20T02:33:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.115+0000 7f102e703700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280955a0 con 0x7f1028097c50 2022-07-20T02:33:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102d701700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f10280b3260 0x7f10280604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41154/0 (socket says 172.21.15.26:41154) 2022-07-20T02:33:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102d701700 1 -- 172.21.15.26:0/2331417176 learned_addr learned my addr 172.21.15.26:0/2331417176 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102df02700 1 --2- 172.21.15.26:0/2331417176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95857319 0 0) 0x7f10280b3640 con 0x7f10280b3260 2022-07-20T02:33:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003260 con 0x7f10280b3260 2022-07-20T02:33:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487399165 0 0) 0x7f1028095300 con 0x7f1028060c00 2022-07-20T02:33:47.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10280b3640 con 0x7f1028060c00 2022-07-20T02:33:47.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102df02700 1 -- 172.21.15.26:0/2331417176 >> v1:172.21.15.120:6789/0 conn(0x7f1028060c00 legacy=0x7f10281445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102df02700 1 -- 172.21.15.26:0/2331417176 >> v1:172.21.15.26:6789/0 conn(0x7f10280b3260 legacy=0x7f10280604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102df02700 1 -- 172.21.15.26:0/2331417176 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028095840 con 0x7f1028097c50 2022-07-20T02:33:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102df02700 1 --2- 172.21.15.26:0/2331417176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028144cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f10200032f0 tx=0x7f10200091d0).ready entity=mon.2 client_cookie=dce75ed2406a936e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f102000b640 con 0x7f1028097c50 2022-07-20T02:33:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102000b7a0 con 0x7f1028097c50 2022-07-20T02:33:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f101ffff700 1 -- 172.21.15.26:0/2331417176 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f102000b910 con 0x7f1028097c50 2022-07-20T02:33:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 -- 172.21.15.26:0/2331417176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 msgr2=0x7f1028144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 --2- 172.21.15.26:0/2331417176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028144cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f10200032f0 tx=0x7f10200091d0).stop 2022-07-20T02:33:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 -- 172.21.15.26:0/2331417176 shutdown_connections 2022-07-20T02:33:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 --2- 172.21.15.26:0/2331417176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028144cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 -- 172.21.15.26:0/2331417176 >> 172.21.15.26:0/2331417176 conn(0x7f10280b67f0 msgr2=0x7f10280b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 -- 172.21.15.26:0/2331417176 shutdown_connections 2022-07-20T02:33:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.119+0000 7f102e703700 1 -- 172.21.15.26:0/2331417176 wait complete. 2022-07-20T02:33:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 Processor -- start 2022-07-20T02:33:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- start start 2022-07-20T02:33:47.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1028060c00 0x7f1028156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028094640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028155be0 con 0x7f10280b3260 2022-07-20T02:33:47.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028156a40 con 0x7f1028060c00 2022-07-20T02:33:47.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280976c0 con 0x7f1028097c50 2022-07-20T02:33:47.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102d701700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1028060c00 0x7f1028156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41882/0 (socket says 172.21.15.26:41882) 2022-07-20T02:33:47.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 -- 172.21.15.26:0/2905178880 learned_addr learned my addr 172.21.15.26:0/2905178880 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:47.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102cf00700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028094640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 -- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 msgr2=0x7f1028094640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028094640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 -- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1028060c00 msgr2=0x7f1028156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1028060c00 0x7f1028156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 -- 172.21.15.26:0/2905178880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020009470 con 0x7f10280b3260 2022-07-20T02:33:47.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102df02700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f10200032f0 tx=0x7f1020016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f102000b910 con 0x7f10280b3260 2022-07-20T02:33:47.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102001f3f0 con 0x7f10280b3260 2022-07-20T02:33:47.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f102000b640 con 0x7f10280b3260 2022-07-20T02:33:47.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1028155e10 con 0x7f10280b3260 2022-07-20T02:33:47.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.123+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1028094b30 con 0x7f10280b3260 2022-07-20T02:33:47.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10280993f0 con 0x7f10280b3260 2022-07-20T02:33:47.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1020026030 con 0x7f10280b3260 2022-07-20T02:33:47.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f101e7fc700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 0x7f1010055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f102006c810 con 0x7f10280b3260 2022-07-20T02:33:47.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f102d701700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 0x7f1010055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.127+0000 7f102d701700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 0x7f1010055fc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f101800d000 tx=0x7f1018009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.135+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10200438b0 con 0x7f10280b3260 2022-07-20T02:33:47.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10280574e0 con 0x7f10280b3260 2022-07-20T02:33:47.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f101e7fc700 1 -- 172.21.15.26:0/2905178880 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f1020024630 con 0x7f10280b3260 2022-07-20T02:33:47.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:47.303 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/2232384404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2232384404},{"type":"v1","addr":"172.21.15.26:6837","nonce":2232384404}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":58}],"filesystems":[]} 2022-07-20T02:33:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 msgr2=0x7f1010055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 0x7f1010055fc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f101800d000 tx=0x7f1018009040).stop 2022-07-20T02:33:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 msgr2=0x7f1028152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f10200032f0 tx=0x7f1020016000).stop 2022-07-20T02:33:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 shutdown_connections 2022-07-20T02:33:47.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f10280b3260 0x7f1028152570 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1010053b00 0x7f1010055fc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1028097c50 0x7f1028094640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 --2- 172.21.15.26:0/2905178880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1028060c00 0x7f1028156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 >> 172.21.15.26:0/2905178880 conn(0x7f10280b67f0 msgr2=0x7f10281408b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 shutdown_connections 2022-07-20T02:33:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.299+0000 7f102e703700 1 -- 172.21.15.26:0/2905178880 wait complete. 2022-07-20T02:33:47.313 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 58 2022-07-20T02:33:47.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:47.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 Processor -- start 2022-07-20T02:33:47.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 -- start start 2022-07-20T02:33:47.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34ac0b3380 0x7f34ac14b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac064880 con 0x7f34ac064450 2022-07-20T02:33:47.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0b85f0 con 0x7f34ac0b3380 2022-07-20T02:33:47.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.415+0000 7f34abfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0b80b0 con 0x7f34ac0670c0 2022-07-20T02:33:47.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34ac0b3380 0x7f34ac14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aaffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aaffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33346/0 (socket says 172.21.15.26:33346) 2022-07-20T02:33:47.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34ac0b3380 0x7f34ac14b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41890/0 (socket says 172.21.15.26:41890) 2022-07-20T02:33:47.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34a27fc700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f34ac064450 0x7f34ac064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:55380/0 (socket says 172.21.15.26:55380) 2022-07-20T02:33:47.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 -- 172.21.15.26:0/1848720117 learned_addr learned my addr 172.21.15.26:0/1848720117 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 -- 172.21.15.26:0/1848720117 >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 msgr2=0x7f34ac0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:47.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 --2- 172.21.15.26:0/1848720117 >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 -- 172.21.15.26:0/1848720117 >> v1:172.21.15.120:6789/0 conn(0x7f34ac064450 legacy=0x7f34ac064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aa7fc700 1 -- 172.21.15.26:0/1848720117 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823751746 0 0) 0x7f34ac064880 con 0x7f34ac064450 2022-07-20T02:33:47.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 -- 172.21.15.26:0/1848720117 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac0b8350 con 0x7f34ac0b3380 2022-07-20T02:33:47.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aaffd700 1 --2- 172.21.15.26:0/1848720117 >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:47.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34ab7fe700 1 --2- 172.21.15.26:0/1848720117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac0b3380 0x7f34ac14b840 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f34a40032f0 tx=0x7f34a4009290).ready entity=mon.0 client_cookie=73781839551e2bcf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aa7fc700 1 -- 172.21.15.26:0/1848720117 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a400b870 con 0x7f34ac0b3380 2022-07-20T02:33:47.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aa7fc700 1 -- 172.21.15.26:0/1848720117 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a400b9d0 con 0x7f34ac0b3380 2022-07-20T02:33:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34aa7fc700 1 -- 172.21.15.26:0/1848720117 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a400bb40 con 0x7f34ac0b3380 2022-07-20T02:33:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- 172.21.15.26:0/1848720117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac0b3380 msgr2=0x7f34ac14b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 --2- 172.21.15.26:0/1848720117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac0b3380 0x7f34ac14b840 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f34a40032f0 tx=0x7f34a4009290).stop 2022-07-20T02:33:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- 172.21.15.26:0/1848720117 shutdown_connections 2022-07-20T02:33:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 --2- 172.21.15.26:0/1848720117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac0b3380 0x7f34ac14b840 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 --2- 172.21.15.26:0/1848720117 >> v2:172.21.15.120:3300/0 conn(0x7f34ac0670c0 0x7f34ac0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- 172.21.15.26:0/1848720117 >> 172.21.15.26:0/1848720117 conn(0x7f34ac05f840 msgr2=0x7f34ac05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- 172.21.15.26:0/1848720117 shutdown_connections 2022-07-20T02:33:47.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- 172.21.15.26:0/1848720117 wait complete. 2022-07-20T02:33:47.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 Processor -- start 2022-07-20T02:33:47.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.419+0000 7f34abfff700 1 -- start start 2022-07-20T02:33:47.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34ac0670c0 0x7f34ac142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ac150360 0x7f34ac153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0aedd0 con 0x7f34ac142f30 2022-07-20T02:33:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0ae380 con 0x7f34ac150360 2022-07-20T02:33:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0aeb30 con 0x7f34ac0670c0 2022-07-20T02:33:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41894/0 (socket says 172.21.15.26:41894) 2022-07-20T02:33:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 -- 172.21.15.26:0/1617101567 learned_addr learned my addr 172.21.15.26:0/1617101567 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:47.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34aaffd700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34ac0670c0 0x7f34ac142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34ab7fe700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ac150360 0x7f34ac153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 -- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34ac0670c0 msgr2=0x7f34ac142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34ac0670c0 0x7f34ac142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 -- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ac150360 msgr2=0x7f34ac153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ac150360 0x7f34ac153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 -- 172.21.15.26:0/1617101567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c00b180 con 0x7f34ac142f30 2022-07-20T02:33:47.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a27fc700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f3494000fe0 tx=0x7f3494007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34940072e0 con 0x7f34ac142f30 2022-07-20T02:33:47.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3494005550 con 0x7f34ac142f30 2022-07-20T02:33:47.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4009530 con 0x7f34ac142f30 2022-07-20T02:33:47.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34940058b0 con 0x7f34ac142f30 2022-07-20T02:33:47.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac0b0860 con 0x7f34ac142f30 2022-07-20T02:33:47.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.423+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac0af1e0 con 0x7f34ac142f30 2022-07-20T02:33:47.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.427+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f349401a060 con 0x7f34ac142f30 2022-07-20T02:33:47.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.427+0000 7f34a8ff9700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 0x7f3490055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.427+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f349406b240 con 0x7f34ac142f30 2022-07-20T02:33:47.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.427+0000 7f34aaffd700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 0x7f3490055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.427+0000 7f34aaffd700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 0x7f3490055ed0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f349c00b000 tx=0x7f349c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.435+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34940422e0 con 0x7f34ac142f30 2022-07-20T02:33:47.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34ac0b09f0 con 0x7f34ac142f30 2022-07-20T02:33:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34a8ff9700 1 -- 172.21.15.26:0/1617101567 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f3494018020 con 0x7f34ac142f30 2022-07-20T02:33:47.603 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:47.603 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/947865113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":947865113},{"type":"v1","addr":"172.21.15.120:6833","nonce":947865113}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51},{"gid":7392,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/1628051411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1628051411},{"type":"v1","addr":"172.21.15.26:6835","nonce":1628051411}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/2232384404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":2232384404},{"type":"v1","addr":"172.21.15.26:6837","nonce":2232384404}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":58}],"filesystems":[]} 2022-07-20T02:33:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 msgr2=0x7f3490055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 0x7f3490055ed0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f349c00b000 tx=0x7f349c008040).stop 2022-07-20T02:33:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 msgr2=0x7f34ac14fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f3494000fe0 tx=0x7f3494007b40).stop 2022-07-20T02:33:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 shutdown_connections 2022-07-20T02:33:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34ac142f30 0x7f34ac14fe70 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3490053a10 0x7f3490055ed0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ac150360 0x7f34ac153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 --2- 172.21.15.26:0/1617101567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34ac0670c0 0x7f34ac142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 >> 172.21.15.26:0/1617101567 conn(0x7f34ac05f840 msgr2=0x7f34ac05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 shutdown_connections 2022-07-20T02:33:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.599+0000 7f34abfff700 1 -- 172.21.15.26:0/1617101567 wait complete. 2022-07-20T02:33:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 58 2022-07-20T02:33:47.619 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:33:47.620 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:33:47.621 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:33:47.622 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:47.622 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:33:47.623 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:33:47.624 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:33:47.624 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:47.625 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:33:47.625 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:33:47.626 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:33:47.626 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:33:47.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:47.662 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:33:47.663 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:33:47.664 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:33:47.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:33:47.667 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:33:47.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:33:47.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:47.714 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:33:47.715 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:47.720 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:33:47.722 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:47.732 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:47.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:33:47.797 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:33:47.798 DEBUG:teuthology.parallel:result is None 2022-07-20T02:33:47.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:33:47.800 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:33:47.801 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:33:47.833 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:33:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 Processor -- start 2022-07-20T02:33:47.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 -- start start 2022-07-20T02:33:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58a0147000 0x7f58a0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a01474e0 con 0x7f58a0060f00 2022-07-20T02:33:47.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a00b4e40 con 0x7f58a005cfd0 2022-07-20T02:33:47.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a005d3b0 con 0x7f58a0147000 2022-07-20T02:33:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58a0060f00 0x7f58a013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41208/0 (socket says 172.21.15.26:41208) 2022-07-20T02:33:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589f7fe700 1 -- 172.21.15.26:0/2808044339 learned_addr learned my addr 172.21.15.26:0/2808044339 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:47.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> v2:172.21.15.120:3300/0 conn(0x7f58a0147000 0x7f58a0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801284857 0 0) 0x7f58a01474e0 con 0x7f58a0060f00 2022-07-20T02:33:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003260 con 0x7f58a0060f00 2022-07-20T02:33:47.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1142708248 0 0) 0x7f5880003260 con 0x7f58a0060f00 2022-07-20T02:33:47.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 >> v2:172.21.15.120:3300/0 conn(0x7f58a0147000 msgr2=0x7f58a0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:47.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 --2- 172.21.15.26:0/2808044339 >> v2:172.21.15.120:3300/0 conn(0x7f58a0147000 0x7f58a0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 >> v1:172.21.15.120:6789/0 conn(0x7f58a005cfd0 legacy=0x7f58a00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a01431e0 con 0x7f58a0060f00 2022-07-20T02:33:47.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f58980019e0 con 0x7f58a0060f00 2022-07-20T02:33:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> v2:172.21.15.120:3300/0 conn(0x7f58a0147000 0x7f58a0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:47.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 >> v1:172.21.15.26:6789/0 conn(0x7f58a0060f00 legacy=0x7f58a013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5880003750 0x7f5880003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 0x7f5880004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5880004e70 0x7f588000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a005d3b0 con 0x7f5880004500 2022-07-20T02:33:47.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588000bd20 con 0x7f5880004e70 2022-07-20T02:33:47.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588000bee0 con 0x7f5880003750 2022-07-20T02:33:47.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 0x7f5880004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589effd700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5880003750 0x7f5880003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589f7fe700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5880004e70 0x7f588000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 -- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5880003750 msgr2=0x7f5880003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:47.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5880003750 0x7f5880003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 -- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5880004e70 msgr2=0x7f588000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:47.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5880004e70 0x7f588000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f589f7fe700 1 -- 172.21.15.26:0/2808044339 reap_dead start 2022-07-20T02:33:47.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 -- 172.21.15.26:0/2808044339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880003260 con 0x7f5880004500 2022-07-20T02:33:47.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.907+0000 7f5897fff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 0x7f5880004980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7f588800c000 tx=0x7f5888006df0).ready entity=mon.0 client_cookie=72aa40b5c4559479 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588800e800 con 0x7f5880004500 2022-07-20T02:33:47.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800e960 con 0x7f5880004500 2022-07-20T02:33:47.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589e7fc700 1 -- 172.21.15.26:0/2808044339 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588800ecc0 con 0x7f5880004500 2022-07-20T02:33:47.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 msgr2=0x7f5880004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 0x7f5880004980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7f588800c000 tx=0x7f5888006df0).stop 2022-07-20T02:33:47.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- 172.21.15.26:0/2808044339 shutdown_connections 2022-07-20T02:33:47.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 --2- 172.21.15.26:0/2808044339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5880004500 0x7f5880004980 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- 172.21.15.26:0/2808044339 >> 172.21.15.26:0/2808044339 conn(0x7f58a00620c0 msgr2=0x7f58a005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:47.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- 172.21.15.26:0/2808044339 shutdown_connections 2022-07-20T02:33:47.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- 172.21.15.26:0/2808044339 wait complete. 2022-07-20T02:33:47.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 Processor -- start 2022-07-20T02:33:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- start start 2022-07-20T02:33:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58a00b85b0 0x7f58a013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58a0140f00 0x7f58a0141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00b90a0 con 0x7f58a01406b0 2022-07-20T02:33:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0153000 con 0x7f58a00b85b0 2022-07-20T02:33:47.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0142620 con 0x7f58a0140f00 2022-07-20T02:33:47.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41938/0 (socket says 172.21.15.26:41938) 2022-07-20T02:33:47.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 -- 172.21.15.26:0/1042845593 learned_addr learned my addr 172.21.15.26:0/1042845593 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:47.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589f7fe700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58a0140f00 0x7f58a0141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f589effd700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58a00b85b0 0x7f58a013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:47.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 -- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58a0140f00 msgr2=0x7f58a0141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58a0140f00 0x7f58a0141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 -- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58a00b85b0 msgr2=0x7f58a013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:47.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58a00b85b0 0x7f58a013f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:47.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 -- 172.21.15.26:0/1042845593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a01428c0 con 0x7f58a01406b0 2022-07-20T02:33:47.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.911+0000 7f5897fff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f588800c000 tx=0x7f5888019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:47.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588801c9d0 con 0x7f58a01406b0 2022-07-20T02:33:48.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888000e90 con 0x7f58a01406b0 2022-07-20T02:33:48.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a0152c80 con 0x7f58a01406b0 2022-07-20T02:33:48.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588801cd20 con 0x7f58a01406b0 2022-07-20T02:33:48.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a00b8990 con 0x7f58a01406b0 2022-07-20T02:33:48.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a0142a50 con 0x7f58a01406b0 2022-07-20T02:33:48.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f58880243f0 con 0x7f58a01406b0 2022-07-20T02:33:48.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 0x7f5884055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:48.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f588806d4e0 con 0x7f58a01406b0 2022-07-20T02:33:48.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589effd700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 0x7f5884055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:48.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.915+0000 7f589effd700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 0x7f5884055e10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f589000d000 tx=0x7f5890009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:48.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:47.919+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5888047f50 con 0x7f58a01406b0 2022-07-20T02:33:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.055+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f58a0143480 con 0x7f58a01406b0 2022-07-20T02:33:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.851+0000 7f589cff9700 1 -- 172.21.15.26:0/1042845593 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v116) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5888047b40 con 0x7f58a01406b0 2022-07-20T02:33:48.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 msgr2=0x7f5884055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:48.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 0x7f5884055e10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f589000d000 tx=0x7f5890009040).stop 2022-07-20T02:33:48.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 msgr2=0x7f58a013fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:48.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f588800c000 tx=0x7f5888019000).stop 2022-07-20T02:33:48.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 shutdown_connections 2022-07-20T02:33:48.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f58a01406b0 0x7f58a013fca0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:48.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5884053950 0x7f5884055e10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:48.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58a0140f00 0x7f58a0141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:48.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 --2- 172.21.15.26:0/1042845593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58a00b85b0 0x7f58a013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:48.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 >> 172.21.15.26:0/1042845593 conn(0x7f5880004500 msgr2=0x7f58a0139cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:48.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 shutdown_connections 2022-07-20T02:33:48.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.855+0000 7f589ffff700 1 -- 172.21.15.26:0/1042845593 wait complete. 2022-07-20T02:33:48.874 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:33:48.876 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:33:48.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:33:49.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.995+0000 7f85e54d7700 1 Processor -- start 2022-07-20T02:33:49.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.995+0000 7f85e54d7700 1 -- start start 2022-07-20T02:33:49.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 0x7f85e0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85e0060f00 0x7f85e013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e01474e0 con 0x7f85e0147000 2022-07-20T02:33:49.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e00b4e40 con 0x7f85e0060f00 2022-07-20T02:33:49.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e005d3b0 con 0x7f85e005cfd0 2022-07-20T02:33:49.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e4cd6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85e0060f00 0x7f85e013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 0x7f85e0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85d77fe700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f85e0147000 0x7f85e00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:55440/0 (socket says 172.21.15.26:55440) 2022-07-20T02:33:49.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85d77fe700 1 -- 172.21.15.26:0/1731355791 learned_addr learned my addr 172.21.15.26:0/1731355791 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:49.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85df7fe700 1 -- 172.21.15.26:0/1731355791 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494616917 0 0) 0x7f85e01474e0 con 0x7f85e0147000 2022-07-20T02:33:49.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85df7fe700 1 -- 172.21.15.26:0/1731355791 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c0003260 con 0x7f85e0147000 2022-07-20T02:33:49.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 -- 172.21.15.26:0/1731355791 >> v2:172.21.15.120:3300/0 conn(0x7f85e0060f00 msgr2=0x7f85e013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:49.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 --2- 172.21.15.26:0/1731355791 >> v2:172.21.15.120:3300/0 conn(0x7f85e0060f00 0x7f85e013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 -- 172.21.15.26:0/1731355791 >> v1:172.21.15.120:6789/0 conn(0x7f85e0147000 legacy=0x7f85e00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 -- 172.21.15.26:0/1731355791 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e01431e0 con 0x7f85e005cfd0 2022-07-20T02:33:49.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85dffff700 1 --2- 172.21.15.26:0/1731355791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 0x7f85e0143bc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f85d0002f70 tx=0x7f85d0006df0).ready entity=mon.2 client_cookie=4f5ce6df7b00731c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85df7fe700 1 -- 172.21.15.26:0/1731355791 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d000d4a0 con 0x7f85e005cfd0 2022-07-20T02:33:49.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85df7fe700 1 -- 172.21.15.26:0/1731355791 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0000e90 con 0x7f85e005cfd0 2022-07-20T02:33:49.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85df7fe700 1 -- 172.21.15.26:0/1731355791 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d000d7f0 con 0x7f85e005cfd0 2022-07-20T02:33:49.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- 172.21.15.26:0/1731355791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 msgr2=0x7f85e0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 --2- 172.21.15.26:0/1731355791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 0x7f85e0143bc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f85d0002f70 tx=0x7f85d0006df0).stop 2022-07-20T02:33:49.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- 172.21.15.26:0/1731355791 shutdown_connections 2022-07-20T02:33:49.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 --2- 172.21.15.26:0/1731355791 >> v2:172.21.15.120:3300/0 conn(0x7f85e0060f00 0x7f85e013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 --2- 172.21.15.26:0/1731355791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e005cfd0 0x7f85e0143bc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- 172.21.15.26:0/1731355791 >> 172.21.15.26:0/1731355791 conn(0x7f85e00620c0 msgr2=0x7f85e005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- 172.21.15.26:0/1731355791 shutdown_connections 2022-07-20T02:33:49.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:48.999+0000 7f85e54d7700 1 -- 172.21.15.26:0/1731355791 wait complete. 2022-07-20T02:33:49.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 Processor -- start 2022-07-20T02:33:49.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- start start 2022-07-20T02:33:49.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e0060f00 0x7f85e00ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 0x7f85e0145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00b0790 con 0x7f85e0141690 2022-07-20T02:33:49.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e01561d0 con 0x7f85e0060f00 2022-07-20T02:33:49.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e0142f30 con 0x7f85e0147000 2022-07-20T02:33:49.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e4cd6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 0x7f85e0145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e4cd6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 0x7f85e0145290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41958/0 (socket says 172.21.15.26:41958) 2022-07-20T02:33:49.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35240/0 (socket says 172.21.15.26:35240) 2022-07-20T02:33:49.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e4cd6700 1 -- 172.21.15.26:0/3252933067 learned_addr learned my addr 172.21.15.26:0/3252933067 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:49.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85dffff700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e0060f00 0x7f85e00ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 -- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e0060f00 msgr2=0x7f85e00ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e0060f00 0x7f85e00ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 -- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 msgr2=0x7f85e0145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 0x7f85e0145290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 -- 172.21.15.26:0/3252933067 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d000b180 con 0x7f85e0147000 2022-07-20T02:33:49.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85d77fe700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f85c8001110 tx=0x7f85c8007a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85c800ce80 con 0x7f85e0147000 2022-07-20T02:33:49.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c80070d0 con 0x7f85e0147000 2022-07-20T02:33:49.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e0144f00 con 0x7f85e0147000 2022-07-20T02:33:49.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85c8007430 con 0x7f85e0147000 2022-07-20T02:33:49.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.003+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e01566e0 con 0x7f85e0147000 2022-07-20T02:33:49.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0143480 con 0x7f85e0147000 2022-07-20T02:33:49.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f85c8010070 con 0x7f85e0147000 2022-07-20T02:33:49.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85ddffb700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 0x7f85c4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f85c806b300 con 0x7f85e0147000 2022-07-20T02:33:49.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85dffff700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 0x7f85c4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.007+0000 7f85dffff700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 0x7f85c4055ea0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f85d001a000 tx=0x7f85d0014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.015+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85c8042290 con 0x7f85e0147000 2022-07-20T02:33:49.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.183+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f85e00574e0 con 0x7f85e0147000 2022-07-20T02:33:49.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85ddffb700 1 -- 172.21.15.26:0/3252933067 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f85c8007bc0 con 0x7f85e0147000 2022-07-20T02:33:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:49.189 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:33:49.189 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:33:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:33:49.190 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:33:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:49.191 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.192 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:33:49.194 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:33:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.195 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:49.196 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.198 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:33:49.200 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:33:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.201 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.202 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.204 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:33:49.207 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:33:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.209 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:33:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:33:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.210 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.223 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:33:49.224 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:33:49.239 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.239 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.249 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:33:49.249 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:33:49.250 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.250 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.251 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.251 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.253 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:33:49.253 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:33:49.254 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.255 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.255 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.259 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:33:49.259 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:33:49.260 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.260 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.261 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.261 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.267 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:33:49.268 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:33:49.268 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.269 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.269 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.270 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.272 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:33:49.272 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:33:49.273 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.273 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.273 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.274 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.276 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:33:49.276 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:33:49.277 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.277 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:49.280 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:33:49.280 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:33:49.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.282 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:33:49.283 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:33:49.283 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.285 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:33:49.285 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:33:49.286 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:33:49.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.287 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:33:49.287 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:33:49.288 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.288 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:49.288 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:33:49.288 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:33:49.288 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.289 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:33:49.290 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:33:49.290 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.290 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:49.290 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:33:49.290 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:49.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:49.291 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:33:49.292 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:33:49.292 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:33:49.293 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:33:49.293 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:33:49.293 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:33:49.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:49.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:33:49.294 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:33:49.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 msgr2=0x7f85c4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 0x7f85c4055ea0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f85d001a000 tx=0x7f85d0014000).stop 2022-07-20T02:33:49.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 msgr2=0x7f85e01411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f85c8001110 tx=0x7f85c8007a80).stop 2022-07-20T02:33:49.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 shutdown_connections 2022-07-20T02:33:49.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f85e0141690 0x7f85e0145290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f85c40539e0 0x7f85c4055ea0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e0147000 0x7f85e01411a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 --2- 172.21.15.26:0/3252933067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e0060f00 0x7f85e00ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 >> 172.21.15.26:0/3252933067 conn(0x7f85e005cfd0 msgr2=0x7f85e00b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 shutdown_connections 2022-07-20T02:33:49.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.187+0000 7f85e54d7700 1 -- 172.21.15.26:0/3252933067 wait complete. 2022-07-20T02:33:49.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:49.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.411+0000 7f36557b4700 1 Processor -- start 2022-07-20T02:33:49.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.411+0000 7f36557b4700 1 -- start start 2022-07-20T02:33:49.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f36557b4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f36500957f0 0x7f36500b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f36557b4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3650147080 con 0x7f3650067440 2022-07-20T02:33:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f36557b4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36501472b0 con 0x7f36500b2f20 2022-07-20T02:33:49.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f36557b4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3650143000 con 0x7f36500957f0 2022-07-20T02:33:49.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3654fb3700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3650067440 0x7f36501433f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41244/0 (socket says 172.21.15.26:41244) 2022-07-20T02:33:49.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3654fb3700 1 -- 172.21.15.26:0/1554525011 learned_addr learned my addr 172.21.15.26:0/1554525011 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:49.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3647fff700 1 --2- 172.21.15.26:0/1554525011 >> v2:172.21.15.26:3300/0 conn(0x7f36500957f0 0x7f36500b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285282334 0 0) 0x7f3650147080 con 0x7f3650067440 2022-07-20T02:33:49.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630003260 con 0x7f3650067440 2022-07-20T02:33:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001631624 0 0) 0x7f36501472b0 con 0x7f36500b2f20 2022-07-20T02:33:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3650147080 con 0x7f36500b2f20 2022-07-20T02:33:49.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3647fff700 1 -- 172.21.15.26:0/1554525011 >> v1:172.21.15.120:6789/0 conn(0x7f36500b2f20 legacy=0x7f36500950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1758723579 0 0) 0x7f3630003260 con 0x7f3650067440 2022-07-20T02:33:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3647fff700 1 -- 172.21.15.26:0/1554525011 >> v1:172.21.15.26:6789/0 conn(0x7f3650067440 legacy=0x7f36501433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3647fff700 1 -- 172.21.15.26:0/1554525011 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500a9920 con 0x7f36500957f0 2022-07-20T02:33:49.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f3647fff700 1 --2- 172.21.15.26:0/1554525011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36500957f0 0x7f36500b3300 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f3648001490 tx=0x7f3638006df0).ready entity=mon.0 client_cookie=5a6a82a3a3a512c0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36480019e0 con 0x7f36500957f0 2022-07-20T02:33:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3638000e90 con 0x7f36500957f0 2022-07-20T02:33:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.415+0000 7f364f7fe700 1 -- 172.21.15.26:0/1554525011 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3638012460 con 0x7f36500957f0 2022-07-20T02:33:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- 172.21.15.26:0/1554525011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36500957f0 msgr2=0x7f36500b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 --2- 172.21.15.26:0/1554525011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36500957f0 0x7f36500b3300 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f3648001490 tx=0x7f3638006df0).stop 2022-07-20T02:33:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- 172.21.15.26:0/1554525011 shutdown_connections 2022-07-20T02:33:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 --2- 172.21.15.26:0/1554525011 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f36500957f0 0x7f36500b3300 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- 172.21.15.26:0/1554525011 >> 172.21.15.26:0/1554525011 conn(0x7f36500b67f0 msgr2=0x7f36500b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- 172.21.15.26:0/1554525011 shutdown_connections 2022-07-20T02:33:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- 172.21.15.26:0/1554525011 wait complete. 2022-07-20T02:33:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 Processor -- start 2022-07-20T02:33:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- start start 2022-07-20T02:33:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36500957f0 0x7f36501450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36500b2f20 0x7f365013f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650092b10 con 0x7f3650067440 2022-07-20T02:33:49.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650092db0 con 0x7f36500b2f20 2022-07-20T02:33:49.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f36557b4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3650093050 con 0x7f36500957f0 2022-07-20T02:33:49.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f364ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.419+0000 7f364ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41972/0 (socket says 172.21.15.26:41972) 2022-07-20T02:33:49.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 -- 172.21.15.26:0/469049216 learned_addr learned my addr 172.21.15.26:0/469049216 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:49.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f3647fff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36500957f0 0x7f36501450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f3654fb3700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36500b2f20 0x7f365013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 -- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36500957f0 msgr2=0x7f36501450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36500957f0 0x7f36501450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 -- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36500b2f20 msgr2=0x7f365013f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36500b2f20 0x7f365013f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 -- 172.21.15.26:0/469049216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363800b180 con 0x7f3650067440 2022-07-20T02:33:49.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364ffff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f3640001110 tx=0x7f3640007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3640012400 con 0x7f3650067440 2022-07-20T02:33:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3640012560 con 0x7f3650067440 2022-07-20T02:33:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36500932f0 con 0x7f3650067440 2022-07-20T02:33:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36400128c0 con 0x7f3650067440 2022-07-20T02:33:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3650099fa0 con 0x7f3650067440 2022-07-20T02:33:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36500a95a0 con 0x7f3650067440 2022-07-20T02:33:49.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3640019030 con 0x7f3650067440 2022-07-20T02:33:49.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 0x7f3634055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f3647fff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 0x7f3634055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.423+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f364006b0b0 con 0x7f3650067440 2022-07-20T02:33:49.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.427+0000 7f3647fff700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 0x7f3634055e40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f363801a000 tx=0x7f3638014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.431+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f36400421b0 con 0x7f3650067440 2022-07-20T02:33:49.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.559+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36500574e0 con 0x7f3650067440 2022-07-20T02:33:49.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.559+0000 7f364dffb700 1 -- 172.21.15.26:0/469049216 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+13472 (secure 0 0 0) 0x7f36400457d0 con 0x7f3650067440 2022-07-20T02:33:49.588 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:49.589 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":116,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:48.844832+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:49.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 msgr2=0x7f3634055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 0x7f3634055e40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f363801a000 tx=0x7f3638014000).stop 2022-07-20T02:33:49.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 msgr2=0x7f365009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f3640001110 tx=0x7f3640007b60).stop 2022-07-20T02:33:49.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 shutdown_connections 2022-07-20T02:33:49.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3634053980 0x7f3634055e40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36500b2f20 0x7f365013f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36500957f0 0x7f36501450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 --2- 172.21.15.26:0/469049216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3650067440 0x7f365009b8e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 >> 172.21.15.26:0/469049216 conn(0x7f36500b67f0 msgr2=0x7f3650145c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 shutdown_connections 2022-07-20T02:33:49.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.563+0000 7f36557b4700 1 -- 172.21.15.26:0/469049216 wait complete. 2022-07-20T02:33:49.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:33:49.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.735+0000 7fef0c802700 1 Processor -- start 2022-07-20T02:33:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.735+0000 7fef0c802700 1 -- start start 2022-07-20T02:33:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef080b3260 0x7fef08067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef080980d0 0x7fef08142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0805ff70 con 0x7fef08061860 2022-07-20T02:33:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef080b3640 con 0x7fef080980d0 2022-07-20T02:33:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef080601a0 con 0x7fef080b3260 2022-07-20T02:33:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef077fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef080b3260 0x7fef08067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef080980d0 0x7fef08142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef080980d0 0x7fef08142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33440/0 (socket says 172.21.15.26:33440) 2022-07-20T02:33:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 -- 172.21.15.26:0/3084758439 learned_addr learned my addr 172.21.15.26:0/3084758439 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 -- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef080b3260 msgr2=0x7fef08067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 --2- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef080b3260 0x7fef08067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 -- 172.21.15.26:0/3084758439 >> v1:172.21.15.120:6789/0 conn(0x7fef08061860 legacy=0x7fef080979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 -- 172.21.15.26:0/3084758439 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080606e0 con 0x7fef080980d0 2022-07-20T02:33:49.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef07fff700 1 --2- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080980d0 0x7fef08142d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fef000032f0 tx=0x7fef0000b530).ready entity=mon.1 client_cookie=aef267a41d1da20c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef067fc700 1 -- 172.21.15.26:0/3084758439 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef000153f0 con 0x7fef080980d0 2022-07-20T02:33:49.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef067fc700 1 -- 172.21.15.26:0/3084758439 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef00015550 con 0x7fef080980d0 2022-07-20T02:33:49.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef067fc700 1 -- 172.21.15.26:0/3084758439 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef000158b0 con 0x7fef080980d0 2022-07-20T02:33:49.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080980d0 msgr2=0x7fef08142d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 --2- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080980d0 0x7fef08142d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fef000032f0 tx=0x7fef0000b530).stop 2022-07-20T02:33:49.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- 172.21.15.26:0/3084758439 shutdown_connections 2022-07-20T02:33:49.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 --2- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080980d0 0x7fef08142d30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 --2- 172.21.15.26:0/3084758439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef080b3260 0x7fef08067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- 172.21.15.26:0/3084758439 >> 172.21.15.26:0/3084758439 conn(0x7fef080b67f0 msgr2=0x7fef080b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- 172.21.15.26:0/3084758439 shutdown_connections 2022-07-20T02:33:49.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.739+0000 7fef0c802700 1 -- 172.21.15.26:0/3084758439 wait complete. 2022-07-20T02:33:49.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 Processor -- start 2022-07-20T02:33:49.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 -- start start 2022-07-20T02:33:49.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef080980d0 0x7fef081398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080b3260 0x7fef08139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef08146750 con 0x7fef080980d0 2022-07-20T02:33:49.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080b2650 con 0x7fef080b3260 2022-07-20T02:33:49.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef0c802700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0813ee20 con 0x7fef08061860 2022-07-20T02:33:49.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef077fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.743+0000 7fef06ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef080980d0 0x7fef081398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef07fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080b3260 0x7fef08139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35266/0 (socket says 172.21.15.26:35266) 2022-07-20T02:33:49.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 -- 172.21.15.26:0/67868680 learned_addr learned my addr 172.21.15.26:0/67868680 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:49.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 -- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080b3260 msgr2=0x7fef08139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080b3260 0x7fef08139dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 -- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef080980d0 msgr2=0x7fef081398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef080980d0 0x7fef081398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 -- 172.21.15.26:0/67868680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0000b7d0 con 0x7fef08061860 2022-07-20T02:33:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef077fe700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fef080672c0 tx=0x7feef8006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef8010780 con 0x7fef08061860 2022-07-20T02:33:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef80108e0 con 0x7fef08061860 2022-07-20T02:33:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef080b3970 con 0x7fef08061860 2022-07-20T02:33:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef8010c40 con 0x7fef08061860 2022-07-20T02:33:49.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef080b23b0 con 0x7fef08061860 2022-07-20T02:33:49.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feef801c460 con 0x7fef08061860 2022-07-20T02:33:49.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef080574e0 con 0x7fef08061860 2022-07-20T02:33:49.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 0x7feee8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:49.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7feef806c0f0 con 0x7fef08061860 2022-07-20T02:33:49.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.747+0000 7fef06ffd700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 0x7feee8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:49.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.751+0000 7fef06ffd700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 0x7feee8055fc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7feefc00d000 tx=0x7feefc007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:49.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.759+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feef80430f0 con 0x7fef08061860 2022-07-20T02:33:49.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.887+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fef0806afc0 con 0x7fef08061860 2022-07-20T02:33:49.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.887+0000 7fef04ff9700 1 -- 172.21.15.26:0/67868680 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v116) v1 ==== 110+0+10 (secure 0 0 0) 0x7feef8046710 con 0x7fef08061860 2022-07-20T02:33:49.892 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:49.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.887+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 msgr2=0x7feee8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.887+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 0x7feee8055fc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7feefc00d000 tx=0x7feefc007b90).stop 2022-07-20T02:33:49.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 msgr2=0x7fef080b1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:49.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fef080672c0 tx=0x7feef8006df0).stop 2022-07-20T02:33:49.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 shutdown_connections 2022-07-20T02:33:49.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fef080980d0 0x7fef081398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7feee8053b00 0x7feee8055fc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef080b3260 0x7fef08139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 --2- 172.21.15.26:0/67868680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef08061860 0x7fef080b1890 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:49.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 >> 172.21.15.26:0/67868680 conn(0x7fef080b67f0 msgr2=0x7fef081423d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:49.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 shutdown_connections 2022-07-20T02:33:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:49.891+0000 7fef0c802700 1 -- 172.21.15.26:0/67868680 wait complete. 2022-07-20T02:33:49.924 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:50.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.039+0000 7f102b59e700 1 Processor -- start 2022-07-20T02:33:50.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.039+0000 7f102b59e700 1 -- start start 2022-07-20T02:33:50.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b7920 0x7f102c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f102c0b31b0 0x7f102c0614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f102c0b2c70 0x7f102c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c0608d0 con 0x7f102c0b31b0 2022-07-20T02:33:50.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c0b7d00 con 0x7f102c0b2c70 2022-07-20T02:33:50.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0b4120 con 0x7f102c0b7920 2022-07-20T02:33:50.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f102c0b31b0 0x7f102c0614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f102c0b31b0 0x7f102c0614e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41992/0 (socket says 172.21.15.26:41992) 2022-07-20T02:33:50.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 -- 172.21.15.26:0/1368125501 learned_addr learned my addr 172.21.15.26:0/1368125501 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:50.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102a59c700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b7920 0x7f102c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102ad9d700 1 --2- 172.21.15.26:0/1368125501 >> v2:172.21.15.120:3300/0 conn(0x7f102c0b2c70 0x7f102c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 -- 172.21.15.26:0/1368125501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b7920 msgr2=0x7f102c0b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:50.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b7920 0x7f102c0b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 -- 172.21.15.26:0/1368125501 >> v2:172.21.15.120:3300/0 conn(0x7f102c0b2c70 msgr2=0x7f102c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:50.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1368125501 >> v2:172.21.15.120:3300/0 conn(0x7f102c0b2c70 0x7f102c0b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 -- 172.21.15.26:0/1368125501 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c060b00 con 0x7f102c0b31b0 2022-07-20T02:33:50.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0614e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f101c002f70 tx=0x7f101c006eb0).ready entity=mon.0 client_cookie=45f694f5e781fdd3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1029d9b700 1 -- 172.21.15.26:0/1368125501 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f101c00d570 con 0x7f102c0b31b0 2022-07-20T02:33:50.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1029d9b700 1 -- 172.21.15.26:0/1368125501 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c000e90 con 0x7f102c0b31b0 2022-07-20T02:33:50.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f1029d9b700 1 -- 172.21.15.26:0/1368125501 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f101c00d900 con 0x7f102c0b31b0 2022-07-20T02:33:50.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- 172.21.15.26:0/1368125501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 msgr2=0x7f102c0614e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0614e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f101c002f70 tx=0x7f101c006eb0).stop 2022-07-20T02:33:50.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- 172.21.15.26:0/1368125501 shutdown_connections 2022-07-20T02:33:50.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- 172.21.15.26:0/1368125501 >> v2:172.21.15.120:3300/0 conn(0x7f102c0b2c70 0x7f102c0b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0614e0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 --2- 172.21.15.26:0/1368125501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b7920 0x7f102c0b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- 172.21.15.26:0/1368125501 >> 172.21.15.26:0/1368125501 conn(0x7f102c05f840 msgr2=0x7f102c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- 172.21.15.26:0/1368125501 shutdown_connections 2022-07-20T02:33:50.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- 172.21.15.26:0/1368125501 wait complete. 2022-07-20T02:33:50.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 Processor -- start 2022-07-20T02:33:50.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.043+0000 7f102b59e700 1 -- start start 2022-07-20T02:33:50.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b2c70 0x7f102c0adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102c0b7920 0x7f102c0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c152d50 con 0x7f102c0b31b0 2022-07-20T02:33:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c152fe0 con 0x7f102c0b7920 2022-07-20T02:33:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c153280 con 0x7f102c0b2c70 2022-07-20T02:33:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41998/0 (socket says 172.21.15.26:41998) 2022-07-20T02:33:50.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 -- 172.21.15.26:0/1448880837 learned_addr learned my addr 172.21.15.26:0/1448880837 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102ad9d700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102c0b7920 0x7f102c0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102a59c700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b2c70 0x7f102c0adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 -- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b2c70 msgr2=0x7f102c0adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b2c70 0x7f102c0adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 -- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102c0b7920 msgr2=0x7f102c0af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102c0b7920 0x7f102c0af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 -- 172.21.15.26:0/1448880837 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c00b1e0 con 0x7f102c0b31b0 2022-07-20T02:33:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1021d9b700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f101c002f70 tx=0x7f101c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f101c0116e0 con 0x7f102c0b31b0 2022-07-20T02:33:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c011840 con 0x7f102c0b31b0 2022-07-20T02:33:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0b0e90 con 0x7f102c0b31b0 2022-07-20T02:33:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f101c011ba0 con 0x7f102c0b31b0 2022-07-20T02:33:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c0aea10 con 0x7f102c0b31b0 2022-07-20T02:33:50.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c0b1020 con 0x7f102c0b31b0 2022-07-20T02:33:50.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.047+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f101c025030 con 0x7f102c0b31b0 2022-07-20T02:33:50.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.051+0000 7f1023fff700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 0x7f1004055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.051+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f101c06cd80 con 0x7f102c0b31b0 2022-07-20T02:33:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.051+0000 7f102a59c700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 0x7f1004055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.051+0000 7f102a59c700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 0x7f1004055e10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f101400c000 tx=0x7f1014008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.055+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f101c043e80 con 0x7f102c0b31b0 2022-07-20T02:33:50.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.175+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f102c060da0 con 0x7f102c0b31b0 2022-07-20T02:33:50.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f1023fff700 1 -- 172.21.15.26:0/1448880837 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v116) v1 ==== 92+0+10 (secure 0 0 0) 0x7f101c06e020 con 0x7f102c0b31b0 2022-07-20T02:33:50.181 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:50.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 msgr2=0x7f1004055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 0x7f1004055e10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f101400c000 tx=0x7f1014008040).stop 2022-07-20T02:33:50.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 msgr2=0x7f102c0af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f101c002f70 tx=0x7f101c015000).stop 2022-07-20T02:33:50.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 shutdown_connections 2022-07-20T02:33:50.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f102c0b31b0 0x7f102c0af030 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1004053950 0x7f1004055e10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102c0b7920 0x7f102c0af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 --2- 172.21.15.26:0/1448880837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f102c0b2c70 0x7f102c0adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 >> 172.21.15.26:0/1448880837 conn(0x7f102c05f840 msgr2=0x7f102c0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 shutdown_connections 2022-07-20T02:33:50.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.179+0000 7f102b59e700 1 -- 172.21.15.26:0/1448880837 wait complete. 2022-07-20T02:33:50.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:33:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 Processor -- start 2022-07-20T02:33:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 -- start start 2022-07-20T02:33:50.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f417405f9e0 0x7f41741473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4174060550 0x7f4174066c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41740609c0 con 0x7f41740b0f50 2022-07-20T02:33:50.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41740b13f0 con 0x7f417405f9e0 2022-07-20T02:33:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a9d1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41740b1690 con 0x7f4174060550 2022-07-20T02:33:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41791ce700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41740b0f50 0x7f41740b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41286/0 (socket says 172.21.15.26:41286) 2022-07-20T02:33:50.419 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41791ce700 1 -- 172.21.15.26:0/869141704 learned_addr learned my addr 172.21.15.26:0/869141704 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:50.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 --2- 172.21.15.26:0/869141704 >> v2:172.21.15.120:3300/0 conn(0x7f417405f9e0 0x7f41741473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f417a1d0700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4174060550 0x7f4174066c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41789cd700 1 -- 172.21.15.26:0/869141704 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863321645 0 0) 0x7f41740609c0 con 0x7f41740b0f50 2022-07-20T02:33:50.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41789cd700 1 -- 172.21.15.26:0/869141704 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003260 con 0x7f41740b0f50 2022-07-20T02:33:50.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 -- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4174060550 msgr2=0x7f4174066c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:50.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4174060550 0x7f4174066c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 -- 172.21.15.26:0/869141704 >> v1:172.21.15.26:6789/0 conn(0x7f41740b0f50 legacy=0x7f41740b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 -- 172.21.15.26:0/869141704 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740b1930 con 0x7f417405f9e0 2022-07-20T02:33:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.315+0000 7f41799cf700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f41741473e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f415c0032f0 tx=0x7f415c0091d0).ready entity=mon.1 client_cookie=a3a557a2e9a63185 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f41789cd700 1 -- 172.21.15.26:0/869141704 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f415c00b600 con 0x7f417405f9e0 2022-07-20T02:33:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f41789cd700 1 -- 172.21.15.26:0/869141704 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c00b760 con 0x7f417405f9e0 2022-07-20T02:33:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f41789cd700 1 -- 172.21.15.26:0/869141704 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f415c00b8d0 con 0x7f417405f9e0 2022-07-20T02:33:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 msgr2=0x7f41741473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f41741473e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f415c0032f0 tx=0x7f415c0091d0).stop 2022-07-20T02:33:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- 172.21.15.26:0/869141704 shutdown_connections 2022-07-20T02:33:50.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4174060550 0x7f4174066c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- 172.21.15.26:0/869141704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f41741473e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- 172.21.15.26:0/869141704 >> 172.21.15.26:0/869141704 conn(0x7f41740b3050 msgr2=0x7f41740b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- 172.21.15.26:0/869141704 shutdown_connections 2022-07-20T02:33:50.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- 172.21.15.26:0/869141704 wait complete. 2022-07-20T02:33:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 Processor -- start 2022-07-20T02:33:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- start start 2022-07-20T02:33:50.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f417405cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 0x7f41740ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41740aaed0 con 0x7f4174060550 2022-07-20T02:33:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41740ab100 con 0x7f417405f9e0 2022-07-20T02:33:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.319+0000 7f417a9d1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41740ab3a0 con 0x7f41740b0f50 2022-07-20T02:33:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42012/0 (socket says 172.21.15.26:42012) 2022-07-20T02:33:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 -- 172.21.15.26:0/779927155 learned_addr learned my addr 172.21.15.26:0/779927155 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f417a1d0700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 0x7f41740ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41799cf700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f417405cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 -- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 msgr2=0x7f41740ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 0x7f41740ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 -- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 msgr2=0x7f417405cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f417405cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 -- 172.21.15.26:0/779927155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c009470 con 0x7f4174060550 2022-07-20T02:33:50.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f417a1d0700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 0x7f41740ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:50.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41799cf700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f417405cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:50.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f41791ce700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f4164001110 tx=0x7f4164007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f416400ce20 con 0x7f4174060550 2022-07-20T02:33:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41640072e0 con 0x7f4174060550 2022-07-20T02:33:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4174065c70 con 0x7f4174060550 2022-07-20T02:33:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4164005800 con 0x7f4174060550 2022-07-20T02:33:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41740658d0 con 0x7f4174060550 2022-07-20T02:33:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.323+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41740574e0 con 0x7f4174060550 2022-07-20T02:33:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.327+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f416401a060 con 0x7f4174060550 2022-07-20T02:33:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.327+0000 7f4172ffd700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 0x7f4158055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.327+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f416406b0c0 con 0x7f4174060550 2022-07-20T02:33:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.327+0000 7f41799cf700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 0x7f4158055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.327+0000 7f41799cf700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 0x7f4158055f60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f415c01b000 tx=0x7f415c016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.331+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f41640421c0 con 0x7f4174060550 2022-07-20T02:33:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.463+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f41740ab560 con 0x7f4174060550 2022-07-20T02:33:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.463+0000 7f4172ffd700 1 -- 172.21.15.26:0/779927155 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v116) v1 ==== 84+0+143 (secure 0 0 0) 0x7f4164018070 con 0x7f4174060550 2022-07-20T02:33:50.489 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:50.490 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:33:50.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:50.491 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:33:50.491 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:33:50.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:33:50.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:33:50.492 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:33:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:33:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:33:50.494 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:33:50.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 msgr2=0x7f4158055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 0x7f4158055f60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f415c01b000 tx=0x7f415c016000).stop 2022-07-20T02:33:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 msgr2=0x7f41740abf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f4164001110 tx=0x7f4164007b80).stop 2022-07-20T02:33:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 shutdown_connections 2022-07-20T02:33:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4174060550 0x7f41740abf80 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4158053aa0 0x7f4158055f60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41740b0f50 0x7f41740ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 --2- 172.21.15.26:0/779927155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f417405f9e0 0x7f417405cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 >> 172.21.15.26:0/779927155 conn(0x7f41740b3050 msgr2=0x7f417405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 shutdown_connections 2022-07-20T02:33:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.467+0000 7f417a9d1700 1 -- 172.21.15.26:0/779927155 wait complete. 2022-07-20T02:33:50.510 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:33:50.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:33:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.619+0000 7fc0a3133700 1 Processor -- start 2022-07-20T02:33:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.619+0000 7fc0a3133700 1 -- start start 2022-07-20T02:33:50.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc09c060f00 0x7fc09c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c1474e0 con 0x7fc09c147000 2022-07-20T02:33:50.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0b4e40 con 0x7fc09c05cfd0 2022-07-20T02:33:50.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c05d3b0 con 0x7fc09c060f00 2022-07-20T02:33:50.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a1930700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc09c147000 0x7fc09c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41300/0 (socket says 172.21.15.26:41300) 2022-07-20T02:33:50.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a1930700 1 -- 172.21.15.26:0/4004882004 learned_addr learned my addr 172.21.15.26:0/4004882004 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:50.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a2932700 1 --2- 172.21.15.26:0/4004882004 >> v2:172.21.15.120:3300/0 conn(0x7fc09c060f00 0x7fc09c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805587717 0 0) 0x7fc09c1474e0 con 0x7fc09c147000 2022-07-20T02:33:50.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc084003260 con 0x7fc09c147000 2022-07-20T02:33:50.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639544204 0 0) 0x7fc09c0b4e40 con 0x7fc09c05cfd0 2022-07-20T02:33:50.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09c1474e0 con 0x7fc09c05cfd0 2022-07-20T02:33:50.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a2932700 1 -- 172.21.15.26:0/4004882004 >> v1:172.21.15.120:6789/0 conn(0x7fc09c05cfd0 legacy=0x7fc09c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a2932700 1 -- 172.21.15.26:0/4004882004 >> v1:172.21.15.26:6789/0 conn(0x7fc09c147000 legacy=0x7fc09c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a2932700 1 -- 172.21.15.26:0/4004882004 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c1431e0 con 0x7fc09c060f00 2022-07-20T02:33:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a2932700 1 --2- 172.21.15.26:0/4004882004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c060f00 0x7fc09c143bc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fc0940032f0 tx=0x7fc0940091d0).ready entity=mon.1 client_cookie=38f7405a7d67a764 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc09400b600 con 0x7fc09c060f00 2022-07-20T02:33:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400b760 con 0x7fc09c060f00 2022-07-20T02:33:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a112f700 1 -- 172.21.15.26:0/4004882004 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc09400b8d0 con 0x7fc09c060f00 2022-07-20T02:33:50.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- 172.21.15.26:0/4004882004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c060f00 msgr2=0x7fc09c143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 --2- 172.21.15.26:0/4004882004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c060f00 0x7fc09c143bc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fc0940032f0 tx=0x7fc0940091d0).stop 2022-07-20T02:33:50.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- 172.21.15.26:0/4004882004 shutdown_connections 2022-07-20T02:33:50.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 --2- 172.21.15.26:0/4004882004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c060f00 0x7fc09c143bc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- 172.21.15.26:0/4004882004 >> 172.21.15.26:0/4004882004 conn(0x7fc09c0620c0 msgr2=0x7fc09c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- 172.21.15.26:0/4004882004 shutdown_connections 2022-07-20T02:33:50.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.623+0000 7fc0a3133700 1 -- 172.21.15.26:0/4004882004 wait complete. 2022-07-20T02:33:50.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 Processor -- start 2022-07-20T02:33:50.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 -- start start 2022-07-20T02:33:50.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c060f00 0x7fc09c142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c147000 0x7fc09c1380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1501c0 con 0x7fc09c0b3b20 2022-07-20T02:33:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0b01e0 con 0x7fc09c147000 2022-07-20T02:33:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a3133700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c152ed0 con 0x7fc09c060f00 2022-07-20T02:33:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42028/0 (socket says 172.21.15.26:42028) 2022-07-20T02:33:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 -- 172.21.15.26:0/3546420045 learned_addr learned my addr 172.21.15.26:0/3546420045 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:50.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2131700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c060f00 0x7fc09c142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a1930700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c147000 0x7fc09c1380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 -- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c060f00 msgr2=0x7fc09c142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c060f00 0x7fc09c142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 -- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c147000 msgr2=0x7fc09c1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c147000 0x7fc09c1380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 -- 172.21.15.26:0/3546420045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094009470 con 0x7fc09c0b3b20 2022-07-20T02:33:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc0a2932700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fc09c0b3080 tx=0x7fc094016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.627+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc09400b600 con 0x7fc09c0b3b20 2022-07-20T02:33:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc094011e50 con 0x7fc09c0b3b20 2022-07-20T02:33:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c0b1c10 con 0x7fc09c0b3b20 2022-07-20T02:33:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc09401f5e0 con 0x7fc09c0b3b20 2022-07-20T02:33:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c153140 con 0x7fc09c0b3b20 2022-07-20T02:33:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c143480 con 0x7fc09c0b3b20 2022-07-20T02:33:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc094026030 con 0x7fc09c0b3b20 2022-07-20T02:33:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc08b7fe700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 0x7fc07c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fc09406c750 con 0x7fc09c0b3b20 2022-07-20T02:33:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.631+0000 7fc0a2131700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 0x7fc07c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.635+0000 7fc0a2131700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 0x7fc07c055ed0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fc09800d000 tx=0x7fc098007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.639+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc094043850 con 0x7fc09c0b3b20 2022-07-20T02:33:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.767+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc09c0574e0 con 0x7fc09c0b3b20 2022-07-20T02:33:50.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.863+0000 7fc08b7fe700 1 -- 172.21.15.26:0/3546420045 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v117) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc0940150a0 con 0x7fc09c0b3b20 2022-07-20T02:33:50.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 msgr2=0x7fc07c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 0x7fc07c055ed0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fc09800d000 tx=0x7fc098007a60).stop 2022-07-20T02:33:50.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 msgr2=0x7fc09c1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:50.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fc09c0b3080 tx=0x7fc094016000).stop 2022-07-20T02:33:50.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 shutdown_connections 2022-07-20T02:33:50.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc09c0b3b20 0x7fc09c1385a0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc07c053a10 0x7fc07c055ed0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c147000 0x7fc09c1380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 --2- 172.21.15.26:0/3546420045 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c060f00 0x7fc09c142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:50.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 >> 172.21.15.26:0/3546420045 conn(0x7fc09c05cfd0 msgr2=0x7fc09c146760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:50.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 shutdown_connections 2022-07-20T02:33:50.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:50.867+0000 7fc0a3133700 1 -- 172.21.15.26:0/3546420045 wait complete. 2022-07-20T02:33:50.902 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:33:50.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:33:51.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 Processor -- start 2022-07-20T02:33:51.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 -- start start 2022-07-20T02:33:51.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb1f40b7920 0x7fb1f40b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b31b0 0x7fb1f40614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f40608d0 con 0x7fb1f40b2c70 2022-07-20T02:33:51.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40b7d00 con 0x7fb1f40b7920 2022-07-20T02:33:51.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40b4120 con 0x7fb1f40b31b0 2022-07-20T02:33:51.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f2ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb1f40b7920 0x7fb1f40b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f37fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1f40b2c70 0x7fb1f40b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41314/0 (socket says 172.21.15.26:41314) 2022-07-20T02:33:51.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f37fe700 1 -- 172.21.15.26:0/2209783291 learned_addr learned my addr 172.21.15.26:0/2209783291 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f27fc700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b31b0 0x7fb1f40614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f1ffb700 1 -- 172.21.15.26:0/2209783291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304798370 0 0) 0x7fb1f40608d0 con 0x7fb1f40b2c70 2022-07-20T02:33:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f1ffb700 1 -- 172.21.15.26:0/2209783291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d8003260 con 0x7fb1f40b2c70 2022-07-20T02:33:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f2ffd700 1 -- 172.21.15.26:0/2209783291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b31b0 msgr2=0x7fb1f40614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:51.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f2ffd700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b31b0 0x7fb1f40614e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f2ffd700 1 -- 172.21.15.26:0/2209783291 >> v1:172.21.15.26:6789/0 conn(0x7fb1f40b2c70 legacy=0x7fb1f40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.027+0000 7fb1f2ffd700 1 -- 172.21.15.26:0/2209783291 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4060b00 con 0x7fb1f40b7920 2022-07-20T02:33:51.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f2ffd700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b7920 0x7fb1f40b3c30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fb1e8002f70 tx=0x7fb1e8006df0).ready entity=mon.0 client_cookie=1182e5400b26581d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:51.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f1ffb700 1 -- 172.21.15.26:0/2209783291 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e800e470 con 0x7fb1f40b7920 2022-07-20T02:33:51.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f1ffb700 1 -- 172.21.15.26:0/2209783291 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e8000e90 con 0x7fb1f40b7920 2022-07-20T02:33:51.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f1ffb700 1 -- 172.21.15.26:0/2209783291 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e800e7c0 con 0x7fb1f40b7920 2022-07-20T02:33:51.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2209783291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b7920 msgr2=0x7fb1f40b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b7920 0x7fb1f40b3c30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fb1e8002f70 tx=0x7fb1e8006df0).stop 2022-07-20T02:33:51.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2209783291 shutdown_connections 2022-07-20T02:33:51.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b31b0 0x7fb1f40614e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2209783291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b7920 0x7fb1f40b3c30 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2209783291 >> 172.21.15.26:0/2209783291 conn(0x7fb1f405f840 msgr2=0x7fb1f405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2209783291 shutdown_connections 2022-07-20T02:33:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2209783291 wait complete. 2022-07-20T02:33:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 Processor -- start 2022-07-20T02:33:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- start start 2022-07-20T02:33:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f40b2c70 0x7fb1f40a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b7920 0x7fb1f40ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4156960 con 0x7fb1f40b31b0 2022-07-20T02:33:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40afa80 con 0x7fb1f40b2c70 2022-07-20T02:33:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41566c0 con 0x7fb1f40b7920 2022-07-20T02:33:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42040/0 (socket says 172.21.15.26:42040) 2022-07-20T02:33:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f27fc700 1 -- 172.21.15.26:0/2491311145 learned_addr learned my addr 172.21.15.26:0/2491311145 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f2ffd700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f40b2c70 0x7fb1f40a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.031+0000 7fb1f37fe700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b7920 0x7fb1f40ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 -- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b7920 msgr2=0x7fb1f40ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b7920 0x7fb1f40ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 -- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f40b2c70 msgr2=0x7fb1f40a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f40b2c70 0x7fb1f40a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 -- 172.21.15.26:0/2491311145 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e800b120 con 0x7fb1f40b31b0 2022-07-20T02:33:51.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f27fc700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fb1e400c000 tx=0x7fb1e4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:51.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e4010780 con 0x7fb1f40b31b0 2022-07-20T02:33:51.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e40108e0 con 0x7fb1f40b31b0 2022-07-20T02:33:51.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f40b1510 con 0x7fb1f40b31b0 2022-07-20T02:33:51.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e4010c40 con 0x7fb1f40b31b0 2022-07-20T02:33:51.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f40adc80 con 0x7fb1f40b31b0 2022-07-20T02:33:51.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f4060da0 con 0x7fb1f40b31b0 2022-07-20T02:33:51.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.035+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb1e401c460 con 0x7fb1f40b31b0 2022-07-20T02:33:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.039+0000 7fb1e3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 0x7fb1d0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.039+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5954+0+0 (secure 0 0 0) 0x7fb1e406c210 con 0x7fb1f40b31b0 2022-07-20T02:33:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.039+0000 7fb1f2ffd700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 0x7fb1d0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.039+0000 7fb1f2ffd700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 0x7fb1d0055ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fb1e801a000 tx=0x7fb1e8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:51.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.047+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1e4043140 con 0x7fb1f40b31b0 2022-07-20T02:33:51.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.171+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fb1f40574e0 con 0x7fb1f40b31b0 2022-07-20T02:33:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.875+0000 7fb1e3fff700 1 -- 172.21.15.26:0/2491311145 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v118) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb1e4046760 con 0x7fb1f40b31b0 2022-07-20T02:33:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.879+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 msgr2=0x7fb1d0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.879+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 0x7fb1d0055ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fb1e801a000 tx=0x7fb1e8014000).stop 2022-07-20T02:33:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.879+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 msgr2=0x7fb1f40af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.879+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fb1e400c000 tx=0x7fb1e4006df0).stop 2022-07-20T02:33:51.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 shutdown_connections 2022-07-20T02:33:51.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb1f40b31b0 0x7fb1f40af590 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb1d0053a10 0x7fb1d0055ed0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1f40b7920 0x7fb1f40ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 --2- 172.21.15.26:0/2491311145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1f40b2c70 0x7fb1f40a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:51.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 >> 172.21.15.26:0/2491311145 conn(0x7fb1f405f840 msgr2=0x7fb1f405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:51.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 shutdown_connections 2022-07-20T02:33:51.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:51.883+0000 7fb1f3fff700 1 -- 172.21.15.26:0/2491311145 wait complete. 2022-07-20T02:33:51.930 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:33:51.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:33:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 Processor -- start 2022-07-20T02:33:52.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 -- start start 2022-07-20T02:33:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25b005cfd0 0x7f25b0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b01474e0 con 0x7f25b0147000 2022-07-20T02:33:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b00b4e40 con 0x7f25b0060f00 2022-07-20T02:33:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25af59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b005d3b0 con 0x7f25b005cfd0 2022-07-20T02:33:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25add9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25b0147000 0x7f25b00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41330/0 (socket says 172.21.15.26:41330) 2022-07-20T02:33:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25add9b700 1 -- 172.21.15.26:0/2734817438 learned_addr learned my addr 172.21.15.26:0/2734817438 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.051+0000 7f25ae59c700 1 --2- 172.21.15.26:0/2734817438 >> v2:172.21.15.120:3300/0 conn(0x7f25b005cfd0 0x7f25b0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ad59a700 1 -- 172.21.15.26:0/2734817438 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257151326 0 0) 0x7f25b01474e0 con 0x7f25b0147000 2022-07-20T02:33:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ad59a700 1 -- 172.21.15.26:0/2734817438 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003260 con 0x7f25b0147000 2022-07-20T02:33:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ae59c700 1 -- 172.21.15.26:0/2734817438 >> v1:172.21.15.120:6789/0 conn(0x7f25b0060f00 legacy=0x7f25b013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ae59c700 1 -- 172.21.15.26:0/2734817438 >> v1:172.21.15.26:6789/0 conn(0x7f25b0147000 legacy=0x7f25b00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ae59c700 1 -- 172.21.15.26:0/2734817438 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b01431e0 con 0x7f25b005cfd0 2022-07-20T02:33:52.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ae59c700 1 --2- 172.21.15.26:0/2734817438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b005cfd0 0x7f25b0143bc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f25a00032f0 tx=0x7f25a00091d0).ready entity=mon.1 client_cookie=722126e0747bdbb5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ad59a700 1 -- 172.21.15.26:0/2734817438 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a000b6c0 con 0x7f25b005cfd0 2022-07-20T02:33:52.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ad59a700 1 -- 172.21.15.26:0/2734817438 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a000b820 con 0x7f25b005cfd0 2022-07-20T02:33:52.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25ad59a700 1 -- 172.21.15.26:0/2734817438 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a000b990 con 0x7f25b005cfd0 2022-07-20T02:33:52.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 -- 172.21.15.26:0/2734817438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b005cfd0 msgr2=0x7f25b0143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 --2- 172.21.15.26:0/2734817438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b005cfd0 0x7f25b0143bc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f25a00032f0 tx=0x7f25a00091d0).stop 2022-07-20T02:33:52.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 -- 172.21.15.26:0/2734817438 shutdown_connections 2022-07-20T02:33:52.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 --2- 172.21.15.26:0/2734817438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b005cfd0 0x7f25b0143bc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 -- 172.21.15.26:0/2734817438 >> 172.21.15.26:0/2734817438 conn(0x7f25b00620c0 msgr2=0x7f25b005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 -- 172.21.15.26:0/2734817438 shutdown_connections 2022-07-20T02:33:52.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.055+0000 7f25af59e700 1 -- 172.21.15.26:0/2734817438 wait complete. 2022-07-20T02:33:52.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 Processor -- start 2022-07-20T02:33:52.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 -- start start 2022-07-20T02:33:52.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b0060f00 0x7f25b0145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25b01565e0 0x7f25b0140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00ae5d0 con 0x7f25b0147000 2022-07-20T02:33:52.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00ae330 con 0x7f25b0060f00 2022-07-20T02:33:52.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25af59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b0155c50 con 0x7f25b01565e0 2022-07-20T02:33:52.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25add9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25add9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42054/0 (socket says 172.21.15.26:42054) 2022-07-20T02:33:52.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25add9b700 1 -- 172.21.15.26:0/3889404397 learned_addr learned my addr 172.21.15.26:0/3889404397 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:52.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25ae59c700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b0060f00 0x7f25b0145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.059+0000 7f25aed9d700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25b01565e0 0x7f25b0140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 -- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25b01565e0 msgr2=0x7f25b0140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25b01565e0 0x7f25b0140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 -- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b0060f00 msgr2=0x7f25b0145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b0060f00 0x7f25b0145d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 -- 172.21.15.26:0/3889404397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009470 con 0x7f25b0147000 2022-07-20T02:33:52.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25add9b700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f2598001110 tx=0x7f2598007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25980070d0 con 0x7f25b0147000 2022-07-20T02:33:52.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2598007230 con 0x7f25b0147000 2022-07-20T02:33:52.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b00b85d0 con 0x7f25b0147000 2022-07-20T02:33:52.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2598005610 con 0x7f25b0147000 2022-07-20T02:33:52.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b0142320 con 0x7f25b0147000 2022-07-20T02:33:52.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b01569f0 con 0x7f25b0147000 2022-07-20T02:33:52.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.063+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f259801a060 con 0x7f25b0147000 2022-07-20T02:33:52.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.067+0000 7f25a77fe700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 0x7f2594055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.067+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f259806b390 con 0x7f25b0147000 2022-07-20T02:33:52.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.067+0000 7f25ae59c700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 0x7f2594055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.067+0000 7f25ae59c700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 0x7f2594055fc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f25a001b000 tx=0x7f25a0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.071+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25980421d0 con 0x7f25b0147000 2022-07-20T02:33:52.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.231+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f25b0155e10 con 0x7f25b0147000 2022-07-20T02:33:52.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.243+0000 7f25a77fe700 1 -- 172.21.15.26:0/3889404397 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v60) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2598018020 con 0x7f25b0147000 2022-07-20T02:33:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 msgr2=0x7f2594055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 0x7f2594055fc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f25a001b000 tx=0x7f25a0016000).stop 2022-07-20T02:33:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 msgr2=0x7f25b01560f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f2598001110 tx=0x7f2598007b50).stop 2022-07-20T02:33:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 shutdown_connections 2022-07-20T02:33:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25b0147000 0x7f25b01560f0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2594053b00 0x7f2594055fc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25b01565e0 0x7f25b0140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 --2- 172.21.15.26:0/3889404397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b0060f00 0x7f25b0145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.247+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 >> 172.21.15.26:0/3889404397 conn(0x7f25b005cfd0 msgr2=0x7f25b013b300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.251+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 shutdown_connections 2022-07-20T02:33:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.251+0000 7f25af59e700 1 -- 172.21.15.26:0/3889404397 wait complete. 2022-07-20T02:33:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 21 and data pool 22 2022-07-20T02:33:52.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 Processor -- start 2022-07-20T02:33:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 -- start start 2022-07-20T02:33:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5d700b14b0 0x7f5d700b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700ae370 con 0x7f5d700adf90 2022-07-20T02:33:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700b28b0 con 0x7f5d700614b0 2022-07-20T02:33:52.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d785f3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d700b0610 con 0x7f5d700b14b0 2022-07-20T02:33:52.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7638f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5d700adf90 0x7f5d70060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41352/0 (socket says 172.21.15.26:41352) 2022-07-20T02:33:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7638f700 1 -- 172.21.15.26:0/1200401301 learned_addr learned my addr 172.21.15.26:0/1200401301 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d75b8e700 1 --2- 172.21.15.26:0/1200401301 >> v2:172.21.15.120:3300/0 conn(0x7f5d700b14b0 0x7f5d700b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327391903 0 0) 0x7f5d700ae370 con 0x7f5d700adf90 2022-07-20T02:33:52.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58005470 con 0x7f5d700adf90 2022-07-20T02:33:52.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d75b8e700 1 -- 172.21.15.26:0/1200401301 >> v1:172.21.15.120:6789/0 conn(0x7f5d700614b0 legacy=0x7f5d70061910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d75b8e700 1 -- 172.21.15.26:0/1200401301 >> v1:172.21.15.26:6789/0 conn(0x7f5d700adf90 legacy=0x7f5d70060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2543452942 0 0) 0x7f5d58005470 con 0x7f5d700adf90 2022-07-20T02:33:52.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d75b8e700 1 -- 172.21.15.26:0/1200401301 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700b08b0 con 0x7f5d700b14b0 2022-07-20T02:33:52.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d75b8e700 1 --2- 172.21.15.26:0/1200401301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d700b2c60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f5d64001820 tx=0x7f5d680091d0).ready entity=mon.1 client_cookie=8245591fa4a932c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d64001680 con 0x7f5d700b14b0 2022-07-20T02:33:52.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6800b580 con 0x7f5d700b14b0 2022-07-20T02:33:52.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.375+0000 7f5d7538d700 1 -- 172.21.15.26:0/1200401301 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6800b6f0 con 0x7f5d700b14b0 2022-07-20T02:33:52.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- 172.21.15.26:0/1200401301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 msgr2=0x7f5d700b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 --2- 172.21.15.26:0/1200401301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d700b2c60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f5d64001820 tx=0x7f5d680091d0).stop 2022-07-20T02:33:52.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- 172.21.15.26:0/1200401301 shutdown_connections 2022-07-20T02:33:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 --2- 172.21.15.26:0/1200401301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d700b2c60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- 172.21.15.26:0/1200401301 >> 172.21.15.26:0/1200401301 conn(0x7f5d700b66e0 msgr2=0x7f5d700b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- 172.21.15.26:0/1200401301 shutdown_connections 2022-07-20T02:33:52.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- 172.21.15.26:0/1200401301 wait complete. 2022-07-20T02:33:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 Processor -- start 2022-07-20T02:33:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.379+0000 7f5d785f3700 1 -- start start 2022-07-20T02:33:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d700adf90 0x7f5d70151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d70151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d701522f0 con 0x7f5d700614b0 2022-07-20T02:33:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d70066200 con 0x7f5d700b14b0 2022-07-20T02:33:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700a2440 con 0x7f5d700adf90 2022-07-20T02:33:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42078/0 (socket says 172.21.15.26:42078) 2022-07-20T02:33:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 -- 172.21.15.26:0/3207875680 learned_addr learned my addr 172.21.15.26:0/3207875680 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d75b8e700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d700adf90 0x7f5d70151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d76b90700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d70151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 -- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d700adf90 msgr2=0x7f5d70151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d700adf90 0x7f5d70151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 -- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 msgr2=0x7f5d70151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d70151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 -- 172.21.15.26:0/3207875680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68009470 con 0x7f5d700614b0 2022-07-20T02:33:52.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d7638f700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f5d640018e0 tx=0x7f5d64001b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d64007170 con 0x7f5d700614b0 2022-07-20T02:33:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d640072d0 con 0x7f5d700614b0 2022-07-20T02:33:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d700a26e0 con 0x7f5d700614b0 2022-07-20T02:33:52.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d64004920 con 0x7f5d700614b0 2022-07-20T02:33:52.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.383+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d700a2980 con 0x7f5d700614b0 2022-07-20T02:33:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d70151360 con 0x7f5d700614b0 2022-07-20T02:33:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d6401b070 con 0x7f5d700614b0 2022-07-20T02:33:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d637fe700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 0x7f5d50055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5d6406b810 con 0x7f5d700614b0 2022-07-20T02:33:52.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d75b8e700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 0x7f5d50055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.387+0000 7f5d75b8e700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 0x7f5d50055ea0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f5d6801c000 tx=0x7f5d68016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.395+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d64042570 con 0x7f5d700614b0 2022-07-20T02:33:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.523+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d700574e0 con 0x7f5d700614b0 2022-07-20T02:33:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d637fe700 1 -- 172.21.15.26:0/3207875680 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f5d64045b90 con 0x7f5d700614b0 2022-07-20T02:33:52.533 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:52.533 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":119,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:52.234948+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:50.826732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:51.226718+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 msgr2=0x7f5d50055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 0x7f5d50055ea0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f5d6801c000 tx=0x7f5d68016000).stop 2022-07-20T02:33:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 msgr2=0x7f5d7009eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f5d640018e0 tx=0x7f5d64001b50).stop 2022-07-20T02:33:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 shutdown_connections 2022-07-20T02:33:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5d500539e0 0x7f5d50055ea0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d700b14b0 0x7f5d70151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d700adf90 0x7f5d70151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 --2- 172.21.15.26:0/3207875680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d700614b0 0x7f5d7009eea0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 >> 172.21.15.26:0/3207875680 conn(0x7f5d700b66e0 msgr2=0x7f5d701447c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 shutdown_connections 2022-07-20T02:33:52.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.527+0000 7f5d785f3700 1 -- 172.21.15.26:0/3207875680 wait complete. 2022-07-20T02:33:52.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 Processor -- start 2022-07-20T02:33:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- start start 2022-07-20T02:33:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f16dc0b5ef0 0x7f16dc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a8c40 0x7f16dc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc0b62d0 con 0x7f16dc0a7fc0 2022-07-20T02:33:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16dc14aac0 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc14ad60 con 0x7f16dc0a8c40 2022-07-20T02:33:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f16dc0b5ef0 0x7f16dc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e189e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f16dc0a7fc0 0x7f16dc0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41366/0 (socket says 172.21.15.26:41366) 2022-07-20T02:33:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e189e700 1 -- 172.21.15.26:0/504304670 learned_addr learned my addr 172.21.15.26:0/504304670 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e28a0700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a8c40 0x7f16dc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e109d700 1 -- 172.21.15.26:0/504304670 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242277631 0 0) 0x7f16dc0b62d0 con 0x7f16dc0a7fc0 2022-07-20T02:33:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e109d700 1 -- 172.21.15.26:0/504304670 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c4003260 con 0x7f16dc0a7fc0 2022-07-20T02:33:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 -- 172.21.15.26:0/504304670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a8c40 msgr2=0x7f16dc0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:52.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a8c40 0x7f16dc0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 -- 172.21.15.26:0/504304670 >> v1:172.21.15.26:6789/0 conn(0x7f16dc0a7fc0 legacy=0x7f16dc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 -- 172.21.15.26:0/504304670 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc14b000 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e209f700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc1474d0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f16d8002f70 tx=0x7f16d8006df0).ready entity=mon.0 client_cookie=21f9626e6f7563f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e109d700 1 -- 172.21.15.26:0/504304670 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16d800d570 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e109d700 1 -- 172.21.15.26:0/504304670 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8000e90 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e109d700 1 -- 172.21.15.26:0/504304670 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16d800d8c0 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- 172.21.15.26:0/504304670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0b5ef0 msgr2=0x7f16dc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc1474d0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f16d8002f70 tx=0x7f16d8006df0).stop 2022-07-20T02:33:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- 172.21.15.26:0/504304670 shutdown_connections 2022-07-20T02:33:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a8c40 0x7f16dc0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 --2- 172.21.15.26:0/504304670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc1474d0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- 172.21.15.26:0/504304670 >> 172.21.15.26:0/504304670 conn(0x7f16dc060310 msgr2=0x7f16dc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- 172.21.15.26:0/504304670 shutdown_connections 2022-07-20T02:33:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.663+0000 7f16e30a1700 1 -- 172.21.15.26:0/504304670 wait complete. 2022-07-20T02:33:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 Processor -- start 2022-07-20T02:33:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 -- start start 2022-07-20T02:33:52.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a7fc0 0x7f16dc063390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc065440 con 0x7f16dc0a8c40 2022-07-20T02:33:52.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc062ae0 con 0x7f16dc0b5ef0 2022-07-20T02:33:52.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e30a1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc065670 con 0x7f16dc0a7fc0 2022-07-20T02:33:52.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e189e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e189e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42092/0 (socket says 172.21.15.26:42092) 2022-07-20T02:33:52.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.667+0000 7f16e189e700 1 -- 172.21.15.26:0/3944800162 learned_addr learned my addr 172.21.15.26:0/3944800162 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:52.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e209f700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a7fc0 0x7f16dc063390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e28a0700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 -- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a7fc0 msgr2=0x7f16dc063390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a7fc0 0x7f16dc063390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 -- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 msgr2=0x7f16dc156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 -- 172.21.15.26:0/3944800162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d800b180 con 0x7f16dc0a8c40 2022-07-20T02:33:52.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e28a0700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:52.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e189e700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f16cc001110 tx=0x7f16cc007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16cc00ce80 con 0x7f16dc0a8c40 2022-07-20T02:33:52.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc0072e0 con 0x7f16dc0a8c40 2022-07-20T02:33:52.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc065910 con 0x7f16dc0a8c40 2022-07-20T02:33:52.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16cc005800 con 0x7f16dc0a8c40 2022-07-20T02:33:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc156f30 con 0x7f16dc0a8c40 2022-07-20T02:33:52.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc065aa0 con 0x7f16dc0a8c40 2022-07-20T02:33:52.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f16cc01a060 con 0x7f16dc0a8c40 2022-07-20T02:33:52.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 0x7f16bc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f16cc06b4e0 con 0x7f16dc0a8c40 2022-07-20T02:33:52.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e209f700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 0x7f16bc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.671+0000 7f16e209f700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 0x7f16bc055ff0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f16d8015000 tx=0x7f16d8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.679+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16cc0422a0 con 0x7f16dc0a8c40 2022-07-20T02:33:52.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.807+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16dc14b2a0 con 0x7f16dc0a8c40 2022-07-20T02:33:52.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.807+0000 7f16cb7fe700 1 -- 172.21.15.26:0/3944800162 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f16cc018020 con 0x7f16dc0a8c40 2022-07-20T02:33:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:52.813 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":119,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:52.234948+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:50.826732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:51.226718+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 msgr2=0x7f16bc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 0x7f16bc055ff0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f16d8015000 tx=0x7f16d8014000).stop 2022-07-20T02:33:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 msgr2=0x7f16dc156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f16cc001110 tx=0x7f16cc007b50).stop 2022-07-20T02:33:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 shutdown_connections 2022-07-20T02:33:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16dc0a8c40 0x7f16dc156550 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f16bc053b30 0x7f16bc055ff0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc0b5ef0 0x7f16dc156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 --2- 172.21.15.26:0/3944800162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0a7fc0 0x7f16dc063390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 >> 172.21.15.26:0/3944800162 conn(0x7f16dc060310 msgr2=0x7f16dc148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 shutdown_connections 2022-07-20T02:33:52.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.811+0000 7f16e30a1700 1 -- 172.21.15.26:0/3944800162 wait complete. 2022-07-20T02:33:52.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:33:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 Processor -- start 2022-07-20T02:33:52.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 -- start start 2022-07-20T02:33:52.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c05cfd0 0x7f183c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f183c0b1160 0x7f183c140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 0x7f183c1472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c1435b0 con 0x7f183c0b1160 2022-07-20T02:33:52.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c05d3b0 con 0x7f183c146e80 2022-07-20T02:33:52.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f1841cdb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0b4bc0 con 0x7f183c05cfd0 2022-07-20T02:33:52.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f183c0b1160 0x7f183c140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f183c0b1160 0x7f183c140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42102/0 (socket says 172.21.15.26:42102) 2022-07-20T02:33:52.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 -- 172.21.15.26:0/485165983 learned_addr learned my addr 172.21.15.26:0/485165983 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:52.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183bfff700 1 --2- 172.21.15.26:0/485165983 >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 0x7f183c1472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183b7fe700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c05cfd0 0x7f183c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 -- 172.21.15.26:0/485165983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c05cfd0 msgr2=0x7f183c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:52.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c05cfd0 0x7f183c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 -- 172.21.15.26:0/485165983 >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 msgr2=0x7f183c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:52.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 --2- 172.21.15.26:0/485165983 >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 0x7f183c1472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 -- 172.21.15.26:0/485165983 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0b4e60 con 0x7f183c0b1160 2022-07-20T02:33:52.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183bfff700 1 --2- 172.21.15.26:0/485165983 >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 0x7f183c1472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:52.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183affd700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c0b1160 0x7f183c140cb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f1830002f70 tx=0x7f1830006eb0).ready entity=mon.0 client_cookie=f961963286151409 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:52.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183a7fc700 1 -- 172.21.15.26:0/485165983 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f183000d570 con 0x7f183c0b1160 2022-07-20T02:33:52.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183a7fc700 1 -- 172.21.15.26:0/485165983 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1830000e90 con 0x7f183c0b1160 2022-07-20T02:33:52.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.935+0000 7f183a7fc700 1 -- 172.21.15.26:0/485165983 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f183000d900 con 0x7f183c0b1160 2022-07-20T02:33:52.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- 172.21.15.26:0/485165983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c0b1160 msgr2=0x7f183c140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:52.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c0b1160 0x7f183c140cb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f1830002f70 tx=0x7f1830006eb0).stop 2022-07-20T02:33:52.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- 172.21.15.26:0/485165983 shutdown_connections 2022-07-20T02:33:52.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 --2- 172.21.15.26:0/485165983 >> v2:172.21.15.120:3300/0 conn(0x7f183c146e80 0x7f183c1472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c0b1160 0x7f183c140cb0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 --2- 172.21.15.26:0/485165983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c05cfd0 0x7f183c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:52.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- 172.21.15.26:0/485165983 >> 172.21.15.26:0/485165983 conn(0x7f183c066f70 msgr2=0x7f183c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- 172.21.15.26:0/485165983 shutdown_connections 2022-07-20T02:33:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- 172.21.15.26:0/485165983 wait complete. 2022-07-20T02:33:52.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 Processor -- start 2022-07-20T02:33:52.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.939+0000 7f1841cdb700 1 -- start start 2022-07-20T02:33:52.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f183c0b1160 0x7f183c0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c146e80 0x7f183c0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:52.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c144d10 con 0x7f183c05cfd0 2022-07-20T02:33:52.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c145ba0 con 0x7f183c0b1160 2022-07-20T02:33:52.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c1448b0 con 0x7f183c146e80 2022-07-20T02:33:52.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42106/0 (socket says 172.21.15.26:42106) 2022-07-20T02:33:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 -- 172.21.15.26:0/1968173533 learned_addr learned my addr 172.21.15.26:0/1968173533 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:53.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183affd700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f183c0b1160 0x7f183c0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183bfff700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c146e80 0x7f183c0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 -- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c146e80 msgr2=0x7f183c0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c146e80 0x7f183c0afde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 -- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f183c0b1160 msgr2=0x7f183c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f183c0b1160 0x7f183c0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 -- 172.21.15.26:0/1968173533 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183000b1e0 con 0x7f183c05cfd0 2022-07-20T02:33:53.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f183b7fe700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f182c0030b0 tx=0x7f182c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f182c011780 con 0x7f183c05cfd0 2022-07-20T02:33:53.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c0118e0 con 0x7f183c05cfd0 2022-07-20T02:33:53.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c1450e0 con 0x7f183c05cfd0 2022-07-20T02:33:53.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f182c011c40 con 0x7f183c05cfd0 2022-07-20T02:33:53.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c13ff20 con 0x7f183c05cfd0 2022-07-20T02:33:53.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.943+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c0574e0 con 0x7f183c05cfd0 2022-07-20T02:33:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.947+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f182c01c460 con 0x7f183c05cfd0 2022-07-20T02:33:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.947+0000 7f1838ff9700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 0x7f181c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.947+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f182c06c350 con 0x7f183c05cfd0 2022-07-20T02:33:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.947+0000 7f183affd700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 0x7f181c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.947+0000 7f183affd700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 0x7f181c055e40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f183001c000 tx=0x7f1830015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:52.951+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f182c043010 con 0x7f183c05cfd0 2022-07-20T02:33:53.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.115+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f183c060030 con 0x7f183c05cfd0 2022-07-20T02:33:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.243+0000 7f1838ff9700 1 -- 172.21.15.26:0/1968173533 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f182c046630 con 0x7f183c05cfd0 2022-07-20T02:33:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.243+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 msgr2=0x7f181c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 0x7f181c055e40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f183001c000 tx=0x7f1830015000).stop 2022-07-20T02:33:53.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 msgr2=0x7f183c1464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f182c0030b0 tx=0x7f182c008040).stop 2022-07-20T02:33:53.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 shutdown_connections 2022-07-20T02:33:53.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f181c053980 0x7f181c055e40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f183c146e80 0x7f183c0afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f183c0b1160 0x7f183c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 --2- 172.21.15.26:0/1968173533 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f183c05cfd0 0x7f183c1464a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 >> 172.21.15.26:0/1968173533 conn(0x7f183c066f70 msgr2=0x7f183c143700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:53.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 shutdown_connections 2022-07-20T02:33:53.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.247+0000 7f1841cdb700 1 -- 172.21.15.26:0/1968173533 wait complete. 2022-07-20T02:33:53.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:53.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.443+0000 7f4066f2d700 1 Processor -- start 2022-07-20T02:33:53.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.443+0000 7f4066f2d700 1 -- start start 2022-07-20T02:33:53.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 0x7f4060148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4060066a80 0x7f4060066ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406014b040 con 0x7f4060061050 2022-07-20T02:33:53.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40600a8860 con 0x7f4060066a80 2022-07-20T02:33:53.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600b1bc0 con 0x7f40600b2920 2022-07-20T02:33:53.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f405ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 0x7f4060148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35398/0 (socket says 172.21.15.26:35398) 2022-07-20T02:33:53.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f40654ca700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4060066a80 0x7f4060066ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 -- 172.21.15.26:0/3614629480 learned_addr learned my addr 172.21.15.26:0/3614629480 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:53.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f405ffff700 1 --2- 172.21.15.26:0/3614629480 >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 0x7f4060148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42116/0 (socket says 172.21.15.26:42116) 2022-07-20T02:33:53.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 -- 172.21.15.26:0/3614629480 >> v2:172.21.15.120:3300/0 conn(0x7f4060066a80 msgr2=0x7f4060066ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:53.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3614629480 >> v2:172.21.15.120:3300/0 conn(0x7f4060066a80 0x7f4060066ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 -- 172.21.15.26:0/3614629480 >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 msgr2=0x7f4060148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:53.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3614629480 >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 0x7f4060148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 -- 172.21.15.26:0/3614629480 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060147700 con 0x7f40600b2920 2022-07-20T02:33:53.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3614629480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058009290).ready entity=mon.2 client_cookie=d6897e7b5f7b2ec0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f405f7fe700 1 -- 172.21.15.26:0/3614629480 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405800b680 con 0x7f40600b2920 2022-07-20T02:33:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f405f7fe700 1 -- 172.21.15.26:0/3614629480 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405800b7e0 con 0x7f40600b2920 2022-07-20T02:33:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f405f7fe700 1 -- 172.21.15.26:0/3614629480 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405800b950 con 0x7f40600b2920 2022-07-20T02:33:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- 172.21.15.26:0/3614629480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 msgr2=0x7f4060147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3614629480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058009290).stop 2022-07-20T02:33:53.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- 172.21.15.26:0/3614629480 shutdown_connections 2022-07-20T02:33:53.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3614629480 >> v2:172.21.15.120:3300/0 conn(0x7f4060066a80 0x7f4060066ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3614629480 >> v2:172.21.15.26:3300/0 conn(0x7f4060061050 0x7f4060148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3614629480 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f4060147d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- 172.21.15.26:0/3614629480 >> 172.21.15.26:0/3614629480 conn(0x7f40600b60a0 msgr2=0x7f40600b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:53.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- 172.21.15.26:0/3614629480 shutdown_connections 2022-07-20T02:33:53.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.447+0000 7f4066f2d700 1 -- 172.21.15.26:0/3614629480 wait complete. 2022-07-20T02:33:53.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 Processor -- start 2022-07-20T02:33:53.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 -- start start 2022-07-20T02:33:53.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4060066a80 0x7f406015a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f406015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40600abc30 con 0x7f4060061050 2022-07-20T02:33:53.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4060159fc0 con 0x7f4060066a80 2022-07-20T02:33:53.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4066f2d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600af6c0 con 0x7f40600b2920 2022-07-20T02:33:53.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.451+0000 7f4064cc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42120/0 (socket says 172.21.15.26:42120) 2022-07-20T02:33:53.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 -- 172.21.15.26:0/3354000434 learned_addr learned my addr 172.21.15.26:0/3354000434 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:53.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f40654ca700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f406015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405ffff700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4060066a80 0x7f406015a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 -- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 msgr2=0x7f406015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f406015a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 -- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4060066a80 msgr2=0x7f406015a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4060066a80 0x7f406015a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 -- 172.21.15.26:0/3354000434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058009530 con 0x7f4060061050 2022-07-20T02:33:53.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f40654ca700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f406015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:53.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4064cc9700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405801f070 con 0x7f4060061050 2022-07-20T02:33:53.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058011e50 con 0x7f4060061050 2022-07-20T02:33:53.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40600ab390 con 0x7f4060061050 2022-07-20T02:33:53.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405800b680 con 0x7f4060061050 2022-07-20T02:33:53.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40600aaf90 con 0x7f4060061050 2022-07-20T02:33:53.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406015a180 con 0x7f4060061050 2022-07-20T02:33:53.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f405800f070 con 0x7f4060061050 2022-07-20T02:33:53.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 0x7f4040055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f405806d700 con 0x7f4060061050 2022-07-20T02:33:53.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.455+0000 7f405ffff700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 0x7f4040055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.459+0000 7f405ffff700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 0x7f4040055e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f405400b000 tx=0x7f4054008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.463+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4058044330 con 0x7f4060061050 2022-07-20T02:33:53.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406014ada0 con 0x7f4060061050 2022-07-20T02:33:53.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f405dffb700 1 -- 172.21.15.26:0/3354000434 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f4058047950 con 0x7f4060061050 2022-07-20T02:33:53.631 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:53.631 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":61,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7495,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2095505434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2095505434},{"type":"v1","addr":"172.21.15.26:6835","nonce":2095505434}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":7504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3685430630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3685430630},{"type":"v1","addr":"172.21.15.120:6833","nonce":3685430630}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:52.235120+0000","modified":"2022-07-20T02:33:53.237719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/126332545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":126332545},{"type":"v1","addr":"172.21.15.26:6837","nonce":126332545}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T02:33:53.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 msgr2=0x7f4040055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 0x7f4040055e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f405400b000 tx=0x7f4054008040).stop 2022-07-20T02:33:53.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 msgr2=0x7f40600b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.627+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058017000).stop 2022-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 shutdown_connections 2022-07-20T02:33:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4040053950 0x7f4040055e10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40600b2920 0x7f406015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4060066a80 0x7f406015a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 --2- 172.21.15.26:0/3354000434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4060061050 0x7f40600b3780 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 >> 172.21.15.26:0/3354000434 conn(0x7f40600b60a0 msgr2=0x7f40600b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:53.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 shutdown_connections 2022-07-20T02:33:53.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.631+0000 7f4066f2d700 1 -- 172.21.15.26:0/3354000434 wait complete. 2022-07-20T02:33:53.658 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 61 2022-07-20T02:33:53.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:53.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 Processor -- start 2022-07-20T02:33:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 -- start start 2022-07-20T02:33:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5500b5ef0 0x7ff5501474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 0x7ff5501479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5500b62d0 con 0x7ff5500a8c40 2022-07-20T02:33:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff550147eb0 con 0x7ff5500b5ef0 2022-07-20T02:33:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55014aac0 con 0x7ff5500a7fc0 2022-07-20T02:33:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff5500a8c40 0x7ff5500aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41412/0 (socket says 172.21.15.26:41412) 2022-07-20T02:33:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54f7fe700 1 -- 172.21.15.26:0/1158305641 learned_addr learned my addr 172.21.15.26:0/1158305641 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:53.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54e7fc700 1 --2- 172.21.15.26:0/1158305641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 0x7ff5501479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.767+0000 7ff54effd700 1 --2- 172.21.15.26:0/1158305641 >> v2:172.21.15.120:3300/0 conn(0x7ff5500b5ef0 0x7ff5501474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54e7fc700 1 -- 172.21.15.26:0/1158305641 >> v2:172.21.15.120:3300/0 conn(0x7ff5500b5ef0 msgr2=0x7ff5501474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:53.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54e7fc700 1 --2- 172.21.15.26:0/1158305641 >> v2:172.21.15.120:3300/0 conn(0x7ff5500b5ef0 0x7ff5501474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54e7fc700 1 -- 172.21.15.26:0/1158305641 >> v1:172.21.15.26:6789/0 conn(0x7ff5500a8c40 legacy=0x7ff5500aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54e7fc700 1 -- 172.21.15.26:0/1158305641 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55014ad60 con 0x7ff5500a7fc0 2022-07-20T02:33:53.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54e7fc700 1 --2- 172.21.15.26:0/1158305641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 0x7ff5501479c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff548001110 tx=0x7ff5400091d0).ready entity=mon.2 client_cookie=3608ce3e2016960 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54dffb700 1 -- 172.21.15.26:0/1158305641 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54000b580 con 0x7ff5500a7fc0 2022-07-20T02:33:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54dffb700 1 -- 172.21.15.26:0/1158305641 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000b6e0 con 0x7ff5500a7fc0 2022-07-20T02:33:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54dffb700 1 -- 172.21.15.26:0/1158305641 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54000b850 con 0x7ff5500a7fc0 2022-07-20T02:33:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 -- 172.21.15.26:0/1158305641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 msgr2=0x7ff5501479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 --2- 172.21.15.26:0/1158305641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 0x7ff5501479c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff548001110 tx=0x7ff5400091d0).stop 2022-07-20T02:33:53.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 -- 172.21.15.26:0/1158305641 shutdown_connections 2022-07-20T02:33:53.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 --2- 172.21.15.26:0/1158305641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a7fc0 0x7ff5501479c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 --2- 172.21.15.26:0/1158305641 >> v2:172.21.15.120:3300/0 conn(0x7ff5500b5ef0 0x7ff5501474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 -- 172.21.15.26:0/1158305641 >> 172.21.15.26:0/1158305641 conn(0x7ff550060310 msgr2=0x7ff5500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:53.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 -- 172.21.15.26:0/1158305641 shutdown_connections 2022-07-20T02:33:53.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.771+0000 7ff54ffff700 1 -- 172.21.15.26:0/1158305641 wait complete. 2022-07-20T02:33:53.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 Processor -- start 2022-07-20T02:33:53.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 -- start start 2022-07-20T02:33:53.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5500a7fc0 0x7ff55014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a8c40 0x7ff550156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff550064d10 con 0x7ff5500b5ef0 2022-07-20T02:33:53.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff550064f70 con 0x7ff5500a7fc0 2022-07-20T02:33:53.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff550065210 con 0x7ff5500a8c40 2022-07-20T02:33:53.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42138/0 (socket says 172.21.15.26:42138) 2022-07-20T02:33:53.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 -- 172.21.15.26:0/3723843295 learned_addr learned my addr 172.21.15.26:0/3723843295 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:53.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54e7fc700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a8c40 0x7ff550156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54effd700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5500a7fc0 0x7ff55014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 -- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a8c40 msgr2=0x7ff550156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a8c40 0x7ff550156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 -- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5500a7fc0 msgr2=0x7ff55014e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5500a7fc0 0x7ff55014e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.775+0000 7ff54f7fe700 1 -- 172.21.15.26:0/3723843295 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff540009470 con 0x7ff5500b5ef0 2022-07-20T02:33:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54f7fe700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7ff5480011d0 tx=0x7ff548007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5480070d0 con 0x7ff5500b5ef0 2022-07-20T02:33:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548007230 con 0x7ff5500b5ef0 2022-07-20T02:33:53.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff548005610 con 0x7ff5500b5ef0 2022-07-20T02:33:53.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5500654b0 con 0x7ff5500b5ef0 2022-07-20T02:33:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5500ac190 con 0x7ff5500b5ef0 2022-07-20T02:33:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff550065640 con 0x7ff5500b5ef0 2022-07-20T02:33:53.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff548010070 con 0x7ff5500b5ef0 2022-07-20T02:33:53.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 0x7ff534055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:53.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff54806b790 con 0x7ff5500b5ef0 2022-07-20T02:33:53.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54effd700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 0x7ff534055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:53.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.779+0000 7ff54effd700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 0x7ff534055ea0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7ff53800b000 tx=0x7ff538006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:53.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.787+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff548042550 con 0x7ff5500b5ef0 2022-07-20T02:33:53.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.911+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff55014b000 con 0x7ff5500b5ef0 2022-07-20T02:33:53.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff547fff700 1 -- 172.21.15.26:0/3723843295 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16333 (secure 0 0 0) 0x7ff548045b70 con 0x7ff5500b5ef0 2022-07-20T02:33:53.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:53.920 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":120,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:53.238412+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:50.826732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:51.226718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:53.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 msgr2=0x7ff534055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 0x7ff534055ea0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7ff53800b000 tx=0x7ff538006df0).stop 2022-07-20T02:33:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 msgr2=0x7ff550156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7ff5480011d0 tx=0x7ff548007b50).stop 2022-07-20T02:33:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 shutdown_connections 2022-07-20T02:33:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5500b5ef0 0x7ff550156c80 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5340539e0 0x7ff534055ea0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5500a8c40 0x7ff550156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 --2- 172.21.15.26:0/3723843295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5500a7fc0 0x7ff55014e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 >> 172.21.15.26:0/3723843295 conn(0x7ff550060310 msgr2=0x7ff5500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 shutdown_connections 2022-07-20T02:33:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:53.915+0000 7ff54ffff700 1 -- 172.21.15.26:0/3723843295 wait complete. 2022-07-20T02:33:53.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:54.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 Processor -- start 2022-07-20T02:33:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- start start 2022-07-20T02:33:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd1640b2ac0 0x7fd1640aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd1640a7620 0x7fd1640b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640b2ea0 con 0x7fd164148580 2022-07-20T02:33:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd164147900 con 0x7fd1640a7620 2022-07-20T02:33:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd164147ba0 con 0x7fd1640b2ac0 2022-07-20T02:33:54.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd1640a7620 0x7fd1640b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd1640a7620 0x7fd1640b0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42144/0 (socket says 172.21.15.26:42144) 2022-07-20T02:33:54.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 -- 172.21.15.26:0/1186095128 learned_addr learned my addr 172.21.15.26:0/1186095128 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:54.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd168923700 1 --2- 172.21.15.26:0/1186095128 >> v2:172.21.15.120:3300/0 conn(0x7fd1640b2ac0 0x7fd1640aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 -- 172.21.15.26:0/1186095128 >> v2:172.21.15.120:3300/0 conn(0x7fd1640b2ac0 msgr2=0x7fd1640aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 --2- 172.21.15.26:0/1186095128 >> v2:172.21.15.120:3300/0 conn(0x7fd1640b2ac0 0x7fd1640aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 -- 172.21.15.26:0/1186095128 >> v1:172.21.15.120:6789/0 conn(0x7fd164148580 legacy=0x7fd164066850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 -- 172.21.15.26:0/1186095128 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164147e40 con 0x7fd1640a7620 2022-07-20T02:33:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd163fff700 1 --2- 172.21.15.26:0/1186095128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd1640b0460 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7fd1640a84f0 tx=0x7fd158006eb0).ready entity=mon.0 client_cookie=26d1048a2dbe9901 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd1637fe700 1 -- 172.21.15.26:0/1186095128 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd15800d5c0 con 0x7fd1640a7620 2022-07-20T02:33:54.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd1637fe700 1 -- 172.21.15.26:0/1186095128 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158000e90 con 0x7fd1640a7620 2022-07-20T02:33:54.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd1637fe700 1 -- 172.21.15.26:0/1186095128 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd15800d950 con 0x7fd1640a7620 2022-07-20T02:33:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- 172.21.15.26:0/1186095128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 msgr2=0x7fd1640b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1186095128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd1640b0460 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7fd1640a84f0 tx=0x7fd158006eb0).stop 2022-07-20T02:33:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- 172.21.15.26:0/1186095128 shutdown_connections 2022-07-20T02:33:54.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1186095128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd1640b0460 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1186095128 >> v2:172.21.15.120:3300/0 conn(0x7fd1640b2ac0 0x7fd1640aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- 172.21.15.26:0/1186095128 >> 172.21.15.26:0/1186095128 conn(0x7fd1640b60a0 msgr2=0x7fd1640b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- 172.21.15.26:0/1186095128 shutdown_connections 2022-07-20T02:33:54.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.059+0000 7fd16ab87700 1 -- 172.21.15.26:0/1186095128 wait complete. 2022-07-20T02:33:54.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 Processor -- start 2022-07-20T02:33:54.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- start start 2022-07-20T02:33:54.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 0x7fd1640b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 0x7fd1640adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640b3fa0 con 0x7fd1640a7620 2022-07-20T02:33:54.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640b41d0 con 0x7fd164148580 2022-07-20T02:33:54.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640b4470 con 0x7fd1640b2ac0 2022-07-20T02:33:54.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42148/0 (socket says 172.21.15.26:42148) 2022-07-20T02:33:54.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 -- 172.21.15.26:0/1697255018 learned_addr learned my addr 172.21.15.26:0/1697255018 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:54.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd163fff700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 0x7fd1640b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd169124700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 0x7fd1640adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 -- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 msgr2=0x7fd1640b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 0x7fd1640b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 -- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 msgr2=0x7fd1640adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 0x7fd1640adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 -- 172.21.15.26:0/1697255018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15800b1e0 con 0x7fd1640a7620 2022-07-20T02:33:54.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd169124700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 0x7fd1640adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:54.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd163fff700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 0x7fd1640b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:54.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd168923700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fd1540030b0 tx=0x7fd154008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd154011780 con 0x7fd1640a7620 2022-07-20T02:33:54.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540118e0 con 0x7fd1640a7620 2022-07-20T02:33:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1640b4710 con 0x7fd1640a7620 2022-07-20T02:33:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd154011c40 con 0x7fd1640a7620 2022-07-20T02:33:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.063+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164156870 con 0x7fd1640a7620 2022-07-20T02:33:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1641480e0 con 0x7fd1640a7620 2022-07-20T02:33:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd154012420 con 0x7fd1640a7620 2022-07-20T02:33:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd161ffb700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 0x7fd144055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd15406bf10 con 0x7fd1640a7620 2022-07-20T02:33:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd163fff700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 0x7fd144055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.067+0000 7fd163fff700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 0x7fd144055ed0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fd158017000 tx=0x7fd158015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.071+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd154042c70 con 0x7fd1640a7620 2022-07-20T02:33:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.239+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd1640574e0 con 0x7fd1640a7620 2022-07-20T02:33:54.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.247+0000 7fd161ffb700 1 -- 172.21.15.26:0/1697255018 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd154046290 con 0x7fd1640a7620 2022-07-20T02:33:54.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 msgr2=0x7fd144055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 0x7fd144055ed0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fd158017000 tx=0x7fd158015000).stop 2022-07-20T02:33:54.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 msgr2=0x7fd16414d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fd1540030b0 tx=0x7fd154008040).stop 2022-07-20T02:33:54.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 shutdown_connections 2022-07-20T02:33:54.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd144053a10 0x7fd144055ed0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd164148580 0x7fd1640adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1640b2ac0 0x7fd1640b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 --2- 172.21.15.26:0/1697255018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1640a7620 0x7fd16414d560 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 >> 172.21.15.26:0/1697255018 conn(0x7fd1640b60a0 msgr2=0x7fd164067010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 shutdown_connections 2022-07-20T02:33:54.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.251+0000 7fd16ab87700 1 -- 172.21.15.26:0/1697255018 wait complete. 2022-07-20T02:33:54.276 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:33:54.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:33:54.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 Processor -- start 2022-07-20T02:33:54.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 -- start start 2022-07-20T02:33:54.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f75941465b0 0x7f7594066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 0x7f75940604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940b1ae0 con 0x7f75940672a0 2022-07-20T02:33:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940b2e20 con 0x7f75941465b0 2022-07-20T02:33:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7598a81700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940b1d80 con 0x7f75940b24c0 2022-07-20T02:33:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f75937fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f75941465b0 0x7f7594066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7593fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f75940672a0 0x7f7594146990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41438/0 (socket says 172.21.15.26:41438) 2022-07-20T02:33:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7593fff700 1 -- 172.21.15.26:0/3423901067 learned_addr learned my addr 172.21.15.26:0/3423901067 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.407+0000 7f7592ffd700 1 --2- 172.21.15.26:0/3423901067 >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 0x7f75940604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75937fe700 1 -- 172.21.15.26:0/3423901067 >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 msgr2=0x7f75940604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75937fe700 1 --2- 172.21.15.26:0/3423901067 >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 0x7f75940604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75937fe700 1 -- 172.21.15.26:0/3423901067 >> v1:172.21.15.26:6789/0 conn(0x7f75940672a0 legacy=0x7f7594146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75937fe700 1 -- 172.21.15.26:0/3423901067 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940b2020 con 0x7f75941465b0 2022-07-20T02:33:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7592ffd700 1 --2- 172.21.15.26:0/3423901067 >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 0x7f75940604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75937fe700 1 --2- 172.21.15.26:0/3423901067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75941465b0 0x7f7594066db0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f7584002f70 tx=0x7f7584006df0).ready entity=mon.0 client_cookie=31584169ecfcdc8e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75927fc700 1 -- 172.21.15.26:0/3423901067 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f758400d5c0 con 0x7f75941465b0 2022-07-20T02:33:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75927fc700 1 -- 172.21.15.26:0/3423901067 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7584000e90 con 0x7f75941465b0 2022-07-20T02:33:54.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f75927fc700 1 -- 172.21.15.26:0/3423901067 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f758400d910 con 0x7f75941465b0 2022-07-20T02:33:54.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 -- 172.21.15.26:0/3423901067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75941465b0 msgr2=0x7f7594066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 --2- 172.21.15.26:0/3423901067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75941465b0 0x7f7594066db0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f7584002f70 tx=0x7f7584006df0).stop 2022-07-20T02:33:54.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 -- 172.21.15.26:0/3423901067 shutdown_connections 2022-07-20T02:33:54.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 --2- 172.21.15.26:0/3423901067 >> v2:172.21.15.120:3300/0 conn(0x7f75940b24c0 0x7f75940604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 --2- 172.21.15.26:0/3423901067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75941465b0 0x7f7594066db0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 -- 172.21.15.26:0/3423901067 >> 172.21.15.26:0/3423901067 conn(0x7f75940b3050 msgr2=0x7f75940b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 -- 172.21.15.26:0/3423901067 shutdown_connections 2022-07-20T02:33:54.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.411+0000 7f7598a81700 1 -- 172.21.15.26:0/3423901067 wait complete. 2022-07-20T02:33:54.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 Processor -- start 2022-07-20T02:33:54.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- start start 2022-07-20T02:33:54.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75940672a0 0x7f7594060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75941465b0 0x7f759413f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75941438a0 con 0x7f7594145c10 2022-07-20T02:33:54.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594153070 con 0x7f75940672a0 2022-07-20T02:33:54.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7594153310 con 0x7f75941465b0 2022-07-20T02:33:54.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42166/0 (socket says 172.21.15.26:42166) 2022-07-20T02:33:54.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 -- 172.21.15.26:0/1203392327 learned_addr learned my addr 172.21.15.26:0/1203392327 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:54.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f75937fe700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75940672a0 0x7f7594060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7592ffd700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75941465b0 0x7f759413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 -- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75941465b0 msgr2=0x7f759413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75941465b0 0x7f759413f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 -- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75940672a0 msgr2=0x7f7594060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75940672a0 0x7f7594060a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 -- 172.21.15.26:0/1203392327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758400b180 con 0x7f7594145c10 2022-07-20T02:33:54.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7593fff700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f75940b0dc0 tx=0x7f758c007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f758c005ba0 con 0x7f7594145c10 2022-07-20T02:33:54.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c005d00 con 0x7f7594145c10 2022-07-20T02:33:54.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f758c0072c0 con 0x7f7594145c10 2022-07-20T02:33:54.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75941401e0 con 0x7f7594145c10 2022-07-20T02:33:54.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.415+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7594143d30 con 0x7f7594145c10 2022-07-20T02:33:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f758c010070 con 0x7f7594145c10 2022-07-20T02:33:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f7590ff9700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 0x7f7574055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f758c06b9d0 con 0x7f7594145c10 2022-07-20T02:33:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f75937fe700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 0x7f7574055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7594143a10 con 0x7f7594145c10 2022-07-20T02:33:54.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.419+0000 7f75937fe700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 0x7f7574055ff0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f758401a000 tx=0x7f7584014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.427+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7594143a10 con 0x7f7594145c10 2022-07-20T02:33:54.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.587+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7594143a10 con 0x7f7594145c10 2022-07-20T02:33:54.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7590ff9700 1 -- 172.21.15.26:0/1203392327 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7594143a10 con 0x7f7594145c10 2022-07-20T02:33:54.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 msgr2=0x7f7574055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 0x7f7574055ff0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f758401a000 tx=0x7f7584014000).stop 2022-07-20T02:33:54.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 msgr2=0x7f759413fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f75940b0dc0 tx=0x7f758c007f10).stop 2022-07-20T02:33:54.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 shutdown_connections 2022-07-20T02:33:54.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7594145c10 0x7f759413fcf0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7574053b30 0x7f7574055ff0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75941465b0 0x7f759413f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 --2- 172.21.15.26:0/1203392327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75940672a0 0x7f7594060a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 >> 172.21.15.26:0/1203392327 conn(0x7f75940b3050 msgr2=0x7f759405cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 shutdown_connections 2022-07-20T02:33:54.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.595+0000 7f7598a81700 1 -- 172.21.15.26:0/1203392327 wait complete. 2022-07-20T02:33:54.619 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:33:54.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.743+0000 7f9bd25f7700 1 Processor -- start 2022-07-20T02:33:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.743+0000 7f9bd25f7700 1 -- start start 2022-07-20T02:33:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9bcc147000 0x7f9bcc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc060f00 0x7f9bcc13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc1474e0 con 0x7f9bcc05cfd0 2022-07-20T02:33:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b4e40 con 0x7f9bcc147000 2022-07-20T02:33:54.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc05d3b0 con 0x7f9bcc060f00 2022-07-20T02:33:54.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9bcc147000 0x7f9bcc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd15f5700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9bcc05cfd0 0x7f9bcc0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41450/0 (socket says 172.21.15.26:41450) 2022-07-20T02:33:54.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd15f5700 1 -- 172.21.15.26:0/3789033417 learned_addr learned my addr 172.21.15.26:0/3789033417 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:54.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd1df6700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc060f00 0x7f9bcc13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 -- 172.21.15.26:0/3789033417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc060f00 msgr2=0x7f9bcc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:54.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc060f00 0x7f9bcc13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 -- 172.21.15.26:0/3789033417 >> v1:172.21.15.26:6789/0 conn(0x7f9bcc05cfd0 legacy=0x7f9bcc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 -- 172.21.15.26:0/3789033417 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc1431e0 con 0x7f9bcc147000 2022-07-20T02:33:54.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd0df4700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc147000 0x7f9bcc143bc0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f9bbc002f70 tx=0x7f9bbc006df0).ready entity=mon.0 client_cookie=bbeaf3568c726873 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bbbfff700 1 -- 172.21.15.26:0/3789033417 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bbc00d570 con 0x7f9bcc147000 2022-07-20T02:33:54.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bbbfff700 1 -- 172.21.15.26:0/3789033417 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc000e90 con 0x7f9bcc147000 2022-07-20T02:33:54.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bbbfff700 1 -- 172.21.15.26:0/3789033417 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bbc00d8c0 con 0x7f9bcc147000 2022-07-20T02:33:54.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- 172.21.15.26:0/3789033417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc147000 msgr2=0x7f9bcc143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc147000 0x7f9bcc143bc0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f9bbc002f70 tx=0x7f9bbc006df0).stop 2022-07-20T02:33:54.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- 172.21.15.26:0/3789033417 shutdown_connections 2022-07-20T02:33:54.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc060f00 0x7f9bcc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/3789033417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc147000 0x7f9bcc143bc0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- 172.21.15.26:0/3789033417 >> 172.21.15.26:0/3789033417 conn(0x7f9bcc0620c0 msgr2=0x7f9bcc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- 172.21.15.26:0/3789033417 shutdown_connections 2022-07-20T02:33:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.747+0000 7f9bd25f7700 1 -- 172.21.15.26:0/3789033417 wait complete. 2022-07-20T02:33:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 Processor -- start 2022-07-20T02:33:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- start start 2022-07-20T02:33:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc060f00 0x7f9bcc0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 0x7f9bcc1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc155d90 con 0x7f9bcc1409e0 2022-07-20T02:33:54.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0af100 con 0x7f9bcc060f00 2022-07-20T02:33:54.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0af3f0 con 0x7f9bcc147000 2022-07-20T02:33:54.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42180/0 (socket says 172.21.15.26:42180) 2022-07-20T02:33:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 -- 172.21.15.26:0/1273923213 learned_addr learned my addr 172.21.15.26:0/1273923213 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd15f5700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc060f00 0x7f9bcc0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd0df4700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 0x7f9bcc1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 -- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 msgr2=0x7f9bcc1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 0x7f9bcc1558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 -- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc060f00 msgr2=0x7f9bcc0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc060f00 0x7f9bcc0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 -- 172.21.15.26:0/1273923213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc00b180 con 0x7f9bcc1409e0 2022-07-20T02:33:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd0df4700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 0x7f9bcc1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd1df6700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f9bc400a000 tx=0x7f9bc4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc40127e0 con 0x7f9bcc1409e0 2022-07-20T02:33:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4012940 con 0x7f9bcc1409e0 2022-07-20T02:33:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc0af690 con 0x7f9bcc1409e0 2022-07-20T02:33:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc4012ca0 con 0x7f9bcc1409e0 2022-07-20T02:33:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.751+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc0aed50 con 0x7f9bcc1409e0 2022-07-20T02:33:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc143480 con 0x7f9bcc1409e0 2022-07-20T02:33:54.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9bc4013480 con 0x7f9bcc1409e0 2022-07-20T02:33:54.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bba7fc700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 0x7f9bac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:54.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9bc406c880 con 0x7f9bcc1409e0 2022-07-20T02:33:54.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bd15f5700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 0x7f9bac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:54.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.755+0000 7f9bd15f5700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 0x7f9bac055fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f9bc800d000 tx=0x7f9bc8007be0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:54.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.763+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9bc4043540 con 0x7f9bcc1409e0 2022-07-20T02:33:54.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bcc0574e0 con 0x7f9bcc1409e0 2022-07-20T02:33:54.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bba7fc700 1 -- 172.21.15.26:0/1273923213 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f9bc4012e00 con 0x7f9bcc1409e0 2022-07-20T02:33:54.930 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:54.930 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":61,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7495,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2095505434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2095505434},{"type":"v1","addr":"172.21.15.26:6835","nonce":2095505434}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":7504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3685430630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3685430630},{"type":"v1","addr":"172.21.15.120:6833","nonce":3685430630}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:52.235120+0000","modified":"2022-07-20T02:33:53.237719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/126332545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":126332545},{"type":"v1","addr":"172.21.15.26:6837","nonce":126332545}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T02:33:54.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 msgr2=0x7f9bac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 0x7f9bac055fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f9bc800d000 tx=0x7f9bc8007be0).stop 2022-07-20T02:33:54.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 msgr2=0x7f9bcc0ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:54.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f9bc400a000 tx=0x7f9bc4008040).stop 2022-07-20T02:33:54.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 shutdown_connections 2022-07-20T02:33:54.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bcc1409e0 0x7f9bcc0ad9a0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9bac053b00 0x7f9bac055fc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bcc147000 0x7f9bcc1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 --2- 172.21.15.26:0/1273923213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc060f00 0x7f9bcc0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:54.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 >> 172.21.15.26:0/1273923213 conn(0x7f9bcc05cfd0 msgr2=0x7f9bcc1427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:54.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 shutdown_connections 2022-07-20T02:33:54.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:54.927+0000 7f9bd25f7700 1 -- 172.21.15.26:0/1273923213 wait complete. 2022-07-20T02:33:54.946 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 61 2022-07-20T02:33:55.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:52.235120+0000', 'modified': '2022-07-20T02:33:53.237719+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'c', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/126332545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 126332545}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 126332545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:55.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:55.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:55.395 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7725,"want_state":"up:active","state":"up:active","rank_uptime":3.1470869819999998,"mdsmap_epoch":61,"osdmap_epoch":120,"osdmap_epoch_barrier":119,"uptime":7.5515002779999998} 2022-07-20T02:33:55.396 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7725, "mdsmap_epoch": 61, "osdmap_epoch": 120, "osdmap_epoch_barrier": 119, "rank_uptime": 3.147086982, "state": "up:active", "uptime": 7.551500278, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:55.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 Processor -- start 2022-07-20T02:33:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 -- start start 2022-07-20T02:33:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd47c144ff0 0x7fd47c061510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd47c05c730 0x7fd47c0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c14b5a0 con 0x7fd47c14aef0 2022-07-20T02:33:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c095be0 con 0x7fd47c144ff0 2022-07-20T02:33:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd482f08700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c14b840 con 0x7fd47c05c730 2022-07-20T02:33:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd47c144ff0 0x7fd47c061510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd480ca4700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd47c14aef0 0x7fd47c1448e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41464/0 (socket says 172.21.15.26:41464) 2022-07-20T02:33:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd47c144ff0 0x7fd47c061510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42188/0 (socket says 172.21.15.26:42188) 2022-07-20T02:33:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd480ca4700 1 -- 172.21.15.26:0/2030727405 learned_addr learned my addr 172.21.15.26:0/2030727405 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd4814a5700 1 --2- 172.21.15.26:0/2030727405 >> v2:172.21.15.120:3300/0 conn(0x7fd47c05c730 0x7fd47c0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 -- 172.21.15.26:0/2030727405 >> v2:172.21.15.120:3300/0 conn(0x7fd47c05c730 msgr2=0x7fd47c0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 --2- 172.21.15.26:0/2030727405 >> v2:172.21.15.120:3300/0 conn(0x7fd47c05c730 0x7fd47c0b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 -- 172.21.15.26:0/2030727405 >> v1:172.21.15.26:6789/0 conn(0x7fd47c14aef0 legacy=0x7fd47c1448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47b7fe700 1 -- 172.21.15.26:0/2030727405 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653085124 0 0) 0x7fd47c14b5a0 con 0x7fd47c14aef0 2022-07-20T02:33:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 -- 172.21.15.26:0/2030727405 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0608e0 con 0x7fd47c144ff0 2022-07-20T02:33:55.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47bfff700 1 --2- 172.21.15.26:0/2030727405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c144ff0 0x7fd47c061510 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fd4640032f0 tx=0x7fd4640091d0).ready entity=mon.0 client_cookie=34eebd0577bd2daf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47b7fe700 1 -- 172.21.15.26:0/2030727405 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46400b680 con 0x7fd47c144ff0 2022-07-20T02:33:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47b7fe700 1 -- 172.21.15.26:0/2030727405 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46400b7e0 con 0x7fd47c144ff0 2022-07-20T02:33:55.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.499+0000 7fd47b7fe700 1 -- 172.21.15.26:0/2030727405 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46400b950 con 0x7fd47c144ff0 2022-07-20T02:33:55.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- 172.21.15.26:0/2030727405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c144ff0 msgr2=0x7fd47c061510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 --2- 172.21.15.26:0/2030727405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c144ff0 0x7fd47c061510 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fd4640032f0 tx=0x7fd4640091d0).stop 2022-07-20T02:33:55.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- 172.21.15.26:0/2030727405 shutdown_connections 2022-07-20T02:33:55.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 --2- 172.21.15.26:0/2030727405 >> v2:172.21.15.120:3300/0 conn(0x7fd47c05c730 0x7fd47c0b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 --2- 172.21.15.26:0/2030727405 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c144ff0 0x7fd47c061510 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- 172.21.15.26:0/2030727405 >> 172.21.15.26:0/2030727405 conn(0x7fd47c0b5e70 msgr2=0x7fd47c0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:55.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- 172.21.15.26:0/2030727405 shutdown_connections 2022-07-20T02:33:55.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- 172.21.15.26:0/2030727405 wait complete. 2022-07-20T02:33:55.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 Processor -- start 2022-07-20T02:33:55.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 -- start start 2022-07-20T02:33:55.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.503+0000 7fd482f08700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 0x7fd47c14d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 0x7fd47c0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c146030 con 0x7fd47c05c730 2022-07-20T02:33:55.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c145af0 con 0x7fd47c144ff0 2022-07-20T02:33:55.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c145d90 con 0x7fd47c14aef0 2022-07-20T02:33:55.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42190/0 (socket says 172.21.15.26:42190) 2022-07-20T02:33:55.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd47bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 0x7fd47c14d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 -- 172.21.15.26:0/3461049567 learned_addr learned my addr 172.21.15.26:0/3461049567 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:55.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd4814a5700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 0x7fd47c0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 -- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 msgr2=0x7fd47c0b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 0x7fd47c0b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 -- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 msgr2=0x7fd47c14d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 0x7fd47c14d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 -- 172.21.15.26:0/3461049567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd464009470 con 0x7fd47c05c730 2022-07-20T02:33:55.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd47bfff700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 0x7fd47c14d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:55.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd4814a5700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 0x7fd47c0b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:55.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd480ca4700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fd46c000fe0 tx=0x7fd46c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46c00ce20 con 0x7fd47c05c730 2022-07-20T02:33:55.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c0072e0 con 0x7fd47c05c730 2022-07-20T02:33:55.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46c005800 con 0x7fd47c05c730 2022-07-20T02:33:55.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c147550 con 0x7fd47c05c730 2022-07-20T02:33:55.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c1454c0 con 0x7fd47c05c730 2022-07-20T02:33:55.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.507+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c0574e0 con 0x7fd47c05c730 2022-07-20T02:33:55.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.511+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd46c010070 con 0x7fd47c05c730 2022-07-20T02:33:55.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.511+0000 7fd479ffb700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 0x7fd454055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.511+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd46c06b870 con 0x7fd47c05c730 2022-07-20T02:33:55.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.511+0000 7fd47bfff700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 0x7fd454055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.511+0000 7fd47bfff700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 0x7fd454055f30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fd46401b000 tx=0x7fd464016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.519+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd46c042520 con 0x7fd47c05c730 2022-07-20T02:33:55.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.643+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd47c0683d0 con 0x7fd47c05c730 2022-07-20T02:33:55.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.643+0000 7fd479ffb700 1 -- 172.21.15.26:0/3461049567 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16333 (secure 0 0 0) 0x7fd46c045b40 con 0x7fd47c05c730 2022-07-20T02:33:55.657 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:55.658 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":120,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:53.238412+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:50.826732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:51.226718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:55.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 msgr2=0x7fd454055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 0x7fd454055f30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fd46401b000 tx=0x7fd464016000).stop 2022-07-20T02:33:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 msgr2=0x7fd47c157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fd46c000fe0 tx=0x7fd46c007c40).stop 2022-07-20T02:33:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 shutdown_connections 2022-07-20T02:33:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd454053a70 0x7fd454055f30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd47c14aef0 0x7fd47c0b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd47c144ff0 0x7fd47c14d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 --2- 172.21.15.26:0/3461049567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd47c05c730 0x7fd47c157320 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 >> 172.21.15.26:0/3461049567 conn(0x7fd47c0b5e70 msgr2=0x7fd47c14a450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 shutdown_connections 2022-07-20T02:33:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.647+0000 7fd482f08700 1 -- 172.21.15.26:0/3461049567 wait complete. 2022-07-20T02:33:55.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:33:50.826732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:33:51.226718+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:55.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.803+0000 7f569088e700 1 Processor -- start 2022-07-20T02:33:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.803+0000 7f569088e700 1 -- start start 2022-07-20T02:33:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f569088e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 0x7f568c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f569088e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f568c0b1160 0x7f568c140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f569088e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c147a00 con 0x7f568c146e80 2022-07-20T02:33:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f569088e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c1435b0 con 0x7f568c0b1160 2022-07-20T02:33:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f569088e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c05d3b0 con 0x7f568c05cfd0 2022-07-20T02:33:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568ad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f568c146e80 0x7f568c1472f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41480/0 (socket says 172.21.15.26:41480) 2022-07-20T02:33:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568ad9d700 1 -- 172.21.15.26:0/2628918912 learned_addr learned my addr 172.21.15.26:0/2628918912 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f5689d9b700 1 --2- 172.21.15.26:0/2628918912 >> v2:172.21.15.120:3300/0 conn(0x7f568c0b1160 0x7f568c140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 --2- 172.21.15.26:0/2628918912 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 0x7f568c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568959a700 1 -- 172.21.15.26:0/2628918912 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934749238 0 0) 0x7f568c147a00 con 0x7f568c146e80 2022-07-20T02:33:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568959a700 1 -- 172.21.15.26:0/2628918912 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c003260 con 0x7f568c146e80 2022-07-20T02:33:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 -- 172.21.15.26:0/2628918912 >> v2:172.21.15.120:3300/0 conn(0x7f568c0b1160 msgr2=0x7f568c140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 --2- 172.21.15.26:0/2628918912 >> v2:172.21.15.120:3300/0 conn(0x7f568c0b1160 0x7f568c140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 -- 172.21.15.26:0/2628918912 >> v1:172.21.15.26:6789/0 conn(0x7f568c146e80 legacy=0x7f568c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 -- 172.21.15.26:0/2628918912 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0b4bc0 con 0x7f568c05cfd0 2022-07-20T02:33:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568a59c700 1 --2- 172.21.15.26:0/2628918912 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 0x7f568c1407c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5674002f70 tx=0x7f5674006df0).ready entity=mon.2 client_cookie=6f5cee913ef2e363 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568959a700 1 -- 172.21.15.26:0/2628918912 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f567400d4a0 con 0x7f568c05cfd0 2022-07-20T02:33:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568959a700 1 -- 172.21.15.26:0/2628918912 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5674000e90 con 0x7f568c05cfd0 2022-07-20T02:33:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.807+0000 7f568959a700 1 -- 172.21.15.26:0/2628918912 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f567400d7f0 con 0x7f568c05cfd0 2022-07-20T02:33:55.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- 172.21.15.26:0/2628918912 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 msgr2=0x7f568c1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- 172.21.15.26:0/2628918912 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 0x7f568c1407c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5674002f70 tx=0x7f5674006df0).stop 2022-07-20T02:33:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- 172.21.15.26:0/2628918912 shutdown_connections 2022-07-20T02:33:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- 172.21.15.26:0/2628918912 >> v2:172.21.15.120:3300/0 conn(0x7f568c0b1160 0x7f568c140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- 172.21.15.26:0/2628918912 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c05cfd0 0x7f568c1407c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- 172.21.15.26:0/2628918912 >> 172.21.15.26:0/2628918912 conn(0x7f568c066f70 msgr2=0x7f568c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:55.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- 172.21.15.26:0/2628918912 shutdown_connections 2022-07-20T02:33:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- 172.21.15.26:0/2628918912 wait complete. 2022-07-20T02:33:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 Processor -- start 2022-07-20T02:33:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- start start 2022-07-20T02:33:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 0x7f568c13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 0x7f568c156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c156880 con 0x7f568c146e80 2022-07-20T02:33:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c142d70 con 0x7f568c05cfd0 2022-07-20T02:33:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f569088e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c143010 con 0x7f568c0b1160 2022-07-20T02:33:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568a59c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 0x7f568c13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f5689d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 0x7f568c156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f5689d9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 0x7f568c156390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:35490/0 (socket says 172.21.15.26:35490) 2022-07-20T02:33:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568a59c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 0x7f568c13d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33664/0 (socket says 172.21.15.26:33664) 2022-07-20T02:33:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f5689d9b700 1 -- 172.21.15.26:0/2704885017 learned_addr learned my addr 172.21.15.26:0/2704885017 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42208/0 (socket says 172.21.15.26:42208) 2022-07-20T02:33:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 -- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 msgr2=0x7f568c156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 0x7f568c156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 -- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 msgr2=0x7f568c13d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 0x7f568c13d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.811+0000 7f568ad9d700 1 -- 172.21.15.26:0/2704885017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567400b180 con 0x7f568c146e80 2022-07-20T02:33:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f568ad9d700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f5684001490 tx=0x7f5684007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5684005ba0 con 0x7f568c146e80 2022-07-20T02:33:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684005d00 con 0x7f568c146e80 2022-07-20T02:33:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c1432b0 con 0x7f568c146e80 2022-07-20T02:33:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56840072c0 con 0x7f568c146e80 2022-07-20T02:33:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c137960 con 0x7f568c146e80 2022-07-20T02:33:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c141790 con 0x7f568c146e80 2022-07-20T02:33:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.815+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5684010070 con 0x7f568c146e80 2022-07-20T02:33:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.819+0000 7f56837fe700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 0x7f5670055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.819+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f568406b950 con 0x7f568c146e80 2022-07-20T02:33:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.819+0000 7f568a59c700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 0x7f5670055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.819+0000 7f568a59c700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 0x7f5670055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f567401a000 tx=0x7f5674014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.823+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f568c141790 con 0x7f568c146e80 2022-07-20T02:33:55.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.943+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f568c1526a0 con 0x7f568c146e80 2022-07-20T02:33:55.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f56837fe700 1 -- 172.21.15.26:0/2704885017 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v120) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5684018070 con 0x7f568c146e80 2022-07-20T02:33:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:55.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 msgr2=0x7f5670055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 0x7f5670055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f567401a000 tx=0x7f5674014000).stop 2022-07-20T02:33:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 msgr2=0x7f568c136ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f5684001490 tx=0x7f5684007d80).stop 2022-07-20T02:33:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 shutdown_connections 2022-07-20T02:33:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f568c146e80 0x7f568c136ea0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5670053a70 0x7f5670055f30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f568c0b1160 0x7f568c156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 --2- 172.21.15.26:0/2704885017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f568c05cfd0 0x7f568c13d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 >> 172.21.15.26:0/2704885017 conn(0x7f568c066f70 msgr2=0x7f568c0b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 shutdown_connections 2022-07-20T02:33:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:55.947+0000 7f569088e700 1 -- 172.21.15.26:0/2704885017 wait complete. 2022-07-20T02:33:55.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:56.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 Processor -- start 2022-07-20T02:33:56.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 -- start start 2022-07-20T02:33:56.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f06ac05c720 0x7f06ac147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0afe40 0x7f06ac147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac05d4d0 con 0x7f06ac0b1200 2022-07-20T02:33:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac0a9ff0 con 0x7f06ac05c720 2022-07-20T02:33:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.127+0000 7f06b0956700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac067e60 con 0x7f06ac0afe40 2022-07-20T02:33:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f06ac05c720 0x7f06ac147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f06ac05c720 0x7f06ac147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42212/0 (socket says 172.21.15.26:42212) 2022-07-20T02:33:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 -- 172.21.15.26:0/1166143084 learned_addr learned my addr 172.21.15.26:0/1166143084 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aa59c700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0afe40 0x7f06ac147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 -- 172.21.15.26:0/1166143084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0afe40 msgr2=0x7f06ac147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0afe40 0x7f06ac147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 -- 172.21.15.26:0/1166143084 >> v1:172.21.15.120:6789/0 conn(0x7f06ac0b1200 legacy=0x7f06ac0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 -- 172.21.15.26:0/1166143084 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac067bc0 con 0x7f06ac05c720 2022-07-20T02:33:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06aad9d700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac147550 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f06ac0aa5d0 tx=0x7f06a4009290).ready entity=mon.0 client_cookie=eb8bf61335649e3d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06a9d9b700 1 -- 172.21.15.26:0/1166143084 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06a400b860 con 0x7f06ac05c720 2022-07-20T02:33:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06a9d9b700 1 -- 172.21.15.26:0/1166143084 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a400b9c0 con 0x7f06ac05c720 2022-07-20T02:33:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06a9d9b700 1 -- 172.21.15.26:0/1166143084 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06a400bb70 con 0x7f06ac05c720 2022-07-20T02:33:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- 172.21.15.26:0/1166143084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 msgr2=0x7f06ac147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac147550 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f06ac0aa5d0 tx=0x7f06a4009290).stop 2022-07-20T02:33:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- 172.21.15.26:0/1166143084 shutdown_connections 2022-07-20T02:33:56.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0afe40 0x7f06ac147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 --2- 172.21.15.26:0/1166143084 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac147550 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- 172.21.15.26:0/1166143084 >> 172.21.15.26:0/1166143084 conn(0x7f06ac060310 msgr2=0x7f06ac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- 172.21.15.26:0/1166143084 shutdown_connections 2022-07-20T02:33:56.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- 172.21.15.26:0/1166143084 wait complete. 2022-07-20T02:33:56.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 Processor -- start 2022-07-20T02:33:56.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.131+0000 7f06b0956700 1 -- start start 2022-07-20T02:33:56.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 0x7f06ac0ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 0x7f06ac0ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0ada00 con 0x7f06ac05c720 2022-07-20T02:33:56.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0adca0 con 0x7f06ac0afe40 2022-07-20T02:33:56.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0adf40 con 0x7f06ac0b1200 2022-07-20T02:33:56.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42218/0 (socket says 172.21.15.26:42218) 2022-07-20T02:33:56.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 -- 172.21.15.26:0/429752407 learned_addr learned my addr 172.21.15.26:0/429752407 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:56.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06ab59e700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 0x7f06ac0ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 -- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 msgr2=0x7f06ac0ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aa59c700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 0x7f06ac0ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 0x7f06ac0ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 -- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 msgr2=0x7f06ac0ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 0x7f06ac0ae3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 -- 172.21.15.26:0/429752407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4009530 con 0x7f06ac05c720 2022-07-20T02:33:56.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06ab59e700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 0x7f06ac0ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:56.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aa59c700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 0x7f06ac0ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:56.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06aad9d700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f06ac0aa5d0 tx=0x7f06a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06a4021670 con 0x7f06ac05c720 2022-07-20T02:33:56.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a40217d0 con 0x7f06ac05c720 2022-07-20T02:33:56.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06a4021b30 con 0x7f06ac05c720 2022-07-20T02:33:56.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac066360 con 0x7f06ac05c720 2022-07-20T02:33:56.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac067540 con 0x7f06ac05c720 2022-07-20T02:33:56.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.135+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac0aa250 con 0x7f06ac05c720 2022-07-20T02:33:56.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.139+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f06a400f070 con 0x7f06ac05c720 2022-07-20T02:33:56.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.139+0000 7f069bfff700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 0x7f0688055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.139+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f06a406d2b0 con 0x7f06ac05c720 2022-07-20T02:33:56.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.139+0000 7f06aa59c700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 0x7f0688055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.139+0000 7f06aa59c700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 0x7f0688055f30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f06a0008000 tx=0x7f06a0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.143+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06a40440b0 con 0x7f06ac05c720 2022-07-20T02:33:56.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.267+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06ac0574e0 con 0x7f06ac05c720 2022-07-20T02:33:56.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f069bfff700 1 -- 172.21.15.26:0/429752407 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v120) v1 ==== 92+0+10 (secure 0 0 0) 0x7f06a4015020 con 0x7f06ac05c720 2022-07-20T02:33:56.273 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:56.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 msgr2=0x7f0688055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 0x7f0688055f30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f06a0008000 tx=0x7f06a0008040).stop 2022-07-20T02:33:56.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 msgr2=0x7f06ac066eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f06ac0aa5d0 tx=0x7f06a4016000).stop 2022-07-20T02:33:56.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 shutdown_connections 2022-07-20T02:33:56.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0688053a70 0x7f0688055f30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ac0b1200 0x7f06ac0ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ac0afe40 0x7f06ac0ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 --2- 172.21.15.26:0/429752407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06ac05c720 0x7f06ac066eb0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 >> 172.21.15.26:0/429752407 conn(0x7f06ac060310 msgr2=0x7f06ac146a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 shutdown_connections 2022-07-20T02:33:56.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.271+0000 7f06b0956700 1 -- 172.21.15.26:0/429752407 wait complete. 2022-07-20T02:33:56.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:56.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.395+0000 7f346359e700 1 Processor -- start 2022-07-20T02:33:56.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.395+0000 7f346359e700 1 -- start start 2022-07-20T02:33:56.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34640b06d0 0x7f346413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34640ac8e0 0x7f346413fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640b1090 con 0x7f34640a9520 2022-07-20T02:33:56.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640b0bd0 con 0x7f34640ac8e0 2022-07-20T02:33:56.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640b12c0 con 0x7f34640b06d0 2022-07-20T02:33:56.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f34640ac8e0 0x7f346413fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346259c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f34640a9520 0x7f346405c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41506/0 (socket says 172.21.15.26:41506) 2022-07-20T02:33:56.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346259c700 1 -- 172.21.15.26:0/756368827 learned_addr learned my addr 172.21.15.26:0/756368827 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:56.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f345bfff700 1 --2- 172.21.15.26:0/756368827 >> v2:172.21.15.120:3300/0 conn(0x7f34640b06d0 0x7f346413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 -- 172.21.15.26:0/756368827 >> v2:172.21.15.120:3300/0 conn(0x7f34640b06d0 msgr2=0x7f346413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:56.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 --2- 172.21.15.26:0/756368827 >> v2:172.21.15.120:3300/0 conn(0x7f34640b06d0 0x7f346413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 -- 172.21.15.26:0/756368827 >> v1:172.21.15.26:6789/0 conn(0x7f34640a9520 legacy=0x7f346405c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 -- 172.21.15.26:0/756368827 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640b1560 con 0x7f34640ac8e0 2022-07-20T02:33:56.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3462d9d700 1 --2- 172.21.15.26:0/756368827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640ac8e0 0x7f346413fbd0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f345c0032f0 tx=0x7f345c0091d0).ready entity=mon.0 client_cookie=bca79e9dcea18260 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3461d9b700 1 -- 172.21.15.26:0/756368827 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f345c00b680 con 0x7f34640ac8e0 2022-07-20T02:33:56.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3461d9b700 1 -- 172.21.15.26:0/756368827 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c00b7e0 con 0x7f34640ac8e0 2022-07-20T02:33:56.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f3461d9b700 1 -- 172.21.15.26:0/756368827 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f345c00b950 con 0x7f34640ac8e0 2022-07-20T02:33:56.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- 172.21.15.26:0/756368827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640ac8e0 msgr2=0x7f346413fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 --2- 172.21.15.26:0/756368827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640ac8e0 0x7f346413fbd0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f345c0032f0 tx=0x7f345c0091d0).stop 2022-07-20T02:33:56.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- 172.21.15.26:0/756368827 shutdown_connections 2022-07-20T02:33:56.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 --2- 172.21.15.26:0/756368827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640ac8e0 0x7f346413fbd0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 --2- 172.21.15.26:0/756368827 >> v2:172.21.15.120:3300/0 conn(0x7f34640b06d0 0x7f346413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- 172.21.15.26:0/756368827 >> 172.21.15.26:0/756368827 conn(0x7f3464060310 msgr2=0x7f34640b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- 172.21.15.26:0/756368827 shutdown_connections 2022-07-20T02:33:56.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.399+0000 7f346359e700 1 -- 172.21.15.26:0/756368827 wait complete. 2022-07-20T02:33:56.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 Processor -- start 2022-07-20T02:33:56.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 -- start start 2022-07-20T02:33:56.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34640a9520 0x7f3464139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34640ac8e0 0x7f3464066e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464138680 con 0x7f34640b06d0 2022-07-20T02:33:56.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346413f360 con 0x7f34640a9520 2022-07-20T02:33:56.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3464138f30 con 0x7f34640ac8e0 2022-07-20T02:33:56.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42236/0 (socket says 172.21.15.26:42236) 2022-07-20T02:33:56.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 -- 172.21.15.26:0/2581394840 learned_addr learned my addr 172.21.15.26:0/2581394840 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:56.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f346259c700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34640a9520 0x7f3464139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f345bfff700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34640ac8e0 0x7f3464066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 -- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34640ac8e0 msgr2=0x7f3464066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34640ac8e0 0x7f3464066e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 -- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34640a9520 msgr2=0x7f3464139670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34640a9520 0x7f3464139670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 -- 172.21.15.26:0/2581394840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009470 con 0x7f34640b06d0 2022-07-20T02:33:56.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f3462d9d700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f345c0032f0 tx=0x7f345c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f345c00b680 con 0x7f34640b06d0 2022-07-20T02:33:56.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.403+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c021630 con 0x7f34640b06d0 2022-07-20T02:33:56.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34641391d0 con 0x7f34640b06d0 2022-07-20T02:33:56.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f345c021990 con 0x7f34640b06d0 2022-07-20T02:33:56.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3464136c60 con 0x7f34640b06d0 2022-07-20T02:33:56.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f345c027020 con 0x7f34640b06d0 2022-07-20T02:33:56.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3464139360 con 0x7f34640b06d0 2022-07-20T02:33:56.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f345b7fe700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 0x7f343c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f345c06d450 con 0x7f34640b06d0 2022-07-20T02:33:56.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f346259c700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 0x7f343c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.407+0000 7f346259c700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 0x7f343c055e10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f345400d000 tx=0x7f3454009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.415+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f345c044210 con 0x7f34640b06d0 2022-07-20T02:33:56.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.543+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34640574e0 con 0x7f34640b06d0 2022-07-20T02:33:56.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.543+0000 7f345b7fe700 1 -- 172.21.15.26:0/2581394840 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+10 (secure 0 0 0) 0x7f345c00b950 con 0x7f34640b06d0 2022-07-20T02:33:56.548 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:56.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 msgr2=0x7f343c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 0x7f343c055e10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f345400d000 tx=0x7f3454009040).stop 2022-07-20T02:33:56.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 msgr2=0x7f3464155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f345c0032f0 tx=0x7f345c016000).stop 2022-07-20T02:33:56.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 shutdown_connections 2022-07-20T02:33:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f34640b06d0 0x7f3464155af0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f343c053950 0x7f343c055e10 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34640ac8e0 0x7f3464066e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 --2- 172.21.15.26:0/2581394840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34640a9520 0x7f3464139670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 >> 172.21.15.26:0/2581394840 conn(0x7f3464060310 msgr2=0x7f34640b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 shutdown_connections 2022-07-20T02:33:56.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.547+0000 7f346359e700 1 -- 172.21.15.26:0/2581394840 wait complete. 2022-07-20T02:33:56.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 Processor -- start 2022-07-20T02:33:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 -- start start 2022-07-20T02:33:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff39013f660 0x7ff39013faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3900a8cc0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39005ce10 con 0x7ff3900ac940 2022-07-20T02:33:56.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff395ee2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3900aba20 con 0x7ff39013f660 2022-07-20T02:33:56.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff38ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff39013f660 0x7ff39013faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff3956e1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff3900a9ea0 0x7ff3900aa330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41524/0 (socket says 172.21.15.26:41524) 2022-07-20T02:33:56.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.671+0000 7ff3956e1700 1 -- 172.21.15.26:0/171510708 learned_addr learned my addr 172.21.15.26:0/171510708 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:56.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38ffff700 1 -- 172.21.15.26:0/171510708 >> v1:172.21.15.120:6789/0 conn(0x7ff3900ac940 legacy=0x7ff3900acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38ffff700 1 -- 172.21.15.26:0/171510708 >> v1:172.21.15.26:6789/0 conn(0x7ff3900a9ea0 legacy=0x7ff3900aa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38ffff700 1 -- 172.21.15.26:0/171510708 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3900ab240 con 0x7ff39013f660 2022-07-20T02:33:56.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38ffff700 1 --2- 172.21.15.26:0/171510708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff39013f660 0x7ff39013faf0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7ff3800032f0 tx=0x7ff3800091d0).ready entity=mon.0 client_cookie=582e43a7a9ffe79d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38f7fe700 1 -- 172.21.15.26:0/171510708 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff380009ea0 con 0x7ff39013f660 2022-07-20T02:33:56.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38f7fe700 1 -- 172.21.15.26:0/171510708 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38000b8a0 con 0x7ff39013f660 2022-07-20T02:33:56.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff38f7fe700 1 -- 172.21.15.26:0/171510708 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff38000ba10 con 0x7ff39013f660 2022-07-20T02:33:56.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- 172.21.15.26:0/171510708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff39013f660 msgr2=0x7ff39013faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 --2- 172.21.15.26:0/171510708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff39013f660 0x7ff39013faf0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7ff3800032f0 tx=0x7ff3800091d0).stop 2022-07-20T02:33:56.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- 172.21.15.26:0/171510708 shutdown_connections 2022-07-20T02:33:56.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 --2- 172.21.15.26:0/171510708 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff39013f660 0x7ff39013faf0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- 172.21.15.26:0/171510708 >> 172.21.15.26:0/171510708 conn(0x7ff390060310 msgr2=0x7ff3900b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- 172.21.15.26:0/171510708 shutdown_connections 2022-07-20T02:33:56.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- 172.21.15.26:0/171510708 wait complete. 2022-07-20T02:33:56.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 Processor -- start 2022-07-20T02:33:56.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.675+0000 7ff395ee2700 1 -- start start 2022-07-20T02:33:56.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3900ac940 0x7ff390142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39013f660 0x7ff390142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3900ab4e0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff390140cf0 con 0x7ff39013f660 2022-07-20T02:33:56.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff390130580 con 0x7ff3900ac940 2022-07-20T02:33:56.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42246/0 (socket says 172.21.15.26:42246) 2022-07-20T02:33:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 -- 172.21.15.26:0/2051639479 learned_addr learned my addr 172.21.15.26:0/2051639479 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff3956e1700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39013f660 0x7ff390142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff38ffff700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3900ac940 0x7ff390142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 -- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3900ac940 msgr2=0x7ff390142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3900ac940 0x7ff390142820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 -- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39013f660 msgr2=0x7ff390142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39013f660 0x7ff390142d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 -- 172.21.15.26:0/2051639479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff380009470 con 0x7ff3900a9ea0 2022-07-20T02:33:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff394ee0700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7ff3900a9190 tx=0x7ff378007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff37800ce30 con 0x7ff3900a9ea0 2022-07-20T02:33:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff378007240 con 0x7ff3900a9ea0 2022-07-20T02:33:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3900ad9b0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff378006700 con 0x7ff3900a9ea0 2022-07-20T02:33:56.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff390141f10 con 0x7ff3900a9ea0 2022-07-20T02:33:56.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.679+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3901421e0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.683+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff378010030 con 0x7ff3900a9ea0 2022-07-20T02:33:56.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.683+0000 7ff38dffb700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 0x7ff374055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.683+0000 7ff38ffff700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 0x7ff374055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.683+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff37806bcc0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.683+0000 7ff38ffff700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 0x7ff374055ea0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7ff38001a000 tx=0x7ff380016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.687+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3901421e0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.815+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3901421e0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.815+0000 7ff38dffb700 1 -- 172.21.15.26:0/2051639479 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff3901421e0 con 0x7ff3900a9ea0 2022-07-20T02:33:56.825 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 msgr2=0x7ff374055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 0x7ff374055ea0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7ff38001a000 tx=0x7ff380016000).stop 2022-07-20T02:33:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 msgr2=0x7ff39013ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7ff3900a9190 tx=0x7ff378007c10).stop 2022-07-20T02:33:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 shutdown_connections 2022-07-20T02:33:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3740539e0 0x7ff374055ea0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff39013f660 0x7ff390142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3900ac940 0x7ff390142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 --2- 172.21.15.26:0/2051639479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3900a9ea0 0x7ff39013ea60 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 >> 172.21.15.26:0/2051639479 conn(0x7ff390060310 msgr2=0x7ff3900b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 shutdown_connections 2022-07-20T02:33:56.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.819+0000 7ff395ee2700 1 -- 172.21.15.26:0/2051639479 wait complete. 2022-07-20T02:33:56.838 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:56.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:56.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 Processor -- start 2022-07-20T02:33:56.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 -- start start 2022-07-20T02:33:56.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 0x7f486005fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4860061750 0x7f48600b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4860094fc0 con 0x7f48600b0d70 2022-07-20T02:33:56.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48600602b0 con 0x7f4860061750 2022-07-20T02:33:56.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f4864b33700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48600b2120 con 0x7f486005f9e0 2022-07-20T02:33:56.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.951+0000 7f485dd9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4860061750 0x7f48600b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4860061750 0x7f48600b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42254/0 (socket says 172.21.15.26:42254) 2022-07-20T02:33:56.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 -- 172.21.15.26:0/3607698753 learned_addr learned my addr 172.21.15.26:0/3607698753 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485e59c700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 0x7f486005fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 -- 172.21.15.26:0/3607698753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 msgr2=0x7f486005fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 0x7f486005fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 -- 172.21.15.26:0/3607698753 >> v1:172.21.15.120:6789/0 conn(0x7f48600b0d70 legacy=0x7f4860146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 -- 172.21.15.26:0/3607698753 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600b1e80 con 0x7f4860061750 2022-07-20T02:33:56.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485e59c700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 0x7f486005fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:33:56.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485dd9b700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4860061750 0x7f48600b0880 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f4850002f70 tx=0x7f4850006eb0).ready entity=mon.0 client_cookie=8fee9dba35b224f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:56.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485d59a700 1 -- 172.21.15.26:0/3607698753 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f485000d5c0 con 0x7f4860061750 2022-07-20T02:33:56.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485d59a700 1 -- 172.21.15.26:0/3607698753 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4850000e90 con 0x7f4860061750 2022-07-20T02:33:56.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f485d59a700 1 -- 172.21.15.26:0/3607698753 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f485000d950 con 0x7f4860061750 2022-07-20T02:33:56.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 -- 172.21.15.26:0/3607698753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4860061750 msgr2=0x7f48600b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:56.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4860061750 0x7f48600b0880 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f4850002f70 tx=0x7f4850006eb0).stop 2022-07-20T02:33:56.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 -- 172.21.15.26:0/3607698753 shutdown_connections 2022-07-20T02:33:56.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4860061750 0x7f48600b0880 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 --2- 172.21.15.26:0/3607698753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f486005f9e0 0x7f486005fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:56.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 -- 172.21.15.26:0/3607698753 >> 172.21.15.26:0/3607698753 conn(0x7f48600b5e70 msgr2=0x7f48600b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:56.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 -- 172.21.15.26:0/3607698753 shutdown_connections 2022-07-20T02:33:56.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 -- 172.21.15.26:0/3607698753 wait complete. 2022-07-20T02:33:56.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.955+0000 7f4864b33700 1 Processor -- start 2022-07-20T02:33:56.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- start start 2022-07-20T02:33:56.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4860061750 0x7f486009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 0x7f4860152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:56.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860152760 con 0x7f486005f9e0 2022-07-20T02:33:56.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860065ab0 con 0x7f48600b0d70 2022-07-20T02:33:56.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4860065d50 con 0x7f4860061750 2022-07-20T02:33:56.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:57.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42260/0 (socket says 172.21.15.26:42260) 2022-07-20T02:33:57.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 -- 172.21.15.26:0/1333079063 learned_addr learned my addr 172.21.15.26:0/1333079063 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:57.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485dd9b700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4860061750 0x7f486009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:57.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485ed9d700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 0x7f4860152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 -- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4860061750 msgr2=0x7f486009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4860061750 0x7f486009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 -- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 msgr2=0x7f4860152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:57.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 0x7f4860152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 -- 172.21.15.26:0/1333079063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485000b1e0 con 0x7f486005f9e0 2022-07-20T02:33:57.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485ed9d700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 0x7f4860152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:33:57.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f485e59c700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f4854007000 tx=0x7f4854008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:57.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854010650 con 0x7f486005f9e0 2022-07-20T02:33:57.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48540107b0 con 0x7f486005f9e0 2022-07-20T02:33:57.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4860063e10 con 0x7f486005f9e0 2022-07-20T02:33:57.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854010b10 con 0x7f486005f9e0 2022-07-20T02:33:57.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.959+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48600664c0 con 0x7f486005f9e0 2022-07-20T02:33:57.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4860095220 con 0x7f486005f9e0 2022-07-20T02:33:57.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4854010c70 con 0x7f486005f9e0 2022-07-20T02:33:57.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f484f7fe700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 0x7f483c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f485406c260 con 0x7f486005f9e0 2022-07-20T02:33:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f485dd9b700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 0x7f483c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.963+0000 7f485dd9b700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 0x7f483c055e10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f485001c000 tx=0x7f4850015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:56.967+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4854043020 con 0x7f486005f9e0 2022-07-20T02:33:57.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.131+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48600574e0 con 0x7f486005f9e0 2022-07-20T02:33:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.131+0000 7f484f7fe700 1 -- 172.21.15.26:0/1333079063 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f4854046640 con 0x7f486005f9e0 2022-07-20T02:33:57.137 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:57.137 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":61,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7495,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2095505434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2095505434},{"type":"v1","addr":"172.21.15.26:6835","nonce":2095505434}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":7504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3685430630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3685430630},{"type":"v1","addr":"172.21.15.120:6833","nonce":3685430630}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:52.235120+0000","modified":"2022-07-20T02:33:53.237719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/126332545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":126332545},{"type":"v1","addr":"172.21.15.26:6837","nonce":126332545}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T02:33:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 msgr2=0x7f483c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 0x7f483c055e10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f485001c000 tx=0x7f4850015000).stop 2022-07-20T02:33:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 msgr2=0x7f486009c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f4854007000 tx=0x7f4854008040).stop 2022-07-20T02:33:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 shutdown_connections 2022-07-20T02:33:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f483c053950 0x7f483c055e10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48600b0d70 0x7f4860152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4860061750 0x7f486009b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 --2- 172.21.15.26:0/1333079063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f486005f9e0 0x7f486009c3b0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 >> 172.21.15.26:0/1333079063 conn(0x7f48600b5e70 msgr2=0x7f486009a760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:57.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 shutdown_connections 2022-07-20T02:33:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:57.135+0000 7f4864b33700 1 -- 172.21.15.26:0/1333079063 wait complete. 2022-07-20T02:33:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 61 2022-07-20T02:33:57.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:52.235120+0000', 'modified': '2022-07-20T02:33:53.237719+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'c', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/126332545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 126332545}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 126332545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:33:57.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:33:57.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:33:57.646 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7725,"want_state":"up:active","state":"up:active","rank_uptime":5.3953322960000003,"mdsmap_epoch":61,"osdmap_epoch":120,"osdmap_epoch_barrier":119,"uptime":9.7997456510000003} 2022-07-20T02:33:57.647 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7725, "mdsmap_epoch": 61, "osdmap_epoch": 120, "osdmap_epoch_barrier": 119, "rank_uptime": 5.395332296, "state": "up:active", "uptime": 9.799745651, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:33:57.648 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:33:57.648 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:33:57.649 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:33:57.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:57.651 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:57.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:57.756 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:57.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:57.807 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:57.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:57.808 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:57.876 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:33:57.876 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:57.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:33:57.951 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:33:57.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:33:58.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:33:57.997+0000 7f208660a080 -1 init, newargv = 0x5593a131c9a0 newargc=9 2022-07-20T02:33:58.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23241]: starting ceph client 2022-07-20T02:33:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:33:58.013 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:33:58.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:58.015 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:33:58.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23241]: starting fuse 2022-07-20T02:33:58.033 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:33:58.034 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:33:58.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:58.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:33:58.036 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:33:58.037 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:33:58.038 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:33:58.039 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:33:58.060 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:58.061 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:33:58.062 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:33:58.063 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:33:58.069 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:58.069 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:33:58.070 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:33:58.071 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:33:58.072 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:33:58.073 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:58.073 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:33:58.075 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:33:58.076 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:33:58.077 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:33:58.079 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:33:58.080 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:33:58.081 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:33:58.083 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:33:58.129 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23241.asok 2022-07-20T02:33:58.134 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23241.asok 2022-07-20T02:33:58.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:58.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23241.asok status 2022-07-20T02:33:58.222 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:33:58.223 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:33:58.223 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:33:58.223 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:33:58.224 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:33:58.224 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:33:58.224 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:33:58.235 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23241", 2022-07-20T02:33:58.235 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:33:58.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:58.236 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:33:58.236 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:33:58.236 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 7884, 2022-07-20T02:33:58.236 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:33:58.237 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:33:58.237 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:33:58.237 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 7884 2022-07-20T02:33:58.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:58.237 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:58.239 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:58.239 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:58.239 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3711450689 2022-07-20T02:33:58.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:33:58.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:58.240 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:33:58.240 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:33:58.240 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:33:58.241 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 3711450689 2022-07-20T02:33:58.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:33:58.241 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.7884 172.21.15.159:0/3711450689", 2022-07-20T02:33:58.241 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/3711450689", 2022-07-20T02:33:58.242 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:33:58.242 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 61, 2022-07-20T02:33:58.242 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 120, 2022-07-20T02:33:58.242 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:33:58.243 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:33:58.243 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:33:58.243 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:33:58.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:58.245 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:33:58.245 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:58.271 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:33:58.271 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:33:58.272 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:33:58.272 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:33:58.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:33:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.439+0000 7f5f2e423700 1 Processor -- start 2022-07-20T02:33:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.439+0000 7f5f2e423700 1 -- start start 2022-07-20T02:33:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28067480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 0x7f5f28141490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28144670 con 0x7f5f280b49c0 2022-07-20T02:33:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f280b44a0 con 0x7f5f28141000 2022-07-20T02:33:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280b4f40 con 0x7f5f280b7920 2022-07-20T02:33:58.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f1ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5f280b49c0 0x7f5f28144d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41552/0 (socket says 172.21.15.26:41552) 2022-07-20T02:33:58.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f1ffff700 1 -- 172.21.15.26:0/354176518 learned_addr learned my addr 172.21.15.26:0/354176518 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:58.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2c9c0700 1 --2- 172.21.15.26:0/354176518 >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 0x7f5f28141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 --2- 172.21.15.26:0/354176518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f277fe700 1 -- 172.21.15.26:0/354176518 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713005721 0 0) 0x7f5f28144670 con 0x7f5f280b49c0 2022-07-20T02:33:58.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f277fe700 1 -- 172.21.15.26:0/354176518 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003260 con 0x7f5f280b49c0 2022-07-20T02:33:58.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 -- 172.21.15.26:0/354176518 >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 msgr2=0x7f5f28141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:58.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 --2- 172.21.15.26:0/354176518 >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 0x7f5f28141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 -- 172.21.15.26:0/354176518 >> v1:172.21.15.26:6789/0 conn(0x7f5f280b49c0 legacy=0x7f5f28144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 -- 172.21.15.26:0/354176518 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f281448a0 con 0x7f5f280b7920 2022-07-20T02:33:58.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2c9c0700 1 --2- 172.21.15.26:0/354176518 >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 0x7f5f28141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:58.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f27fff700 1 --2- 172.21.15.26:0/354176518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28067480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5f18002f70 tx=0x7f5f18006df0).ready entity=mon.2 client_cookie=7aeea383a6a99680 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f277fe700 1 -- 172.21.15.26:0/354176518 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f1800d4a0 con 0x7f5f280b7920 2022-07-20T02:33:58.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f277fe700 1 -- 172.21.15.26:0/354176518 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18000e90 con 0x7f5f280b7920 2022-07-20T02:33:58.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f277fe700 1 -- 172.21.15.26:0/354176518 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f1800d7f0 con 0x7f5f280b7920 2022-07-20T02:33:58.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- 172.21.15.26:0/354176518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 msgr2=0x7f5f28067480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 --2- 172.21.15.26:0/354176518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28067480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5f18002f70 tx=0x7f5f18006df0).stop 2022-07-20T02:33:58.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- 172.21.15.26:0/354176518 shutdown_connections 2022-07-20T02:33:58.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 --2- 172.21.15.26:0/354176518 >> v2:172.21.15.120:3300/0 conn(0x7f5f28141000 0x7f5f28141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 --2- 172.21.15.26:0/354176518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28067480 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- 172.21.15.26:0/354176518 >> 172.21.15.26:0/354176518 conn(0x7f5f28064b10 msgr2=0x7f5f28064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:58.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- 172.21.15.26:0/354176518 shutdown_connections 2022-07-20T02:33:58.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.443+0000 7f5f2e423700 1 -- 172.21.15.26:0/354176518 wait complete. 2022-07-20T02:33:58.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 Processor -- start 2022-07-20T02:33:58.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 -- start start 2022-07-20T02:33:58.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f280b49c0 0x7f5f280aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280ab6c0 con 0x7f5f28141000 2022-07-20T02:33:58.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28143530 con 0x7f5f280b49c0 2022-07-20T02:33:58.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2e423700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f281437d0 con 0x7f5f280b7920 2022-07-20T02:33:58.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2c9c0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2c9c0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42278/0 (socket says 172.21.15.26:42278) 2022-07-20T02:33:58.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f2c9c0700 1 -- 172.21.15.26:0/995730298 learned_addr learned my addr 172.21.15.26:0/995730298 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:58.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f1ffff700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.447+0000 7f5f27fff700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f280b49c0 0x7f5f280aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 -- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 msgr2=0x7f5f28142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 -- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f280b49c0 msgr2=0x7f5f280aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f280b49c0 0x7f5f280aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 -- 172.21.15.26:0/995730298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f20009470 con 0x7f5f28141000 2022-07-20T02:33:58.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2c9c0700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f5f2000c000 tx=0x7f5f200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2000ee60 con 0x7f5f28141000 2022-07-20T02:33:58.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1800b180 con 0x7f5f28141000 2022-07-20T02:33:58.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f280ab8b0 con 0x7f5f28141000 2022-07-20T02:33:58.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2000c850 con 0x7f5f28141000 2022-07-20T02:33:58.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2000cbb0 con 0x7f5f28141000 2022-07-20T02:33:58.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00003410 con 0x7f5f28141000 2022-07-20T02:33:58.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f2000cd10 con 0x7f5f28141000 2022-07-20T02:33:58.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 0x7f5f0c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5f2006c780 con 0x7f5f28141000 2022-07-20T02:33:58.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.451+0000 7f5f27fff700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 0x7f5f0c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.455+0000 7f5f27fff700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 0x7f5f0c055e10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5f1801a000 tx=0x7f5f18014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.463+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f200434a0 con 0x7f5f28141000 2022-07-20T02:33:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.591+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f000036c0 con 0x7f5f28141000 2022-07-20T02:33:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f25ffb700 1 -- 172.21.15.26:0/995730298 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f5f20016020 con 0x7f5f28141000 2022-07-20T02:33:58.597 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:33:58.597 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":120,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:33:53.238412+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:33:50.826732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T02:33:51.226718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:33:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 msgr2=0x7f5f0c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 0x7f5f0c055e10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5f1801a000 tx=0x7f5f18014000).stop 2022-07-20T02:33:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 msgr2=0x7f5f281428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f5f2000c000 tx=0x7f5f200091d0).stop 2022-07-20T02:33:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 shutdown_connections 2022-07-20T02:33:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5f28141000 0x7f5f281428f0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5f0c053950 0x7f5f0c055e10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f280b7920 0x7f5f28142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 --2- 172.21.15.26:0/995730298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f280b49c0 0x7f5f280aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 >> 172.21.15.26:0/995730298 conn(0x7f5f28064b10 msgr2=0x7f5f280b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 shutdown_connections 2022-07-20T02:33:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.595+0000 7f5f2e423700 1 -- 172.21.15.26:0/995730298 wait complete. 2022-07-20T02:33:58.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:33:50.826732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:33:51.226718+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:33:58.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:33:58.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 Processor -- start 2022-07-20T02:33:58.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 -- start start 2022-07-20T02:33:58.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f4060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05f4095b10 0x7f05f4095f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f4067dc0 con 0x7f05f40a95a0 2022-07-20T02:33:58.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f40b3640 con 0x7f05f4095b10 2022-07-20T02:33:58.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa957700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40615c0 con 0x7f05f40b3260 2022-07-20T02:33:58.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05f4095b10 0x7f05f4095f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05f9955700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f4060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05f4095b10 0x7f05f4095f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33744/0 (socket says 172.21.15.26:33744) 2022-07-20T02:33:58.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 -- 172.21.15.26:0/1888391847 learned_addr learned my addr 172.21.15.26:0/1888391847 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:58.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 -- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 msgr2=0x7f05f4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:58.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 --2- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f4060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 -- 172.21.15.26:0/1888391847 >> v1:172.21.15.120:6789/0 conn(0x7f05f40a95a0 legacy=0x7f05f40676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 -- 172.21.15.26:0/1888391847 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f4061860 con 0x7f05f4095b10 2022-07-20T02:33:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05fa156700 1 --2- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4095f30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f05ec0032f0 tx=0x7f05ec009290).ready entity=mon.1 client_cookie=ef90b689cd4129c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05f8953700 1 -- 172.21.15.26:0/1888391847 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec00b8a0 con 0x7f05f4095b10 2022-07-20T02:33:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05f8953700 1 -- 172.21.15.26:0/1888391847 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec00ba00 con 0x7f05f4095b10 2022-07-20T02:33:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.723+0000 7f05f8953700 1 -- 172.21.15.26:0/1888391847 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec00bb70 con 0x7f05f4095b10 2022-07-20T02:33:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 msgr2=0x7f05f4095f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4095f30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f05ec0032f0 tx=0x7f05ec009290).stop 2022-07-20T02:33:58.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- 172.21.15.26:0/1888391847 shutdown_connections 2022-07-20T02:33:58.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4095f30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- 172.21.15.26:0/1888391847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f4060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- 172.21.15.26:0/1888391847 >> 172.21.15.26:0/1888391847 conn(0x7f05f40b67f0 msgr2=0x7f05f40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:58.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- 172.21.15.26:0/1888391847 shutdown_connections 2022-07-20T02:33:58.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- 172.21.15.26:0/1888391847 wait complete. 2022-07-20T02:33:58.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 Processor -- start 2022-07-20T02:33:58.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- start start 2022-07-20T02:33:58.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f41535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40958b0 con 0x7f05f40a95a0 2022-07-20T02:33:58.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4064ca0 con 0x7f05f4095b10 2022-07-20T02:33:58.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa957700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40afdf0 con 0x7f05f40b3260 2022-07-20T02:33:58.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05f9154700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05f9955700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05f9955700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33748/0 (socket says 172.21.15.26:33748) 2022-07-20T02:33:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05fa156700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f41535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9955700 1 -- 172.21.15.26:0/964814932 learned_addr learned my addr 172.21.15.26:0/964814932 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.727+0000 7f05f9154700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42290/0 (socket says 172.21.15.26:42290) 2022-07-20T02:33:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 -- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 msgr2=0x7f05f41535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f41535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 -- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 msgr2=0x7f05f4147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 -- 172.21.15.26:0/964814932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec009530 con 0x7f05f40a95a0 2022-07-20T02:33:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05f9154700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f05f4069b90 tx=0x7f05e4007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05e4012400 con 0x7f05f40a95a0 2022-07-20T02:33:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e4012560 con 0x7f05f40a95a0 2022-07-20T02:33:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f40b0090 con 0x7f05f40a95a0 2022-07-20T02:33:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05e40128c0 con 0x7f05f40a95a0 2022-07-20T02:33:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f4153ad0 con 0x7f05f40a95a0 2022-07-20T02:33:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f40afbb0 con 0x7f05f40a95a0 2022-07-20T02:33:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.731+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f05e4019030 con 0x7f05f40a95a0 2022-07-20T02:33:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.735+0000 7f05e2ffd700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 0x7f05d4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.735+0000 7f05f9955700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 0x7f05d4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.735+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f05e406b290 con 0x7f05f40a95a0 2022-07-20T02:33:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.735+0000 7f05f9955700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 0x7f05d4055e10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f05f000b000 tx=0x7f05f0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.739+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05f40afbb0 con 0x7f05f40a95a0 2022-07-20T02:33:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.863+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f05f40afbb0 con 0x7f05f40a95a0 2022-07-20T02:33:58.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.863+0000 7f05e2ffd700 1 -- 172.21.15.26:0/964814932 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v120) v1 ==== 110+0+10 (secure 0 0 0) 0x7f05f40afbb0 con 0x7f05f40a95a0 2022-07-20T02:33:58.892 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:33:58.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 msgr2=0x7f05d4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 0x7f05d4055e10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f05f000b000 tx=0x7f05f0006df0).stop 2022-07-20T02:33:58.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 msgr2=0x7f05f40653a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:58.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f05f4069b90 tx=0x7f05e4007b90).stop 2022-07-20T02:33:58.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 shutdown_connections 2022-07-20T02:33:58.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f05f40a95a0 0x7f05f40653a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f05d4053950 0x7f05d4055e10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05f40b3260 0x7f05f41535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 --2- 172.21.15.26:0/964814932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05f4095b10 0x7f05f4147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:58.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 >> 172.21.15.26:0/964814932 conn(0x7f05f40b67f0 msgr2=0x7f05f40990b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:58.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 shutdown_connections 2022-07-20T02:33:58.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:58.867+0000 7f05fa957700 1 -- 172.21.15.26:0/964814932 wait complete. 2022-07-20T02:33:58.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:33:59.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 Processor -- start 2022-07-20T02:33:59.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 -- start start 2022-07-20T02:33:59.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0618a0 0x7fa34c147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa34c0b22d0 0x7fa34c143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c061c80 con 0x7fa34c143d60 2022-07-20T02:33:59.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c1441f0 con 0x7fa34c0b22d0 2022-07-20T02:33:59.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa352705700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0b2810 con 0x7fa34c0618a0 2022-07-20T02:33:59.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa351f04700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa34c143d60 0x7fa34c0b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41578/0 (socket says 172.21.15.26:41578) 2022-07-20T02:33:59.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa34c0b22d0 0x7fa34c143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa351f04700 1 -- 172.21.15.26:0/1667359705 learned_addr learned my addr 172.21.15.26:0/1667359705 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:59.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa351703700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0618a0 0x7fa34c147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091701924 0 0) 0x7fa34c061c80 con 0x7fa34c143d60 2022-07-20T02:33:59.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa330003260 con 0x7fa34c143d60 2022-07-20T02:33:59.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334402920 0 0) 0x7fa330003260 con 0x7fa34c143d60 2022-07-20T02:33:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 -- 172.21.15.26:0/1667359705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0618a0 msgr2=0x7fa34c147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0618a0 0x7fa34c147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 -- 172.21.15.26:0/1667359705 >> v1:172.21.15.26:6789/0 conn(0x7fa34c143d60 legacy=0x7fa34c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 -- 172.21.15.26:0/1667359705 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c0b2ab0 con 0x7fa34c0b22d0 2022-07-20T02:33:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa350f02700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0b22d0 0x7fa34c143870 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fa344001b80 tx=0x7fa340006df0).ready entity=mon.0 client_cookie=c7f868126b12ea75 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3440019e0 con 0x7fa34c0b22d0 2022-07-20T02:33:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340000e90 con 0x7fa34c0b22d0 2022-07-20T02:33:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.015+0000 7fa33ffff700 1 -- 172.21.15.26:0/1667359705 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa340012460 con 0x7fa34c0b22d0 2022-07-20T02:33:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- 172.21.15.26:0/1667359705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0b22d0 msgr2=0x7fa34c143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0b22d0 0x7fa34c143870 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fa344001b80 tx=0x7fa340006df0).stop 2022-07-20T02:33:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- 172.21.15.26:0/1667359705 shutdown_connections 2022-07-20T02:33:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0b22d0 0x7fa34c143870 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- 172.21.15.26:0/1667359705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0618a0 0x7fa34c147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- 172.21.15.26:0/1667359705 >> 172.21.15.26:0/1667359705 conn(0x7fa34c066f70 msgr2=0x7fa34c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- 172.21.15.26:0/1667359705 shutdown_connections 2022-07-20T02:33:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- 172.21.15.26:0/1667359705 wait complete. 2022-07-20T02:33:59.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 Processor -- start 2022-07-20T02:33:59.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- start start 2022-07-20T02:33:59.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0b22d0 0x7fa34c13fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c143d60 0x7fa34c0b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c1402a0 con 0x7fa34c0618a0 2022-07-20T02:33:59.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c13dd50 con 0x7fa34c0b22d0 2022-07-20T02:33:59.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa352705700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0b5210 con 0x7fa34c143d60 2022-07-20T02:33:59.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.019+0000 7fa351703700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42302/0 (socket says 172.21.15.26:42302) 2022-07-20T02:33:59.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 -- 172.21.15.26:0/1765882898 learned_addr learned my addr 172.21.15.26:0/1765882898 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:59.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa350f02700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0b22d0 0x7fa34c13fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351f04700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c143d60 0x7fa34c0b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 -- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c143d60 msgr2=0x7fa34c0b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c143d60 0x7fa34c0b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 -- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0b22d0 msgr2=0x7fa34c13fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0b22d0 0x7fa34c13fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 -- 172.21.15.26:0/1765882898 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34000b180 con 0x7fa34c0618a0 2022-07-20T02:33:59.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa351703700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7fa34800c000 tx=0x7fa348006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa348010780 con 0x7fa34c0618a0 2022-07-20T02:33:59.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3480108e0 con 0x7fa34c0618a0 2022-07-20T02:33:59.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c0b5950 con 0x7fa34c0618a0 2022-07-20T02:33:59.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa348010c40 con 0x7fa34c0618a0 2022-07-20T02:33:59.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c0af960 con 0x7fa34c0618a0 2022-07-20T02:33:59.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c0b5ae0 con 0x7fa34c0618a0 2022-07-20T02:33:59.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa34801c460 con 0x7fa34c0618a0 2022-07-20T02:33:59.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 0x7fa334055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa34806c500 con 0x7fa34c0618a0 2022-07-20T02:33:59.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.023+0000 7fa350f02700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 0x7fa334055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.027+0000 7fa350f02700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 0x7fa334055f30 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fa340018000 tx=0x7fa340014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.031+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa348043260 con 0x7fa34c0618a0 2022-07-20T02:33:59.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.155+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa34c0574e0 con 0x7fa34c0618a0 2022-07-20T02:33:59.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.155+0000 7fa33e7fc700 1 -- 172.21.15.26:0/1765882898 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v120) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa348046880 con 0x7fa34c0618a0 2022-07-20T02:33:59.164 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 msgr2=0x7fa334055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 0x7fa334055f30 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fa340018000 tx=0x7fa340014000).stop 2022-07-20T02:33:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 msgr2=0x7fa34c146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7fa34800c000 tx=0x7fa348006df0).stop 2022-07-20T02:33:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 shutdown_connections 2022-07-20T02:33:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa334053a70 0x7fa334055f30 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c143d60 0x7fa34c0b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0b22d0 0x7fa34c13fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 --2- 172.21.15.26:0/1765882898 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa34c0618a0 0x7fa34c146f20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 >> 172.21.15.26:0/1765882898 conn(0x7fa34c066f70 msgr2=0x7fa34c12b440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 shutdown_connections 2022-07-20T02:33:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.159+0000 7fa352705700 1 -- 172.21.15.26:0/1765882898 wait complete. 2022-07-20T02:33:59.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:33:59.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 Processor -- start 2022-07-20T02:33:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 -- start start 2022-07-20T02:33:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f375405cfd0 0x7f3754143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3754147000 0x7f37540b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37541474e0 con 0x7f3754060f00 2022-07-20T02:33:59.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375405d3b0 con 0x7f3754147000 2022-07-20T02:33:59.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759e32700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37541431e0 con 0x7f375405cfd0 2022-07-20T02:33:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759631700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3754060f00 0x7f375413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41592/0 (socket says 172.21.15.26:41592) 2022-07-20T02:33:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.319+0000 7f3759631700 1 -- 172.21.15.26:0/614939266 learned_addr learned my addr 172.21.15.26:0/614939266 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 --2- 172.21.15.26:0/614939266 >> v2:172.21.15.120:3300/0 conn(0x7f3754147000 0x7f37540b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3758e30700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f375405cfd0 0x7f3754143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146198452 0 0) 0x7f37541474e0 con 0x7f3754060f00 2022-07-20T02:33:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3734003260 con 0x7f3754060f00 2022-07-20T02:33:59.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 -- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f375405cfd0 msgr2=0x7f3754143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:59.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f375405cfd0 0x7f3754143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 -- 172.21.15.26:0/614939266 >> v1:172.21.15.26:6789/0 conn(0x7f3754060f00 legacy=0x7f375413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3920554979 0 0) 0x7f3734003260 con 0x7f3754060f00 2022-07-20T02:33:59.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 -- 172.21.15.26:0/614939266 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754143480 con 0x7f3754147000 2022-07-20T02:33:59.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3753fff700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754147000 0x7f37540b4730 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f373c002f70 tx=0x7f373c006df0).ready entity=mon.1 client_cookie=57cdaf83a246a8d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f373c00d4a0 con 0x7f3754147000 2022-07-20T02:33:59.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c000e90 con 0x7f3754147000 2022-07-20T02:33:59.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f37537fe700 1 -- 172.21.15.26:0/614939266 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f373c00d7f0 con 0x7f3754147000 2022-07-20T02:33:59.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 -- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754147000 msgr2=0x7f37540b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754147000 0x7f37540b4730 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f373c002f70 tx=0x7f373c006df0).stop 2022-07-20T02:33:59.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 -- 172.21.15.26:0/614939266 shutdown_connections 2022-07-20T02:33:59.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754147000 0x7f37540b4730 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 --2- 172.21.15.26:0/614939266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f375405cfd0 0x7f3754143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 -- 172.21.15.26:0/614939266 >> 172.21.15.26:0/614939266 conn(0x7f37540620c0 msgr2=0x7f375405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 -- 172.21.15.26:0/614939266 shutdown_connections 2022-07-20T02:33:59.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.323+0000 7f3759e32700 1 -- 172.21.15.26:0/614939266 wait complete. 2022-07-20T02:33:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 Processor -- start 2022-07-20T02:33:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- start start 2022-07-20T02:33:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754060f00 0x7f37540b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3754147000 0x7f37541460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540b3a80 con 0x7f37540aebc0 2022-07-20T02:33:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540ba7a0 con 0x7f3754060f00 2022-07-20T02:33:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37540ae690 con 0x7f3754147000 2022-07-20T02:33:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3753fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3754147000 0x7f37541460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3758e30700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754060f00 0x7f37540b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42320/0 (socket says 172.21.15.26:42320) 2022-07-20T02:33:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 -- 172.21.15.26:0/3136833500 learned_addr learned my addr 172.21.15.26:0/3136833500 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 -- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3754147000 msgr2=0x7f37541460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3754147000 0x7f37541460a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 -- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754060f00 msgr2=0x7f37540b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754060f00 0x7f37540b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 -- 172.21.15.26:0/3136833500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c00b180 con 0x7f37540aebc0 2022-07-20T02:33:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759631700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7f37540b3840 tx=0x7f374c007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374c004ab0 con 0x7f37540aebc0 2022-07-20T02:33:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374c004c10 con 0x7f37540aebc0 2022-07-20T02:33:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37540b02f0 con 0x7f37540aebc0 2022-07-20T02:33:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374c007170 con 0x7f37540aebc0 2022-07-20T02:33:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.327+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3754152160 con 0x7f37540aebc0 2022-07-20T02:33:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3754143720 con 0x7f37540aebc0 2022-07-20T02:33:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f374c01b070 con 0x7f37540aebc0 2022-07-20T02:33:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3751ffb700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 0x7f3738055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3758e30700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 0x7f3738055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f374c06baf0 con 0x7f37540aebc0 2022-07-20T02:33:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.331+0000 7f3758e30700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 0x7f3738055f30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f374400b000 tx=0x7f3744006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.339+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f374c0427a0 con 0x7f37540aebc0 2022-07-20T02:33:59.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37540574e0 con 0x7f37540aebc0 2022-07-20T02:33:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3751ffb700 1 -- 172.21.15.26:0/3136833500 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+10 (secure 0 0 0) 0x7f374c045dc0 con 0x7f37540aebc0 2022-07-20T02:33:59.556 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 msgr2=0x7f3738055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 0x7f3738055f30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f374400b000 tx=0x7f3744006df0).stop 2022-07-20T02:33:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 msgr2=0x7f37540b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7f37540b3840 tx=0x7f374c007c20).stop 2022-07-20T02:33:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 shutdown_connections 2022-07-20T02:33:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37540aebc0 0x7f37540b2eb0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3738053a70 0x7f3738055f30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3754147000 0x7f37541460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 --2- 172.21.15.26:0/3136833500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3754060f00 0x7f37540b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 >> 172.21.15.26:0/3136833500 conn(0x7f375405cfd0 msgr2=0x7f37540b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.467+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 shutdown_connections 2022-07-20T02:33:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.471+0000 7f3759e32700 1 -- 172.21.15.26:0/3136833500 wait complete. 2022-07-20T02:33:59.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:33:59.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.683+0000 7f2dd63ae700 1 Processor -- start 2022-07-20T02:33:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.683+0000 7f2dd63ae700 1 -- start start 2022-07-20T02:33:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2dd00b5b70 0x7f2dd01431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00ba060 con 0x7f2dd00b9c80 2022-07-20T02:33:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd0061f80 con 0x7f2dd0062710 2022-07-20T02:33:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd012bee0 con 0x7f2dd00b5b70 2022-07-20T02:33:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd494b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2dd00b5b70 0x7f2dd01431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dcffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2dd00b9c80 0x7f2dd00b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41604/0 (socket says 172.21.15.26:41604) 2022-07-20T02:33:59.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dcffff700 1 -- 172.21.15.26:0/1158475691 learned_addr learned my addr 172.21.15.26:0/1158475691 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:59.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd494b700 1 -- 172.21.15.26:0/1158475691 >> v1:172.21.15.120:6789/0 conn(0x7f2dd0062710 legacy=0x7f2dd01403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd494b700 1 -- 172.21.15.26:0/1158475691 >> v1:172.21.15.26:6789/0 conn(0x7f2dd00b9c80 legacy=0x7f2dd00b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd494b700 1 -- 172.21.15.26:0/1158475691 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0062220 con 0x7f2dd00b5b70 2022-07-20T02:33:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd494b700 1 --2- 172.21.15.26:0/1158475691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b5b70 0x7f2dd01431a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f2dc80032f0 tx=0x7f2dc80091d0).ready entity=mon.0 client_cookie=addaa59123f1586 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dceffd700 1 -- 172.21.15.26:0/1158475691 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc8009ea0 con 0x7f2dd00b5b70 2022-07-20T02:33:59.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dceffd700 1 -- 172.21.15.26:0/1158475691 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc800b8a0 con 0x7f2dd00b5b70 2022-07-20T02:33:59.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dceffd700 1 -- 172.21.15.26:0/1158475691 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc800ba10 con 0x7f2dd00b5b70 2022-07-20T02:33:59.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- 172.21.15.26:0/1158475691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b5b70 msgr2=0x7f2dd01431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/1158475691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b5b70 0x7f2dd01431a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f2dc80032f0 tx=0x7f2dc80091d0).stop 2022-07-20T02:33:59.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- 172.21.15.26:0/1158475691 shutdown_connections 2022-07-20T02:33:59.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/1158475691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b5b70 0x7f2dd01431a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- 172.21.15.26:0/1158475691 >> 172.21.15.26:0/1158475691 conn(0x7f2dd0066f70 msgr2=0x7f2dd005faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- 172.21.15.26:0/1158475691 shutdown_connections 2022-07-20T02:33:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.687+0000 7f2dd63ae700 1 -- 172.21.15.26:0/1158475691 wait complete. 2022-07-20T02:33:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.691+0000 7f2dd63ae700 1 Processor -- start 2022-07-20T02:33:59.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.691+0000 7f2dd63ae700 1 -- start start 2022-07-20T02:33:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 0x7f2dd00b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dd00b5b70 0x7f2dd00b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00b7740 con 0x7f2dd00b9c80 2022-07-20T02:33:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00b2c70 con 0x7f2dd0062710 2022-07-20T02:33:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00b2730 con 0x7f2dd00b5b70 2022-07-20T02:33:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42334/0 (socket says 172.21.15.26:42334) 2022-07-20T02:33:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 0x7f2dd00b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 -- 172.21.15.26:0/2524477443 learned_addr learned my addr 172.21.15.26:0/2524477443 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcf7fe700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dd00b5b70 0x7f2dd00b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 -- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dd00b5b70 msgr2=0x7f2dd00b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dd00b5b70 0x7f2dd00b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 -- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 msgr2=0x7f2dd00b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 0x7f2dd00b7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 -- 172.21.15.26:0/2524477443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8009470 con 0x7f2dd00b9c80 2022-07-20T02:33:59.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcffff700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 0x7f2dd00b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:33:59.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd494b700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f2dc80032f0 tx=0x7f2dc8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc8009ea0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc801f3f0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc801f750 con 0x7f2dd00b9c80 2022-07-20T02:33:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd00b29d0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd013fdc0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.695+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd00b6650 con 0x7f2dd00b9c80 2022-07-20T02:33:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.699+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2dc8026030 con 0x7f2dd00b9c80 2022-07-20T02:33:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.699+0000 7f2dcd7fa700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 0x7f2db0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.699+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2dc806cca0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.699+0000 7f2dcffff700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 0x7f2db0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.699+0000 7f2dcffff700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 0x7f2db0055f60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f2dc400d000 tx=0x7f2dc4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.707+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2dc8071070 con 0x7f2dd00b9c80 2022-07-20T02:33:59.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2dd00574e0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dcd7fa700 1 -- 172.21.15.26:0/2524477443 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2dc80240a0 con 0x7f2dd00b9c80 2022-07-20T02:33:59.849 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:33:59.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 msgr2=0x7f2db0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 0x7f2db0055f60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f2dc400d000 tx=0x7f2dc4009040).stop 2022-07-20T02:33:59.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 msgr2=0x7f2dd00b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f2dc80032f0 tx=0x7f2dc8016000).stop 2022-07-20T02:33:59.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 shutdown_connections 2022-07-20T02:33:59.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2dd00b9c80 0x7f2dd00b1ed0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2db0053aa0 0x7f2db0055f60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dd00b5b70 0x7f2dd00b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 --2- 172.21.15.26:0/2524477443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd0062710 0x7f2dd00b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.831+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 >> 172.21.15.26:0/2524477443 conn(0x7f2dd0066f70 msgr2=0x7f2dd0142650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.835+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 shutdown_connections 2022-07-20T02:33:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.835+0000 7f2dd63ae700 1 -- 172.21.15.26:0/2524477443 wait complete. 2022-07-20T02:33:59.855 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:33:59.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:33:59.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.959+0000 7ff5bb3ee700 1 Processor -- start 2022-07-20T02:33:59.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.959+0000 7ff5bb3ee700 1 -- start start 2022-07-20T02:33:59.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5b4143070 0x7ff5b41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b41407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b4146750 con 0x7ff5b40b5d50 2022-07-20T02:33:59.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b4067ba0 con 0x7ff5b4143070 2022-07-20T02:33:59.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b41469f0 con 0x7ff5b4140320 2022-07-20T02:33:59.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5ba3ec700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff5b40b5d50 0x7ff5b4067490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41618/0 (socket says 172.21.15.26:41618) 2022-07-20T02:33:59.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5ba3ec700 1 -- 172.21.15.26:0/3178545607 learned_addr learned my addr 172.21.15.26:0/3178545607 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:33:59.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/3178545607 >> v2:172.21.15.120:3300/0 conn(0x7ff5b4143070 0x7ff5b41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5babed700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b41407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b93ea700 1 -- 172.21.15.26:0/3178545607 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967889054 0 0) 0x7ff5b4146750 con 0x7ff5b40b5d50 2022-07-20T02:33:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b93ea700 1 -- 172.21.15.26:0/3178545607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c003260 con 0x7ff5b40b5d50 2022-07-20T02:33:59.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 -- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 msgr2=0x7ff5b41407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:33:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b41407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 -- 172.21.15.26:0/3178545607 >> v1:172.21.15.26:6789/0 conn(0x7ff5b40b5d50 legacy=0x7ff5b4067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 -- 172.21.15.26:0/3178545607 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b4146c90 con 0x7ff5b4143070 2022-07-20T02:33:59.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b41434c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7ff5b00032f0 tx=0x7ff5b00091d0).ready entity=mon.1 client_cookie=51447b4208dfc109 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b93ea700 1 -- 172.21.15.26:0/3178545607 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5b000b600 con 0x7ff5b4143070 2022-07-20T02:33:59.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b93ea700 1 -- 172.21.15.26:0/3178545607 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000b760 con 0x7ff5b4143070 2022-07-20T02:33:59.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5b93ea700 1 -- 172.21.15.26:0/3178545607 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5b000b8d0 con 0x7ff5b4143070 2022-07-20T02:33:59.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 msgr2=0x7ff5b41434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b41434c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7ff5b00032f0 tx=0x7ff5b00091d0).stop 2022-07-20T02:33:59.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/3178545607 shutdown_connections 2022-07-20T02:33:59.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b41407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/3178545607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b41434c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/3178545607 >> 172.21.15.26:0/3178545607 conn(0x7ff5b4060310 msgr2=0x7ff5b40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:33:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/3178545607 shutdown_connections 2022-07-20T02:33:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.963+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/3178545607 wait complete. 2022-07-20T02:33:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.967+0000 7ff5bb3ee700 1 Processor -- start 2022-07-20T02:33:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.967+0000 7ff5bb3ee700 1 -- start start 2022-07-20T02:33:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b4155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:33:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40ac460 con 0x7ff5b40b5d50 2022-07-20T02:33:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b4156340 con 0x7ff5b4143070 2022-07-20T02:33:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b4146530 con 0x7ff5b4140320 2022-07-20T02:33:59.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42344/0 (socket says 172.21.15.26:42344) 2022-07-20T02:33:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 -- 172.21.15.26:0/332350215 learned_addr learned my addr 172.21.15.26:0/332350215 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:33:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5babed700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b4155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:33:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 -- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 msgr2=0x7ff5b4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 -- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 msgr2=0x7ff5b4155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:33:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b4155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:33:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 -- 172.21.15.26:0/332350215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b0009470 con 0x7ff5b40b5d50 2022-07-20T02:33:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5ba3ec700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7ff5a4001110 tx=0x7ff5a4007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:33:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5a400ce20 con 0x7ff5b40b5d50 2022-07-20T02:33:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a40072e0 con 0x7ff5b40b5d50 2022-07-20T02:33:59.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5a4005800 con 0x7ff5b40b5d50 2022-07-20T02:33:59.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b40ac1c0 con 0x7ff5b40b5d50 2022-07-20T02:33:59.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b40abd10 con 0x7ff5b40b5d50 2022-07-20T02:33:59.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b4146f30 con 0x7ff5b40b5d50 2022-07-20T02:33:59.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff5a401a070 con 0x7ff5b40b5d50 2022-07-20T02:34:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 0x7ff594055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.971+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff5a406b820 con 0x7ff5b40b5d50 2022-07-20T02:34:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.975+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 0x7ff594055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:00.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.975+0000 7ff5b9beb700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 0x7ff594055e40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7ff5b001b000 tx=0x7ff5b0016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:00.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:33:59.979+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff5a4042580 con 0x7ff5b40b5d50 2022-07-20T02:34:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.139+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b40574e0 con 0x7ff5b40b5d50 2022-07-20T02:34:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5a37fe700 1 -- 172.21.15.26:0/332350215 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2495 (secure 0 0 0) 0x7ff5a4045ba0 con 0x7ff5b40b5d50 2022-07-20T02:34:00.148 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:00.148 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":61,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7495,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/2095505434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2095505434},{"type":"v1","addr":"172.21.15.26:6835","nonce":2095505434}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":7504,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3685430630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3685430630},{"type":"v1","addr":"172.21.15.120:6833","nonce":3685430630}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:33:52.235120+0000","modified":"2022-07-20T02:33:53.237719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/126332545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":126332545},{"type":"v1","addr":"172.21.15.26:6837","nonce":126332545}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T02:34:00.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 msgr2=0x7ff594055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:00.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 0x7ff594055e40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7ff5b001b000 tx=0x7ff5b0016000).stop 2022-07-20T02:34:00.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 msgr2=0x7ff5b413b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:00.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7ff5a4001110 tx=0x7ff5a4007b20).stop 2022-07-20T02:34:00.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 shutdown_connections 2022-07-20T02:34:00.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff594053980 0x7ff594055e40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:00.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5b4143070 0x7ff5b4155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:00.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5b4140320 0x7ff5b4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:00.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 --2- 172.21.15.26:0/332350215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5b40b5d50 0x7ff5b413b740 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:00.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 >> 172.21.15.26:0/332350215 conn(0x7ff5b4060310 msgr2=0x7ff5b413ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:00.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 shutdown_connections 2022-07-20T02:34:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:00.143+0000 7ff5bb3ee700 1 -- 172.21.15.26:0/332350215 wait complete. 2022-07-20T02:34:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 61 2022-07-20T02:34:00.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:33:52.235120+0000', 'modified': '2022-07-20T02:33:53.237719+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'c', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/126332545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 126332545}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 126332545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:34:00.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:34:00.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:34:00.624 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":7725,"want_state":"up:active","state":"up:active","rank_uptime":8.3616459929999998,"mdsmap_epoch":61,"osdmap_epoch":120,"osdmap_epoch_barrier":119,"uptime":12.766060995} 2022-07-20T02:34:00.624 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 7725, "mdsmap_epoch": 61, "osdmap_epoch": 120, "osdmap_epoch_barrier": 119, "rank_uptime": 8.361645993, "state": "up:active", "uptime": 12.766060995, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:34:00.625 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:34:00.626 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:34:00.627 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:34:00.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:00.628 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:00.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:00.712 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:00.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:00.794 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:00.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:00.803 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:00.888 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:34:00.889 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:00.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:00.942 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:00.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:00.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:34:00.987+0000 7fc84b912080 -1 init, newargv = 0x56012b27c9a0 newargc=9ceph-fuse[22858]: starting ceph client 2022-07-20T02:34:00.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr: 2022-07-20T02:34:01.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22858]: starting fuse 2022-07-20T02:34:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:01.010 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:01.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.013 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:01.043 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:34:01.044 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:34:01.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:34:01.047 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:34:01.074 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:34:01.074 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:34:01.075 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:34:01.076 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:01.076 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:34:01.077 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:34:01.078 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:34:01.078 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:01.079 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:34:01.079 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:34:01.081 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:34:01.081 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:34:01.082 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:01.087 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:34:01.088 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:34:01.088 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:34:01.093 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:34:01.103 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:34:01.104 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:01.104 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:34:01.105 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:34:01.187 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22858.asok 2022-07-20T02:34:01.190 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22858.asok 2022-07-20T02:34:01.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22858.asok status 2022-07-20T02:34:01.285 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:34:01.286 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:34:01.287 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:34:01.287 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:34:01.288 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:34:01.288 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:34:01.289 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:34:01.289 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22858", 2022-07-20T02:34:01.290 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:34:01.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:01.290 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:34:01.291 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:34:01.292 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 6776, 2022-07-20T02:34:01.292 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:34:01.292 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:34:01.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:34:01.293 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 6776 2022-07-20T02:34:01.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:01.294 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:01.295 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:01.295 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:01.296 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3813806319 2022-07-20T02:34:01.296 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:34:01.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:01.297 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:01.297 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:01.299 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:01.299 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3813806319 2022-07-20T02:34:01.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:01.300 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.6776 172.21.15.176:0/3813806319", 2022-07-20T02:34:01.301 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/3813806319", 2022-07-20T02:34:01.301 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:34:01.301 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 61, 2022-07-20T02:34:01.302 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 120, 2022-07-20T02:34:01.303 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:34:01.304 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:34:01.304 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:34:01.304 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:34:01.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.319 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.319 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.335 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:01.336 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:01.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.338 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.406 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.408 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:01.462 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:34:01.467 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:01.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.468 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.469 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:01.521 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:34:01.521 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:01.522 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:34:01.525 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:34:01.525 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:01.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23241]: fuse finished with error 0 and tester_r 0 2022-07-20T02:34:01.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.557 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.569 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:01.665 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:34:01.670 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:34:01.671 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:34:01.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:01.673 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:01.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.690 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.692 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.706 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:01.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:01.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.708 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.723 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.735 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:01.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:01.737 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:34:01.737 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:34:01.739 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22858]: fuse finished with error 0 and tester_r 0 2022-07-20T02:34:01.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:01.807 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:01.807 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:01.814 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:34:01.815 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:34:01.816 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:34:07.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:07.927 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:08.019 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:34:08.020 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:34:08.021 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:34:08.021 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:34:08.023 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:34:08.023 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:34:08.052 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:34:08.070 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:34:08.071 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:08.072 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:08.105 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:34:08.106 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:08.107 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2022-07-20T02:34:08.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2022-07-20T02:34:08.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2022-07-20T02:34:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 Processor -- start 2022-07-20T02:34:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 -- start start 2022-07-20T02:34:08.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 0x7fa58c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 0x7fa58c1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c147080 con 0x7fa58c067440 2022-07-20T02:34:08.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c143000 con 0x7fa58c0957f0 2022-07-20T02:34:08.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1472b0 con 0x7fa58c0b2f20 2022-07-20T02:34:08.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 Processor -- start 2022-07-20T02:34:08.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58a7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa58c067440 0x7fa58c0950e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41634/0 (socket says 172.21.15.26:41634) 2022-07-20T02:34:08.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58a7fc700 1 -- 172.21.15.26:0/2482107966 learned_addr learned my addr 172.21.15.26:0/2482107966 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:08.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 -- start start 2022-07-20T02:34:08.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58b7fe700 1 --2- 172.21.15.26:0/2482107966 >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 0x7fa58c1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 --2- 172.21.15.26:0/2482107966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 0x7fa58c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783167585 0 0) 0x7fa58c147080 con 0x7fa58c067440 2022-07-20T02:34:08.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003260 con 0x7fa58c067440 2022-07-20T02:34:08.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 -- 172.21.15.26:0/2482107966 >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 msgr2=0x7fa58c1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:08.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 --2- 172.21.15.26:0/2482107966 >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 0x7fa58c1433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32628553 0 0) 0x7fa570003260 con 0x7fa58c067440 2022-07-20T02:34:08.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 -- 172.21.15.26:0/2482107966 >> v1:172.21.15.26:6789/0 conn(0x7fa58c067440 legacy=0x7fa58c0950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 -- 172.21.15.26:0/2482107966 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0a9920 con 0x7fa58c0b2f20 2022-07-20T02:34:08.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58b7fe700 1 --2- 172.21.15.26:0/2482107966 >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 0x7fa58c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:08.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa58affd700 1 --2- 172.21.15.26:0/2482107966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 0x7fa58c0b3300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa580001800 tx=0x7fa57c006df0).ready entity=mon.2 client_cookie=96bd5920f75a0cb4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29480957f0 0x7f29480b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2948067440 0x7f29481433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2948147080 con 0x7f29480b2f20 2022-07-20T02:34:08.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2948143000 con 0x7f2948067440 2022-07-20T02:34:08.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294f2d9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29481472b0 con 0x7f29480957f0 2022-07-20T02:34:08.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2948067440 0x7f29481433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2948067440 0x7f29481433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42362/0 (socket says 172.21.15.26:42362) 2022-07-20T02:34:08.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 -- 172.21.15.26:0/4081623979 learned_addr learned my addr 172.21.15.26:0/4081623979 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:08.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa580001660 con 0x7fa58c0b2f20 2022-07-20T02:34:08.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c000e90 con 0x7fa58c0b2f20 2022-07-20T02:34:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7fa589ffb700 1 -- 172.21.15.26:0/2482107966 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa57c012460 con 0x7fa58c0b2f20 2022-07-20T02:34:08.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 -- 172.21.15.26:0/4081623979 >> v2:172.21.15.120:3300/0 conn(0x7f29480957f0 msgr2=0x7f29480b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:08.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 --2- 172.21.15.26:0/4081623979 >> v2:172.21.15.120:3300/0 conn(0x7f29480957f0 0x7f29480b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:34:08.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 -- 172.21.15.26:0/4081623979 >> v1:172.21.15.120:6789/0 conn(0x7f29480b2f20 legacy=0x7f29480950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.251+0000 7f294ead8700 1 -- 172.21.15.26:0/4081623979 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29480a9920 con 0x7f2948067440 2022-07-20T02:34:08.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294ead8700 1 --2- 172.21.15.26:0/4081623979 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2948067440 0x7f29481433f0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f29400032f0 tx=0x7f2940009290).ready entity=mon.0 client_cookie=8178d10fbfe9ad54 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- 172.21.15.26:0/2482107966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 msgr2=0x7fa58c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- 172.21.15.26:0/2482107966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 0x7fa58c0b3300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa580001800 tx=0x7fa57c006df0).stop 2022-07-20T02:34:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294d2d5700 1 -- 172.21.15.26:0/4081623979 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f294000b870 con 0x7f2948067440 2022-07-20T02:34:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294d2d5700 1 -- 172.21.15.26:0/4081623979 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294000b9d0 con 0x7f2948067440 2022-07-20T02:34:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294d2d5700 1 -- 172.21.15.26:0/4081623979 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f294000bb80 con 0x7f2948067440 2022-07-20T02:34:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- 172.21.15.26:0/2482107966 shutdown_connections 2022-07-20T02:34:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- 172.21.15.26:0/2482107966 >> v2:172.21.15.120:3300/0 conn(0x7fa58c0957f0 0x7fa58c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- 172.21.15.26:0/2482107966 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0b2f20 0x7fa58c0b3300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- 172.21.15.26:0/2482107966 >> 172.21.15.26:0/2482107966 conn(0x7fa58c0b67f0 msgr2=0x7fa58c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- 172.21.15.26:0/2482107966 shutdown_connections 2022-07-20T02:34:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- 172.21.15.26:0/2482107966 wait complete. 2022-07-20T02:34:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- 172.21.15.26:0/4081623979 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2948067440 msgr2=0x7f29481433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 --2- 172.21.15.26:0/4081623979 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2948067440 0x7f29481433f0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f29400032f0 tx=0x7f2940009290).stop 2022-07-20T02:34:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- 172.21.15.26:0/4081623979 shutdown_connections 2022-07-20T02:34:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 --2- 172.21.15.26:0/4081623979 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2948067440 0x7f29481433f0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 --2- 172.21.15.26:0/4081623979 >> v2:172.21.15.120:3300/0 conn(0x7f29480957f0 0x7f29480b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:34:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- 172.21.15.26:0/4081623979 >> 172.21.15.26:0/4081623979 conn(0x7f29480b67f0 msgr2=0x7f29480b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 Processor -- start 2022-07-20T02:34:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- start start 2022-07-20T02:34:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- 172.21.15.26:0/4081623979 shutdown_connections 2022-07-20T02:34:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- 172.21.15.26:0/4081623979 wait complete. 2022-07-20T02:34:08.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 0x7fa58c099df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 0x7fa58c09a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa58c0b2f20 0x7fa58c092b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c153350 con 0x7fa58c067440 2022-07-20T02:34:08.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c09b7a0 con 0x7fa58c0b2f20 2022-07-20T02:34:08.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c09ba40 con 0x7fa58c0957f0 2022-07-20T02:34:08.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 0x7fa58c099df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 0x7fa58c099df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42364/0 (socket says 172.21.15.26:42364) 2022-07-20T02:34:08.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58affd700 1 -- 172.21.15.26:0/309492845 learned_addr learned my addr 172.21.15.26:0/309492845 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:08.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58a7fc700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 0x7fa58c09a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 Processor -- start 2022-07-20T02:34:08.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7f294f2d9700 1 -- start start 2022-07-20T02:34:08.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.255+0000 7fa58b7fe700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa58c0b2f20 0x7fa58c092b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 -- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa58c0b2f20 msgr2=0x7fa58c092b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa58c0b2f20 0x7fa58c092b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 -- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 msgr2=0x7fa58c099df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 0x7fa58c099df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 -- 172.21.15.26:0/309492845 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa584009470 con 0x7fa58c0957f0 2022-07-20T02:34:08.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 Processor -- start 2022-07-20T02:34:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58a7fc700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 0x7fa58c09a2e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa5800018c0 tx=0x7fa58000c7f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 -- start start 2022-07-20T02:34:08.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2948067440 0x7f29480ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa580007170 con 0x7fa58c0957f0 2022-07-20T02:34:08.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5800072d0 con 0x7fa58c0957f0 2022-07-20T02:34:08.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c00b180 con 0x7fa58c0957f0 2022-07-20T02:34:08.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa580004920 con 0x7fa58c0957f0 2022-07-20T02:34:08.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c09ad40 con 0x7fa58c0957f0 2022-07-20T02:34:08.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 0x7f2948155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948142590 con 0x7f29480957f0 2022-07-20T02:34:08.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29480af9f0 con 0x7f29480b2f20 2022-07-20T02:34:08.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294809b920 con 0x7f2948067440 2022-07-20T02:34:08.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42374/0 (socket says 172.21.15.26:42374) 2022-07-20T02:34:08.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 -- 172.21.15.26:0/3669721530 learned_addr learned my addr 172.21.15.26:0/3669721530 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:08.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294ead8700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 0x7f2948155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 -- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2948067440 msgr2=0x7f29480ac7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-07-20T02:34:08.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2948067440 0x7f29480ac7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T02:34:08.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 -- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 msgr2=0x7f2948155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 0x7f2948155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 -- 172.21.15.26:0/3669721530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940009530 con 0x7f29480957f0 2022-07-20T02:34:08.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294ead8700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 0x7f2948155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:08.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294dad6700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f2938002f70 tx=0x7f2938006e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa38c1432e0 0x7fa38c12c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938013460 con 0x7f29480957f0 2022-07-20T02:34:08.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380135c0 con 0x7f29480957f0 2022-07-20T02:34:08.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938013920 con 0x7f29480957f0 2022-07-20T02:34:08.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c061e00 con 0x7fa38c1468c0 2022-07-20T02:34:08.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c12cc40 con 0x7fa38c061a20 2022-07-20T02:34:08.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa390af6700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c147040 con 0x7fa38c1432e0 2022-07-20T02:34:08.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa38ad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa38c1468c0 0x7fa38c0b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41660/0 (socket says 172.21.15.26:41660) 2022-07-20T02:34:08.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa38ad9d700 1 -- 172.21.15.26:0/1257690699 learned_addr learned my addr 172.21.15.26:0/1257690699 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:08.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa38a59c700 1 --1- 172.21.15.26:0/1257690699 >> v1:172.21.15.120:6789/0 conn(0x7fa38c061a20 0x7fa38c12c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:55868/0 (socket says 172.21.15.26:55868) 2022-07-20T02:34:08.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c153550 con 0x7fa58c0957f0 2022-07-20T02:34:08.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294809bbc0 con 0x7f29480957f0 2022-07-20T02:34:08.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa580004a80 con 0x7fa58c0957f0 2022-07-20T02:34:08.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa383fff700 1 --2- 172.21.15.26:0/1257690699 >> v2:172.21.15.120:3300/0 conn(0x7fa38c1432e0 0x7fa38c12c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896072585 0 0) 0x7fa38c061e00 con 0x7fa38c1468c0 2022-07-20T02:34:08.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa36c003260 con 0x7fa38c1468c0 2022-07-20T02:34:08.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084224775 0 0) 0x7fa38c12cc40 con 0x7fa38c061a20 2022-07-20T02:34:08.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa38c061e00 con 0x7fa38c061a20 2022-07-20T02:34:08.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051156908 0 0) 0x7fa36c003260 con 0x7fa38c1468c0 2022-07-20T02:34:08.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa383fff700 1 -- 172.21.15.26:0/1257690699 >> v1:172.21.15.120:6789/0 conn(0x7fa38c061a20 legacy=0x7fa38c12c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa383fff700 1 -- 172.21.15.26:0/1257690699 >> v1:172.21.15.26:6789/0 conn(0x7fa38c1468c0 legacy=0x7fa38c0b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa383fff700 1 -- 172.21.15.26:0/1257690699 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c1472e0 con 0x7fa38c1432e0 2022-07-20T02:34:08.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29480a9da0 con 0x7f29480957f0 2022-07-20T02:34:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa383fff700 1 --2- 172.21.15.26:0/1257690699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c12c750 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fa384001490 tx=0x7fa374006df0).ready entity=mon.1 client_cookie=dc253c47b09fb71e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 0x7fa568055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.259+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa58006bc40 con 0x7fa58c0957f0 2022-07-20T02:34:08.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3840019e0 con 0x7fa38c1432e0 2022-07-20T02:34:08.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa58affd700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 0x7fa568055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa374000e90 con 0x7fa38c1432e0 2022-07-20T02:34:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa389d9b700 1 -- 172.21.15.26:0/1257690699 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa374012460 con 0x7fa38c1432e0 2022-07-20T02:34:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa58affd700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 0x7fa568055e10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fa57c018000 tx=0x7fa57c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- 172.21.15.26:0/1257690699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 msgr2=0x7fa38c12c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 --2- 172.21.15.26:0/1257690699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c12c750 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fa384001490 tx=0x7fa374006df0).stop 2022-07-20T02:34:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29480a95a0 con 0x7f29480957f0 2022-07-20T02:34:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f293800f070 con 0x7f29480957f0 2022-07-20T02:34:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- 172.21.15.26:0/1257690699 shutdown_connections 2022-07-20T02:34:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 --2- 172.21.15.26:0/1257690699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c12c750 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- 172.21.15.26:0/1257690699 >> 172.21.15.26:0/1257690699 conn(0x7fa38c05faf0 msgr2=0x7fa38c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- 172.21.15.26:0/1257690699 shutdown_connections 2022-07-20T02:34:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- 172.21.15.26:0/1257690699 wait complete. 2022-07-20T02:34:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f29377fe700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 0x7f2928055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f293806b900 con 0x7f29480957f0 2022-07-20T02:34:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f294e2d7700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 0x7f2928055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7f294e2d7700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 0x7f2928055ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f29480b5c80 tx=0x7f2944007810).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 Processor -- start 2022-07-20T02:34:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 -- start start 2022-07-20T02:34:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c061a20 0x7fa38c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c0b9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.263+0000 7fa390af6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0b2e00 con 0x7fa38c1468c0 2022-07-20T02:34:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1564f0 con 0x7fa38c1432e0 2022-07-20T02:34:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c156750 con 0x7fa38c061a20 2022-07-20T02:34:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42390/0 (socket says 172.21.15.26:42390) 2022-07-20T02:34:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 -- 172.21.15.26:0/2208915927 learned_addr learned my addr 172.21.15.26:0/2208915927 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38a59c700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c061a20 0x7fa38c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa383fff700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c0b9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 -- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c061a20 msgr2=0x7fa38c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c061a20 0x7fa38c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 -- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 msgr2=0x7fa38c0b9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c0b9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 -- 172.21.15.26:0/2208915927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37400b180 con 0x7fa38c1468c0 2022-07-20T02:34:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa383fff700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c0b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:08.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa38ad9d700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7fa384001360 tx=0x7fa384001c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa384004ab0 con 0x7fa38c1468c0 2022-07-20T02:34:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384004c10 con 0x7fa38c1468c0 2022-07-20T02:34:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c0b3030 con 0x7fa38c1468c0 2022-07-20T02:34:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa384007170 con 0x7fa38c1468c0 2022-07-20T02:34:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c152440 con 0x7fa38c1468c0 2022-07-20T02:34:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29380426c0 con 0x7f29480957f0 2022-07-20T02:34:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.267+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c147580 con 0x7fa38c1468c0 2022-07-20T02:34:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa3840072d0 con 0x7fa38c1468c0 2022-07-20T02:34:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa3837fe700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 0x7fa370055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa38406bde0 con 0x7fa38c1468c0 2022-07-20T02:34:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa38a59c700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 0x7fa370055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa38a59c700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 0x7fa370055f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fa37c00d000 tx=0x7fa37c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.271+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa580042870 con 0x7fa58c0957f0 2022-07-20T02:34:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.279+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa384042ba0 con 0x7fa38c1468c0 2022-07-20T02:34:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.435+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f29480574e0 con 0x7f29480957f0 2022-07-20T02:34:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.439+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fa58c0a95a0 con 0x7fa58c0957f0 2022-07-20T02:34:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.447+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fa38c0574e0 con 0x7fa38c1468c0 2022-07-20T02:34:08.623 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:34:08.587+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:34:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.595+0000 7f29377fe700 1 -- 172.21.15.26:0/3669721530 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7504 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2938045ce0 con 0x7f29480957f0 2022-07-20T02:34:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.595+0000 7fa3837fe700 1 -- 172.21.15.26:0/2208915927 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7495 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa3840461c0 con 0x7fa38c1468c0 2022-07-20T02:34:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.595+0000 7fa57bfff700 1 -- 172.21.15.26:0/309492845 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7725 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa580045e90 con 0x7fa58c0957f0 2022-07-20T02:34:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 msgr2=0x7fa370055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 0x7fa370055f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fa37c00d000 tx=0x7fa37c009040).stop 2022-07-20T02:34:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 msgr2=0x7fa38c12b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7fa384001360 tx=0x7fa384001c50).stop 2022-07-20T02:34:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 msgr2=0x7f2928055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 0x7f2928055ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f29480b5c80 tx=0x7f2944007810).stop 2022-07-20T02:34:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 msgr2=0x7f29480acd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f2938002f70 tx=0x7f2938006e90).stop 2022-07-20T02:34:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 shutdown_connections 2022-07-20T02:34:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa38c1468c0 0x7fa38c12b260 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa370053a70 0x7fa370055f30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c1432e0 0x7fa38c0b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 --2- 172.21.15.26:0/2208915927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c061a20 0x7fa38c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 >> 172.21.15.26:0/2208915927 conn(0x7fa38c05faf0 msgr2=0x7fa38c1415b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 msgr2=0x7fa568055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 0x7fa568055e10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fa57c018000 tx=0x7fa57c014000).stop 2022-07-20T02:34:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 msgr2=0x7fa58c09a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.599+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 0x7fa58c09a2e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa5800018c0 tx=0x7fa58000c7f0).stop 2022-07-20T02:34:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 shutdown_connections 2022-07-20T02:34:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29480957f0 0x7f29480acd20 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2928053a10 0x7f2928055ed0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480b2f20 0x7f2948155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 --2- 172.21.15.26:0/3669721530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2948067440 0x7f29480ac7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T02:34:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 >> 172.21.15.26:0/3669721530 conn(0x7f29480b67f0 msgr2=0x7f2948060360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 shutdown_connections 2022-07-20T02:34:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa568053950 0x7fa568055e10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa58c0b2f20 0x7fa58c092b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa58c0957f0 0x7fa58c09a2e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 --2- 172.21.15.26:0/309492845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa58c067440 0x7fa58c099df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 >> 172.21.15.26:0/309492845 conn(0x7fa58c0b67f0 msgr2=0x7fa58c0b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 shutdown_connections 2022-07-20T02:34:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 shutdown_connections 2022-07-20T02:34:08.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 shutdown_connections 2022-07-20T02:34:08.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa58bfff700 1 -- 172.21.15.26:0/309492845 wait complete. 2022-07-20T02:34:08.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7fa390af6700 1 -- 172.21.15.26:0/2208915927 wait complete. 2022-07-20T02:34:08.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.603+0000 7f294f2d9700 1 -- 172.21.15.26:0/3669721530 wait complete. 2022-07-20T02:34:08.649 INFO:teuthology.orchestra.run.smithi026.stderr:failed mds gid 7725 2022-07-20T02:34:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:failed mds gid 7504 2022-07-20T02:34:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:failed mds gid 7495 2022-07-20T02:34:08.653 DEBUG:teuthology.parallel:result is '' 2022-07-20T02:34:08.655 DEBUG:teuthology.parallel:result is '' 2022-07-20T02:34:08.656 DEBUG:teuthology.parallel:result is '' 2022-07-20T02:34:08.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:34:08.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.771+0000 7efc04468700 1 Processor -- start 2022-07-20T02:34:08.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.771+0000 7efc04468700 1 -- start start 2022-07-20T02:34:08.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efbfc0b2ac0 0x7efbfc0ab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc0b2ea0 con 0x7efbfc061030 2022-07-20T02:34:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0abb30 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc062420 con 0x7efbfc0aa4a0 2022-07-20T02:34:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01a03700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efbfc061030 0x7efbfc0a9d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41674/0 (socket says 172.21.15.26:41674) 2022-07-20T02:34:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01a03700 1 -- 172.21.15.26:0/1516752394 learned_addr learned my addr 172.21.15.26:0/1516752394 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 --2- 172.21.15.26:0/1516752394 >> v2:172.21.15.26:3300/0 conn(0x7efbfc0b2ac0 0x7efbfc0ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02a05700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01202700 1 -- 172.21.15.26:0/1516752394 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457403774 0 0) 0x7efbfc0b2ea0 con 0x7efbfc061030 2022-07-20T02:34:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01202700 1 -- 172.21.15.26:0/1516752394 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003260 con 0x7efbfc061030 2022-07-20T02:34:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 -- 172.21.15.26:0/1516752394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 msgr2=0x7efbfc140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 -- 172.21.15.26:0/1516752394 >> v1:172.21.15.26:6789/0 conn(0x7efbfc061030 legacy=0x7efbfc0a9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 -- 172.21.15.26:0/1516752394 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0a8fd0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc02204700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc0ab160 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7efbf8002f70 tx=0x7efbf8006df0).ready entity=mon.0 client_cookie=55f04c0d4e624327 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01202700 1 -- 172.21.15.26:0/1516752394 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf800d4a0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01202700 1 -- 172.21.15.26:0/1516752394 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf8000e90 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc01202700 1 -- 172.21.15.26:0/1516752394 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf800d7f0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- 172.21.15.26:0/1516752394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 msgr2=0x7efbfc0ab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc0ab160 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7efbf8002f70 tx=0x7efbf8006df0).stop 2022-07-20T02:34:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- 172.21.15.26:0/1516752394 shutdown_connections 2022-07-20T02:34:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 --2- 172.21.15.26:0/1516752394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc0ab160 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- 172.21.15.26:0/1516752394 >> 172.21.15.26:0/1516752394 conn(0x7efbfc0b60a0 msgr2=0x7efbfc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:08.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- 172.21.15.26:0/1516752394 shutdown_connections 2022-07-20T02:34:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.775+0000 7efc04468700 1 -- 172.21.15.26:0/1516752394 wait complete. 2022-07-20T02:34:08.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 Processor -- start 2022-07-20T02:34:08.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 -- start start 2022-07-20T02:34:08.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc061030 0x7efbfc0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc1528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0a8ab0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc1524b0 con 0x7efbfc061030 2022-07-20T02:34:08.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0a1df0 con 0x7efbfc0aa4a0 2022-07-20T02:34:08.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02204700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc061030 0x7efbfc0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42404/0 (socket says 172.21.15.26:42404) 2022-07-20T02:34:08.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc01a03700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc1528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 -- 172.21.15.26:0/1723151107 learned_addr learned my addr 172.21.15.26:0/1723151107 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 -- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 msgr2=0x7efbfc1528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc1528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 -- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc061030 msgr2=0x7efbfc0aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:08.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc061030 0x7efbfc0aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:08.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 -- 172.21.15.26:0/1723151107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf800b180 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc02a05700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7efbf4006e00 tx=0x7efbf40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf400c640 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf400c7a0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc0a2090 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.779+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf400cb00 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc0a25d0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc0a2790 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efbf400cc60 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efbeb7fe700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 0x7efbdc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:08.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6506+0+0 (secure 0 0 0) 0x7efbf406c280 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.783+0000 7efc02204700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 0x7efbdc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:08.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.787+0000 7efc02204700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 0x7efbdc055ff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7efbf801a000 tx=0x7efbf8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:08.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.791+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efbf4042ff0 con 0x7efbfc0b2ac0 2022-07-20T02:34:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:08.955+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efbfc0a9650 con 0x7efbfc0b2ac0 2022-07-20T02:34:09.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.611+0000 7efbeb7fe700 1 -- 172.21.15.26:0/1723151107 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7efbf4046610 con 0x7efbfc0b2ac0 2022-07-20T02:34:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 msgr2=0x7efbdc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 0x7efbdc055ff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7efbf801a000 tx=0x7efbf8014000).stop 2022-07-20T02:34:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 msgr2=0x7efbfc152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7efbf4006e00 tx=0x7efbf40091d0).stop 2022-07-20T02:34:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 shutdown_connections 2022-07-20T02:34:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efbfc0b2ac0 0x7efbfc152dd0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efbdc053b30 0x7efbdc055ff0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efbfc0aa4a0 0x7efbfc1528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 --2- 172.21.15.26:0/1723151107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efbfc061030 0x7efbfc0aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 >> 172.21.15.26:0/1723151107 conn(0x7efbfc0b60a0 msgr2=0x7efbfc0aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 shutdown_connections 2022-07-20T02:34:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.615+0000 7efc04468700 1 -- 172.21.15.26:0/1723151107 wait complete. 2022-07-20T02:34:09.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:34:09.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.763+0000 7fb101811700 1 Processor -- start 2022-07-20T02:34:09.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.763+0000 7fb101811700 1 -- start start 2022-07-20T02:34:09.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 0x7fb0fc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0fc147000 0x7fb0fc0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc1474e0 con 0x7fb0fc060f00 2022-07-20T02:34:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc05d3b0 con 0x7fb0fc147000 2022-07-20T02:34:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc1431e0 con 0x7fb0fc05cfd0 2022-07-20T02:34:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101010700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb0fc060f00 0x7fb0fc13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41688/0 (socket says 172.21.15.26:41688) 2022-07-20T02:34:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101010700 1 -- 172.21.15.26:0/3275785148 learned_addr learned my addr 172.21.15.26:0/3275785148 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 --2- 172.21.15.26:0/3275785148 >> v2:172.21.15.120:3300/0 conn(0x7fb0fc147000 0x7fb0fc0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb10080f700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 0x7fb0fc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546576356 0 0) 0x7fb0fc1474e0 con 0x7fb0fc060f00 2022-07-20T02:34:09.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0dc003260 con 0x7fb0fc060f00 2022-07-20T02:34:09.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 -- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 msgr2=0x7fb0fc143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:09.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 0x7fb0fc143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 -- 172.21.15.26:0/3275785148 >> v1:172.21.15.26:6789/0 conn(0x7fb0fc060f00 legacy=0x7fb0fc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995624654 0 0) 0x7fb0dc003260 con 0x7fb0fc060f00 2022-07-20T02:34:09.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 -- 172.21.15.26:0/3275785148 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc143480 con 0x7fb0fc147000 2022-07-20T02:34:09.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb10080f700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 0x7fb0fc143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fbfff700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc147000 0x7fb0fc0b4730 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb0f4001b80 tx=0x7fb0e4006df0).ready entity=mon.1 client_cookie=fe1b79dd83516950 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f40019e0 con 0x7fb0fc147000 2022-07-20T02:34:09.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4000e90 con 0x7fb0fc147000 2022-07-20T02:34:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb0fb7fe700 1 -- 172.21.15.26:0/3275785148 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e4012460 con 0x7fb0fc147000 2022-07-20T02:34:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc147000 msgr2=0x7fb0fc0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc147000 0x7fb0fc0b4730 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb0f4001b80 tx=0x7fb0e4006df0).stop 2022-07-20T02:34:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- 172.21.15.26:0/3275785148 shutdown_connections 2022-07-20T02:34:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc147000 0x7fb0fc0b4730 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 --2- 172.21.15.26:0/3275785148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc05cfd0 0x7fb0fc143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- 172.21.15.26:0/3275785148 >> 172.21.15.26:0/3275785148 conn(0x7fb0fc0620c0 msgr2=0x7fb0fc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- 172.21.15.26:0/3275785148 shutdown_connections 2022-07-20T02:34:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.767+0000 7fb101811700 1 -- 172.21.15.26:0/3275785148 wait complete. 2022-07-20T02:34:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 Processor -- start 2022-07-20T02:34:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 -- start start 2022-07-20T02:34:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 0x7fb0fc13afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 0x7fb0fc0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b3420 con 0x7fb0fc146b20 2022-07-20T02:34:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b38b0 con 0x7fb0fc060f00 2022-07-20T02:34:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101811700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc156170 con 0x7fb0fc147000 2022-07-20T02:34:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42416/0 (socket says 172.21.15.26:42416) 2022-07-20T02:34:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 -- 172.21.15.26:0/1899634105 learned_addr learned my addr 172.21.15.26:0/1899634105 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:09.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb10080f700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 0x7fb0fc13afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb0fbfff700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 0x7fb0fc0ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 -- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 msgr2=0x7fb0fc0ad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 0x7fb0fc0ad690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 -- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 msgr2=0x7fb0fc13afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:09.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 0x7fb0fc13afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:09.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 -- 172.21.15.26:0/1899634105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec00b180 con 0x7fb0fc146b20 2022-07-20T02:34:09.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb0fbfff700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 0x7fb0fc0ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:09.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb10080f700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 0x7fb0fc13afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb101010700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7fb0f4001c40 tx=0x7fb0f4001640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.771+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f4004ab0 con 0x7fb0fc146b20 2022-07-20T02:34:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f4004c10 con 0x7fb0fc146b20 2022-07-20T02:34:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e400b180 con 0x7fb0fc146b20 2022-07-20T02:34:09.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f4007170 con 0x7fb0fc146b20 2022-07-20T02:34:09.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc0b2900 con 0x7fb0fc146b20 2022-07-20T02:34:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc155fc0 con 0x7fb0fc146b20 2022-07-20T02:34:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb0f4010070 con 0x7fb0fc146b20 2022-07-20T02:34:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb0f9ffb700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 0x7fb0e0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb0f406bcb0 con 0x7fb0fc146b20 2022-07-20T02:34:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.775+0000 7fb10080f700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 0x7fb0e0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.779+0000 7fb10080f700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 0x7fb0e0055f30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fb0ec00b000 tx=0x7fb0ec006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.783+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0f4042980 con 0x7fb0fc146b20 2022-07-20T02:34:09.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:09.911+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0fc143720 con 0x7fb0fc146b20 2022-07-20T02:34:10.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.623+0000 7fb0f9ffb700 1 -- 172.21.15.26:0/1899634105 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb0f4019030 con 0x7fb0fc146b20 2022-07-20T02:34:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 msgr2=0x7fb0e0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:10.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 0x7fb0e0055f30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fb0ec00b000 tx=0x7fb0ec006df0).stop 2022-07-20T02:34:10.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 msgr2=0x7fb0fc0b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:10.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7fb0f4001c40 tx=0x7fb0f4001640).stop 2022-07-20T02:34:10.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 shutdown_connections 2022-07-20T02:34:10.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0fc146b20 0x7fb0fc0b0130 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb0e0053a70 0x7fb0e0055f30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0fc147000 0x7fb0fc0ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 --2- 172.21.15.26:0/1899634105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0fc060f00 0x7fb0fc13afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 >> 172.21.15.26:0/1899634105 conn(0x7fb0fc05cfd0 msgr2=0x7fb0fc0b58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:10.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 shutdown_connections 2022-07-20T02:34:10.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.627+0000 7fb101811700 1 -- 172.21.15.26:0/1899634105 wait complete. 2022-07-20T02:34:10.685 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:34:10.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:34:10.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 Processor -- start 2022-07-20T02:34:10.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 -- start start 2022-07-20T02:34:10.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 0x7f43f40b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:10.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43f405d050 0x7f43f4143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:10.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f40a9e10 con 0x7f43f4143e00 2022-07-20T02:34:10.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f40617a0 con 0x7f43f405d050 2022-07-20T02:34:10.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43fba50700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f40aa040 con 0x7f43f40b0050 2022-07-20T02:34:10.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43f405d050 0x7f43f4143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:10.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f43f405d050 0x7f43f4143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42422/0 (socket says 172.21.15.26:42422) 2022-07-20T02:34:10.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 -- 172.21.15.26:0/2244624330 learned_addr learned my addr 172.21.15.26:0/2244624330 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:10.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f97ec700 1 --2- 172.21.15.26:0/2244624330 >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 0x7f43f40b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:10.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 -- 172.21.15.26:0/2244624330 >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 msgr2=0x7f43f40b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:10.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 --2- 172.21.15.26:0/2244624330 >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 0x7f43f40b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 -- 172.21.15.26:0/2244624330 >> v1:172.21.15.120:6789/0 conn(0x7f43f4143e00 legacy=0x7f43f40b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:10.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 -- 172.21.15.26:0/2244624330 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f40aa2e0 con 0x7f43f405d050 2022-07-20T02:34:10.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f97ec700 1 --2- 172.21.15.26:0/2244624330 >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 0x7f43f40b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:10.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43f8feb700 1 --2- 172.21.15.26:0/2244624330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4143910 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f43e4002f70 tx=0x7f43e4006ee0).ready entity=mon.0 client_cookie=39931ac76b3a9286 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:10.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43ebfff700 1 -- 172.21.15.26:0/2244624330 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e400bc50 con 0x7f43f405d050 2022-07-20T02:34:10.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43ebfff700 1 -- 172.21.15.26:0/2244624330 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4000e90 con 0x7f43f405d050 2022-07-20T02:34:10.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.807+0000 7f43ebfff700 1 -- 172.21.15.26:0/2244624330 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e400de90 con 0x7f43f405d050 2022-07-20T02:34:10.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- 172.21.15.26:0/2244624330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 msgr2=0x7f43f4143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:10.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- 172.21.15.26:0/2244624330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4143910 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f43e4002f70 tx=0x7f43e4006ee0).stop 2022-07-20T02:34:10.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- 172.21.15.26:0/2244624330 shutdown_connections 2022-07-20T02:34:10.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- 172.21.15.26:0/2244624330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4143910 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- 172.21.15.26:0/2244624330 >> v2:172.21.15.120:3300/0 conn(0x7f43f40b0050 0x7f43f40b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:10.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- 172.21.15.26:0/2244624330 >> 172.21.15.26:0/2244624330 conn(0x7f43f4060950 msgr2=0x7f43f4060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:10.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- 172.21.15.26:0/2244624330 shutdown_connections 2022-07-20T02:34:10.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- 172.21.15.26:0/2244624330 wait complete. 2022-07-20T02:34:10.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 Processor -- start 2022-07-20T02:34:10.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- start start 2022-07-20T02:34:10.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:10.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f40b0050 0x7f43f4065fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:10.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43f4143e00 0x7f43f41405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:10.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f4064f10 con 0x7f43f405d050 2022-07-20T02:34:10.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f4156400 con 0x7f43f40b0050 2022-07-20T02:34:10.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43fba50700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f41566a0 con 0x7f43f4143e00 2022-07-20T02:34:10.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43f97ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43f97ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42426/0 (socket says 172.21.15.26:42426) 2022-07-20T02:34:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43f97ec700 1 -- 172.21.15.26:0/1594250544 learned_addr learned my addr 172.21.15.26:0/1594250544 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:11.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43f8feb700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f40b0050 0x7f43f4065fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.811+0000 7f43f9fed700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43f4143e00 0x7f43f41405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 -- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43f4143e00 msgr2=0x7f43f41405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43f4143e00 0x7f43f41405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 -- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f40b0050 msgr2=0x7f43f4065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f40b0050 0x7f43f4065fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 -- 172.21.15.26:0/1594250544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e400d410 con 0x7f43f405d050 2022-07-20T02:34:11.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43f97ec700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f43f0006940 tx=0x7f43f0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:11.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f0011780 con 0x7f43f405d050 2022-07-20T02:34:11.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00118e0 con 0x7f43f405d050 2022-07-20T02:34:11.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f4064a70 con 0x7f43f405d050 2022-07-20T02:34:11.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f0011c40 con 0x7f43f405d050 2022-07-20T02:34:11.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f4063dd0 con 0x7f43f405d050 2022-07-20T02:34:11.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f4156860 con 0x7f43f405d050 2022-07-20T02:34:11.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.815+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43f0012420 con 0x7f43f405d050 2022-07-20T02:34:11.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.819+0000 7f43ea7fc700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 0x7f43dc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.819+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f43f006bc70 con 0x7f43f405d050 2022-07-20T02:34:11.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.819+0000 7f43f8feb700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 0x7f43dc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.819+0000 7f43f8feb700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 0x7f43dc055e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f43e401b000 tx=0x7f43f40aa470).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:11.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.819+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43f0042a90 con 0x7f43f405d050 2022-07-20T02:34:11.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:10.947+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f43f40ae520 con 0x7f43f405d050 2022-07-20T02:34:11.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.631+0000 7f43ea7fc700 1 -- 172.21.15.26:0/1594250544 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v125) v1 ==== 133+0+0 (secure 0 0 0) 0x7f43f00460b0 con 0x7f43f405d050 2022-07-20T02:34:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 msgr2=0x7f43dc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 0x7f43dc055e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f43e401b000 tx=0x7f43f40aa470).stop 2022-07-20T02:34:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 msgr2=0x7f43f4155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f43f0006940 tx=0x7f43f0008040).stop 2022-07-20T02:34:11.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 shutdown_connections 2022-07-20T02:34:11.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f43dc053950 0x7f43dc055e10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43f4143e00 0x7f43f41405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f40b0050 0x7f43f4065fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 --2- 172.21.15.26:0/1594250544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43f405d050 0x7f43f4155d50 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 >> 172.21.15.26:0/1594250544 conn(0x7f43f4060950 msgr2=0x7f43f4144790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:11.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 shutdown_connections 2022-07-20T02:34:11.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:11.635+0000 7f43fba50700 1 -- 172.21.15.26:0/1594250544 wait complete. 2022-07-20T02:34:11.683 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:34:11.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo rados -p cephfs_metadata put key /etc/fstab 2022-07-20T02:34:11.778 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 Processor -- start 2022-07-20T02:34:11.779 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- start start 2022-07-20T02:34:11.780 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc84440 0x55af2dc8dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.783 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84dc0 0x55af2dc8e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.783 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.791 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2db736f0 con 0x55af2dc84440 2022-07-20T02:34:11.792 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2dc92370 con 0x55af2dc8a210 2022-07-20T02:34:11.792 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af2dc996a0 con 0x55af2dc84dc0 2022-07-20T02:34:11.793 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.793 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.159:43674/0 (socket says 172.21.15.159:43674) 2022-07-20T02:34:11.793 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 -- 172.21.15.159:0/3410113572 learned_addr learned my addr 172.21.15.159:0/3410113572 (peer_addr_for_me v2:172.21.15.159:0/0) 2022-07-20T02:34:11.794 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc551a5c700 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84dc0 0x55af2dc8e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.794 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc55225d700 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc84440 0x55af2dc8dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.794 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 -- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84dc0 msgr2=0x55af2dc8e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:11.795 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84dc0 0x55af2dc8e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.795 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 -- 172.21.15.159:0/3410113572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc84440 msgr2=0x55af2dc8dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:11.795 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc84440 0x55af2dc8dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.795 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 -- 172.21.15.159:0/3410113572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c005cd0 con 0x55af2dc8a210 2022-07-20T02:34:11.796 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc552a5e700 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fc54c001490 tx=0x7fc54c006950).ready entity=mon.1 client_cookie=6b193f6560bef9f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:11.796 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc54bfff700 1 -- 172.21.15.159:0/3410113572 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc54c006110 con 0x55af2dc8a210 2022-07-20T02:34:11.796 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc54bfff700 1 -- 172.21.15.159:0/3410113572 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c006270 con 0x55af2dc8a210 2022-07-20T02:34:11.796 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc54bfff700 1 -- 172.21.15.159:0/3410113572 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc54c004730 con 0x55af2dc8a210 2022-07-20T02:34:11.797 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 msgr2=0x55af2dc8e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.797 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fc54c001490 tx=0x7fc54c006950).stop 2022-07-20T02:34:11.797 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- 172.21.15.159:0/3410113572 shutdown_connections 2022-07-20T02:34:11.797 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc8a210 0x55af2dc8e760 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.798 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84dc0 0x55af2dc8e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.798 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 --2- 172.21.15.159:0/3410113572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc84440 0x55af2dc8dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.798 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- 172.21.15.159:0/3410113572 >> 172.21.15.159:0/3410113572 conn(0x55af2dbf7cf0 msgr2=0x55af2dc87c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:11.799 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- 172.21.15.159:0/3410113572 shutdown_connections 2022-07-20T02:34:11.799 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- 172.21.15.159:0/3410113572 wait complete. 2022-07-20T02:34:11.799 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 Processor -- start 2022-07-20T02:34:11.799 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.741+0000 7fc553e20b80 1 -- start start 2022-07-20T02:34:11.800 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84440 0x55af2dc98e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.800 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 0x55af2dc93ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.800 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc8a210 0x55af2dc944e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.801 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2dc88c60 con 0x55af2dc8a210 2022-07-20T02:34:11.801 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2dc94a90 con 0x55af2dc84dc0 2022-07-20T02:34:11.801 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af2dc94c50 con 0x55af2dc84440 2022-07-20T02:34:11.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 0x55af2dc93ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 0x55af2dc93ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.159:43680/0 (socket says 172.21.15.159:43680) 2022-07-20T02:34:11.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 -- 172.21.15.159:0/1581970504 learned_addr learned my addr 172.21.15.159:0/1581970504 (peer_addr_for_me v2:172.21.15.159:0/0) 2022-07-20T02:34:11.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc552a5e700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc8a210 0x55af2dc944e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.803 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc55225d700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84440 0x55af2dc98e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.803 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 -- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84440 msgr2=0x55af2dc98e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.803 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af2dc84440 0x55af2dc98e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.804 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 -- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc8a210 msgr2=0x55af2dc944e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:11.804 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55af2dc8a210 0x55af2dc944e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:11.804 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 -- 172.21.15.159:0/1581970504 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540001e10 con 0x55af2dc84dc0 2022-07-20T02:34:11.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc551a5c700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 0x55af2dc93ff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fc53c00a000 tx=0x7fc53c00c040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:11.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc53c00f040 con 0x55af2dc84dc0 2022-07-20T02:34:11.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c008e70 con 0x55af2dc84dc0 2022-07-20T02:34:11.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc544005cd0 con 0x55af2dc84dc0 2022-07-20T02:34:11.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc53c010aa0 con 0x55af2dc84dc0 2022-07-20T02:34:11.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.745+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af2db985d0 con 0x55af2dc84dc0 2022-07-20T02:34:11.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc53c010c00 con 0x55af2dc84dc0 2022-07-20T02:34:11.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc54b7fe700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc5340525f0 0x7fc534054ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.807 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6422+0+0 (secure 0 0 0) 0x7fc53c06b5b0 con 0x55af2dc84dc0 2022-07-20T02:34:11.807 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc55225d700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc5340525f0 0x7fc534054ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.808 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc553e20b80 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x55af2dca1c70 0x55af2dca4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:11.808 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 --> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] -- osd_op(unknown.0.0:1 23.0 23:0620e6f9:::key:head [writefull 0~617 in=617b] snapc 0=[] ondisk+write+known_if_redirected e125) v8 -- 0x55af2dca45f0 con 0x55af2dca1c70 2022-07-20T02:34:11.808 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc552a5e700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x55af2dca1c70 0x55af2dca4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:11.808 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc55225d700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc5340525f0 0x7fc534054ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fc544018000 tx=0x7fc544014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:11.809 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:11.749+0000 7fc552a5e700 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x55af2dca1c70 0x55af2dca4060 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:12.785 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.753+0000 7fc54b7fe700 1 -- 172.21.15.159:0/1581970504 <== osd.0 v2:172.21.15.26:6802/23049 1 ==== osd_map(126..126 src has 1..126) v4 ==== 320+0+0 (crc 0 0 0) 0x7fc54c0044f0 con 0x55af2dca1c70 2022-07-20T02:34:12.793 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc552a5e700 1 -- 172.21.15.159:0/1581970504 <== osd.0 v2:172.21.15.26:6802/23049 2 ==== osd_op_reply(1 key [writefull 0~617] v126'1 uv1 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7fc54c0046b0 con 0x55af2dca1c70 2022-07-20T02:34:12.799 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x55af2dca1c70 msgr2=0x55af2dca4060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.801 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x55af2dca1c70 0x55af2dca4060 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:12.801 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc5340525f0 msgr2=0x7fc534054ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc5340525f0 0x7fc534054ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fc544018000 tx=0x7fc544014000).stop 2022-07-20T02:34:12.802 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 msgr2=0x55af2dc93ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.804 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 --2- 172.21.15.159:0/1581970504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af2dc84dc0 0x55af2dc93ff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fc53c00a000 tx=0x7fc53c00c040).stop 2022-07-20T02:34:12.804 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc552a5e700 1 -- 172.21.15.159:0/1581970504 reap_dead start 2022-07-20T02:34:12.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 shutdown_connections 2022-07-20T02:34:12.805 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 >> 172.21.15.159:0/1581970504 conn(0x55af2dbf7cf0 msgr2=0x55af2dc87c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:12.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 shutdown_connections 2022-07-20T02:34:12.806 INFO:teuthology.orchestra.run.smithi159.stderr:2022-07-20T02:34:12.757+0000 7fc553e20b80 1 -- 172.21.15.159:0/1581970504 wait complete. 2022-07-20T02:34:12.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T02:34:12.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 Processor -- start 2022-07-20T02:34:12.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 -- start start 2022-07-20T02:34:12.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f817c0952f0 0x7f817c060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:12.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c1485b0 con 0x7f817c05f840 2022-07-20T02:34:12.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c14b730 con 0x7f817c0b30c0 2022-07-20T02:34:12.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817b59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c095a50 con 0x7f817c0952f0 2022-07-20T02:34:12.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817ad9d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f817c05f840 0x7f817c066f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41718/0 (socket says 172.21.15.26:41718) 2022-07-20T02:34:12.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f817ad9d700 1 -- 172.21.15.26:0/3336311028 learned_addr learned my addr 172.21.15.26:0/3336311028 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:12.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.935+0000 7f8179d9b700 1 --2- 172.21.15.26:0/3336311028 >> v2:172.21.15.120:3300/0 conn(0x7f817c0952f0 0x7f817c060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:12.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706289859 0 0) 0x7f817c1485b0 con 0x7f817c05f840 2022-07-20T02:34:12.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f815c003260 con 0x7f817c05f840 2022-07-20T02:34:12.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962797428 0 0) 0x7f817c14b730 con 0x7f817c0b30c0 2022-07-20T02:34:12.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c1485b0 con 0x7f817c0b30c0 2022-07-20T02:34:12.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f8179d9b700 1 -- 172.21.15.26:0/3336311028 >> v1:172.21.15.120:6789/0 conn(0x7f817c0b30c0 legacy=0x7f817c147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f8179d9b700 1 -- 172.21.15.26:0/3336311028 >> v1:172.21.15.26:6789/0 conn(0x7f817c05f840 legacy=0x7f817c066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f8179d9b700 1 -- 172.21.15.26:0/3336311028 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c14b9d0 con 0x7f817c0952f0 2022-07-20T02:34:12.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f8179d9b700 1 --2- 172.21.15.26:0/3336311028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c060570 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f817c147680 tx=0x7f81640091d0).ready entity=mon.1 client_cookie=8bf6ac8eaf9d27cd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:12.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f816400b600 con 0x7f817c0952f0 2022-07-20T02:34:12.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816400b760 con 0x7f817c0952f0 2022-07-20T02:34:12.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817959a700 1 -- 172.21.15.26:0/3336311028 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f816400b8d0 con 0x7f817c0952f0 2022-07-20T02:34:12.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 -- 172.21.15.26:0/3336311028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 msgr2=0x7f817c060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:12.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 --2- 172.21.15.26:0/3336311028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c060570 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f817c147680 tx=0x7f81640091d0).stop 2022-07-20T02:34:12.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 -- 172.21.15.26:0/3336311028 shutdown_connections 2022-07-20T02:34:12.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 --2- 172.21.15.26:0/3336311028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c060570 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:12.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 -- 172.21.15.26:0/3336311028 >> 172.21.15.26:0/3336311028 conn(0x7f817c0b67f0 msgr2=0x7f817c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:12.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 -- 172.21.15.26:0/3336311028 shutdown_connections 2022-07-20T02:34:12.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.939+0000 7f817b59e700 1 -- 172.21.15.26:0/3336311028 wait complete. 2022-07-20T02:34:12.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 Processor -- start 2022-07-20T02:34:12.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 -- start start 2022-07-20T02:34:12.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:12.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:12.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f817c0b30c0 0x7f817c062f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:12.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c156910 con 0x7f817c05f840 2022-07-20T02:34:12.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c09b790 con 0x7f817c0952f0 2022-07-20T02:34:12.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817b59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c156b70 con 0x7f817c0b30c0 2022-07-20T02:34:12.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:13.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42440/0 (socket says 172.21.15.26:42440) 2022-07-20T02:34:13.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817a59c700 1 -- 172.21.15.26:0/4220976335 learned_addr learned my addr 172.21.15.26:0/4220976335 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:13.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f8179d9b700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:13.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817ad9d700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f817c0b30c0 0x7f817c062f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817a59c700 1 -- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f817c0b30c0 msgr2=0x7f817c062f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.943+0000 7f817a59c700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f817c0b30c0 0x7f817c062f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817a59c700 1 -- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 msgr2=0x7f817c1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817a59c700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817a59c700 1 -- 172.21.15.26:0/4220976335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009470 con 0x7f817c05f840 2022-07-20T02:34:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817a59c700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f816c001110 tx=0x7f816c007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f816c012400 con 0x7f817c05f840 2022-07-20T02:34:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c012560 con 0x7f817c05f840 2022-07-20T02:34:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c156e10 con 0x7f817c05f840 2022-07-20T02:34:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f816c0128c0 con 0x7f817c05f840 2022-07-20T02:34:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c063440 con 0x7f817c05f840 2022-07-20T02:34:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c0b00d0 con 0x7f817c05f840 2022-07-20T02:34:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f816c019030 con 0x7f817c05f840 2022-07-20T02:34:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 0x7f8160055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f816c06b440 con 0x7f817c05f840 2022-07-20T02:34:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.947+0000 7f8179d9b700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 0x7f8160055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.951+0000 7f8179d9b700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 0x7f8160055e10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f816401b000 tx=0x7f8164016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:12.959+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f817c0b00d0 con 0x7f817c05f840 2022-07-20T02:34:13.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f817c0b00d0 con 0x7f817c05f840 2022-07-20T02:34:13.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f81737fe700 1 -- 172.21.15.26:0/4220976335 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1588 (secure 0 0 0) 0x7f817c0b00d0 con 0x7f817c05f840 2022-07-20T02:34:13.143 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:13.144 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:13.144 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:13.145 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:34:13.145 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764491902976, 2022-07-20T02:34:13.146 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 12275712, 2022-07-20T02:34:13.151 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8602210304, 2022-07-20T02:34:13.151 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011126989498734474, 2022-07-20T02:34:13.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num_osds": 8, 2022-07-20T02:34:13.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_osds": 8, 2022-07-20T02:34:13.153 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_omap_osds": 8 2022-07-20T02:34:13.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:13.153 INFO:teuthology.orchestra.run.smithi026.stdout: "stats_by_class": { 2022-07-20T02:34:13.154 INFO:teuthology.orchestra.run.smithi026.stdout: "ssd": { 2022-07-20T02:34:13.154 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:34:13.155 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764491902976, 2022-07-20T02:34:13.155 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 12275712, 2022-07-20T02:34:13.155 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8602210304, 2022-07-20T02:34:13.156 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011126989498734474 2022-07-20T02:34:13.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:13.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:13.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:34:13.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:13.158 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_health_metrics", 2022-07-20T02:34:13.158 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 1, 2022-07-20T02:34:13.158 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:13.159 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:13.159 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 2, 2022-07-20T02:34:13.160 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:13.160 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:13.161 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:13.161 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918477824 2022-07-20T02:34:13.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:13.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:13.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:13.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd", 2022-07-20T02:34:13.163 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 2, 2022-07-20T02:34:13.164 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:13.164 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:13.164 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:34:13.165 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:13.165 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:13.166 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:13.166 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918477824 2022-07-20T02:34:13.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:13.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:13.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:13.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephfs_data", 2022-07-20T02:34:13.169 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 22, 2022-07-20T02:34:13.169 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:13.170 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:13.170 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:34:13.171 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:13.171 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:13.171 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:13.172 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918477824 2022-07-20T02:34:13.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:13.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:13.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:13.182 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:34:13.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 msgr2=0x7f8160055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:13.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 0x7f8160055e10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f816401b000 tx=0x7f8164016000).stop 2022-07-20T02:34:13.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 msgr2=0x7f817c09b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:13.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f816c001110 tx=0x7f816c007b60).stop 2022-07-20T02:34:13.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 shutdown_connections 2022-07-20T02:34:13.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8160053950 0x7f8160055e10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f817c0b30c0 0x7f817c062f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f817c0952f0 0x7f817c1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 --2- 172.21.15.26:0/4220976335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f817c05f840 0x7f817c09b0f0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:13.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 >> 172.21.15.26:0/4220976335 conn(0x7f817c0b67f0 msgr2=0x7f817c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:13.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 shutdown_connections 2022-07-20T02:34:13.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:13.127+0000 7f817b59e700 1 -- 172.21.15.26:0/4220976335 wait complete. 2022-07-20T02:34:13.217 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30)... 2022-07-20T02:34:17.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:17.835 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:17.856 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:17.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:18.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T02:34:18.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 Processor -- start 2022-07-20T02:34:18.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- start start 2022-07-20T02:34:18.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd7dc067900 0x7fd7dc14a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:18.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc05ff70 con 0x7fd7dc061860 2022-07-20T02:34:18.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc0606e0 con 0x7fd7dc0b3260 2022-07-20T02:34:18.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7dc067d90 con 0x7fd7dc067900 2022-07-20T02:34:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e1cd1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd7dc061860 0x7fd7dc0980d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41728/0 (socket says 172.21.15.26:41728) 2022-07-20T02:34:18.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e1cd1700 1 -- 172.21.15.26:0/2617190595 learned_addr learned my addr 172.21.15.26:0/2617190595 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:18.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e2cd3700 1 --2- 172.21.15.26:0/2617190595 >> v2:172.21.15.26:3300/0 conn(0x7fd7dc067900 0x7fd7dc14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:18.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e2cd3700 1 -- 172.21.15.26:0/2617190595 >> v1:172.21.15.120:6789/0 conn(0x7fd7dc0b3260 legacy=0x7fd7dc0979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e2cd3700 1 -- 172.21.15.26:0/2617190595 >> v1:172.21.15.26:6789/0 conn(0x7fd7dc061860 legacy=0x7fd7dc0980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e14d0700 1 -- 172.21.15.26:0/2617190595 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023968384 0 0) 0x7fd7dc05ff70 con 0x7fd7dc061860 2022-07-20T02:34:18.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e2cd3700 1 -- 172.21.15.26:0/2617190595 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc0b3640 con 0x7fd7dc067900 2022-07-20T02:34:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e2cd3700 1 --2- 172.21.15.26:0/2617190595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc14a000 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7fd7dc05fb20 tx=0x7fd7d0006df0).ready entity=mon.0 client_cookie=6d0e3b6dce87ee5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:18.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e14d0700 1 -- 172.21.15.26:0/2617190595 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7d0014070 con 0x7fd7dc067900 2022-07-20T02:34:18.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e14d0700 1 -- 172.21.15.26:0/2617190595 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d0000e90 con 0x7fd7dc067900 2022-07-20T02:34:18.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e14d0700 1 -- 172.21.15.26:0/2617190595 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7d000eb00 con 0x7fd7dc067900 2022-07-20T02:34:18.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- 172.21.15.26:0/2617190595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 msgr2=0x7fd7dc14a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/2617190595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc14a000 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7fd7dc05fb20 tx=0x7fd7d0006df0).stop 2022-07-20T02:34:18.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- 172.21.15.26:0/2617190595 shutdown_connections 2022-07-20T02:34:18.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/2617190595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc14a000 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.379+0000 7fd7e34d4700 1 -- 172.21.15.26:0/2617190595 >> 172.21.15.26:0/2617190595 conn(0x7fd7dc0b67f0 msgr2=0x7fd7dc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:18.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- 172.21.15.26:0/2617190595 shutdown_connections 2022-07-20T02:34:18.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- 172.21.15.26:0/2617190595 wait complete. 2022-07-20T02:34:18.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 Processor -- start 2022-07-20T02:34:18.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- start start 2022-07-20T02:34:18.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7dc061860 0x7fd7dc153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7dc0b3260 0x7fd7dc0aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc154920 con 0x7fd7dc067900 2022-07-20T02:34:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0993d0 con 0x7fd7dc0b3260 2022-07-20T02:34:18.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e34d4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc1543e0 con 0x7fd7dc061860 2022-07-20T02:34:18.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:18.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42456/0 (socket says 172.21.15.26:42456) 2022-07-20T02:34:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 -- 172.21.15.26:0/482609366 learned_addr learned my addr 172.21.15.26:0/482609366 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e2cd3700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7dc0b3260 0x7fd7dc0aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e24d2700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7dc061860 0x7fd7dc153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 -- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7dc061860 msgr2=0x7fd7dc153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7dc061860 0x7fd7dc153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 -- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7dc0b3260 msgr2=0x7fd7dc0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7dc0b3260 0x7fd7dc0aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 -- 172.21.15.26:0/482609366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d000b180 con 0x7fd7dc067900 2022-07-20T02:34:18.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.383+0000 7fd7e1cd1700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7fd7d4000f40 tx=0x7fd7d4007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:18.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7d4005ba0 con 0x7fd7dc067900 2022-07-20T02:34:18.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d4005d00 con 0x7fd7dc067900 2022-07-20T02:34:18.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc154680 con 0x7fd7dc067900 2022-07-20T02:34:18.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7d40072c0 con 0x7fd7dc067900 2022-07-20T02:34:18.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc09aee0 con 0x7fd7dc067900 2022-07-20T02:34:18.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc0601a0 con 0x7fd7dc067900 2022-07-20T02:34:18.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd7d401a060 con 0x7fd7dc067900 2022-07-20T02:34:18.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 0x7fd7bc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6422+0+0 (secure 0 0 0) 0x7fd7d406b6a0 con 0x7fd7dc067900 2022-07-20T02:34:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7e24d2700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 0x7fd7bc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.387+0000 7fd7e24d2700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 0x7fd7bc055ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fd7d800d000 tx=0x7fd7d8007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.395+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd7d4042460 con 0x7fd7dc067900 2022-07-20T02:34:18.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fd7dc0574e0 con 0x7fd7dc067900 2022-07-20T02:34:18.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7cf7fe700 1 -- 172.21.15.26:0/482609366 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1932 (secure 0 0 0) 0x7fd7d4018020 con 0x7fd7dc067900 2022-07-20T02:34:18.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:18.703 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:18.704 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:34:18.705 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764491374592, 2022-07-20T02:34:18.745 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 12804096, 2022-07-20T02:34:18.746 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8602738688, 2022-07-20T02:34:18.748 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011127673089504242, 2022-07-20T02:34:18.780 INFO:teuthology.orchestra.run.smithi026.stdout: "num_osds": 8, 2022-07-20T02:34:18.780 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_osds": 8, 2022-07-20T02:34:18.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num_per_pool_omap_osds": 8 2022-07-20T02:34:18.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "stats_by_class": { 2022-07-20T02:34:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "ssd": { 2022-07-20T02:34:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "total_bytes": 773094113280, 2022-07-20T02:34:18.810 INFO:teuthology.orchestra.run.smithi026.stdout: "total_avail_bytes": 764491374592, 2022-07-20T02:34:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_bytes": 12804096, 2022-07-20T02:34:18.811 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_bytes": 8602738688, 2022-07-20T02:34:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: "total_used_raw_ratio": 0.011127673089504242 2022-07-20T02:34:18.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:18.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:34:18.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_health_metrics", 2022-07-20T02:34:18.835 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 1, 2022-07-20T02:34:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:18.836 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 2, 2022-07-20T02:34:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:18.837 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:18.869 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918313984 2022-07-20T02:34:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd", 2022-07-20T02:34:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 2, 2022-07-20T02:34:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:34:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918313984 2022-07-20T02:34:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephfs_data", 2022-07-20T02:34:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 22, 2022-07-20T02:34:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 0, 2022-07-20T02:34:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 0, 2022-07-20T02:34:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 0, 2022-07-20T02:34:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 0, 2022-07-20T02:34:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 0, 2022-07-20T02:34:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918313984 2022-07-20T02:34:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephfs_metadata", 2022-07-20T02:34:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 23, 2022-07-20T02:34:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "stats": { 2022-07-20T02:34:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "stored": 617, 2022-07-20T02:34:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "objects": 1, 2022-07-20T02:34:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "kb_used": 8, 2022-07-20T02:34:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "bytes_used": 8192, 2022-07-20T02:34:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "percent_used": 1.1286286571987603e-08, 2022-07-20T02:34:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "max_avail": 362918313984 2022-07-20T02:34:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:18.933 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:34:18.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 msgr2=0x7fd7bc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 0x7fd7bc055ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fd7d800d000 tx=0x7fd7d8007b90).stop 2022-07-20T02:34:18.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 msgr2=0x7fd7dc098ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:18.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7fd7d4000f40 tx=0x7fd7d4007f70).stop 2022-07-20T02:34:18.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 shutdown_connections 2022-07-20T02:34:18.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7dc067900 0x7fd7dc098ee0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd7bc0539e0 0x7fd7bc055ea0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7dc0b3260 0x7fd7dc0aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 --2- 172.21.15.26:0/482609366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7dc061860 0x7fd7dc153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:18.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 >> 172.21.15.26:0/482609366 conn(0x7fd7dc0b67f0 msgr2=0x7fd7dc148010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:18.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 shutdown_connections 2022-07-20T02:34:18.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:18.563+0000 7fd7e34d4700 1 -- 172.21.15.26:0/482609366 wait complete. 2022-07-20T02:34:18.939 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2022-07-20T02:34:18.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-20T02:34:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 Processor -- start 2022-07-20T02:34:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 -- start start 2022-07-20T02:34:19.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d00b0050 0x7f67d00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 0x7f67d0143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f67d0143e00 0x7f67d00b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d00a9e10 con 0x7f67d0143e00 2022-07-20T02:34:19.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d00617a0 con 0x7f67d005d050 2022-07-20T02:34:19.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67d504d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00aa040 con 0x7f67d00b0050 2022-07-20T02:34:19.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f67d0143e00 0x7f67d00b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f67d0143e00 0x7f67d00b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42466/0 (socket says 172.21.15.26:42466) 2022-07-20T02:34:19.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 -- 172.21.15.26:0/2367247689 learned_addr learned my addr 172.21.15.26:0/2367247689 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67c659c700 1 --2- 172.21.15.26:0/2367247689 >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 0x7f67d0143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67ced9d700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d00b0050 0x7f67d00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 -- 172.21.15.26:0/2367247689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d00b0050 msgr2=0x7f67d00b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:19.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d00b0050 0x7f67d00b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 -- 172.21.15.26:0/2367247689 >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 msgr2=0x7f67d0143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:19.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 --2- 172.21.15.26:0/2367247689 >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 0x7f67d0143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 -- 172.21.15.26:0/2367247689 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d00aa2e0 con 0x7f67d0143e00 2022-07-20T02:34:19.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67c659c700 1 --2- 172.21.15.26:0/2367247689 >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 0x7f67d0143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:19.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67cf59e700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d0143e00 0x7f67d00b10d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f67c80032f0 tx=0x7f67c8009290).ready entity=mon.0 client_cookie=3b313cb04e4a9c89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67ce59c700 1 -- 172.21.15.26:0/2367247689 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67c800ba60 con 0x7f67d0143e00 2022-07-20T02:34:19.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67ce59c700 1 -- 172.21.15.26:0/2367247689 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c800bbc0 con 0x7f67d0143e00 2022-07-20T02:34:19.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.063+0000 7f67ce59c700 1 -- 172.21.15.26:0/2367247689 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67c8010470 con 0x7f67d0143e00 2022-07-20T02:34:19.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- 172.21.15.26:0/2367247689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d0143e00 msgr2=0x7f67d00b10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d0143e00 0x7f67d00b10d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f67c80032f0 tx=0x7f67c8009290).stop 2022-07-20T02:34:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- 172.21.15.26:0/2367247689 shutdown_connections 2022-07-20T02:34:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d0143e00 0x7f67d00b10d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- 172.21.15.26:0/2367247689 >> v2:172.21.15.120:3300/0 conn(0x7f67d005d050 0x7f67d0143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- 172.21.15.26:0/2367247689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d00b0050 0x7f67d00b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- 172.21.15.26:0/2367247689 >> 172.21.15.26:0/2367247689 conn(0x7f67d0060950 msgr2=0x7f67d0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- 172.21.15.26:0/2367247689 shutdown_connections 2022-07-20T02:34:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- 172.21.15.26:0/2367247689 wait complete. 2022-07-20T02:34:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 Processor -- start 2022-07-20T02:34:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- start start 2022-07-20T02:34:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67d005d050 0x7f67d00aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 0x7f67d0140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d0143e00 0x7f67d0140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d0156a60 con 0x7f67d00b0050 2022-07-20T02:34:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00a3020 con 0x7f67d005d050 2022-07-20T02:34:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.067+0000 7f67d504d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00a32c0 con 0x7f67d0143e00 2022-07-20T02:34:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67c659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 0x7f67d0140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67d005d050 0x7f67d00aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67c659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 0x7f67d0140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42470/0 (socket says 172.21.15.26:42470) 2022-07-20T02:34:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67d005d050 0x7f67d00aca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:33928/0 (socket says 172.21.15.26:33928) 2022-07-20T02:34:19.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67cf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d0143e00 0x7f67d0140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67c659c700 1 -- 172.21.15.26:0/735016016 learned_addr learned my addr 172.21.15.26:0/735016016 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 -- 172.21.15.26:0/735016016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d0143e00 msgr2=0x7f67d0140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d0143e00 0x7f67d0140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 -- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 msgr2=0x7f67d0140a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 0x7f67d0140a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 -- 172.21.15.26:0/735016016 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c8009530 con 0x7f67d005d050 2022-07-20T02:34:19.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67c659c700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f67d00b0050 0x7f67d0140a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:19.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67cf59e700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67d0143e00 0x7f67d0140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:19.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ced9d700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67d005d050 0x7f67d00aca30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f67c000a000 tx=0x7f67c0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67c0011590 con 0x7f67d005d050 2022-07-20T02:34:19.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c00116f0 con 0x7f67d005d050 2022-07-20T02:34:19.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67d504d700 1 -- 172.21.15.26:0/735016016 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d00a3560 con 0x7f67d005d050 2022-07-20T02:34:19.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67c0011a50 con 0x7f67d005d050 2022-07-20T02:34:19.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67d504d700 1 -- 172.21.15.26:0/735016016 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d0065720 con 0x7f67d005d050 2022-07-20T02:34:19.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.071+0000 7f67d504d700 1 -- 172.21.15.26:0/735016016 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d00a36f0 con 0x7f67d005d050 2022-07-20T02:34:19.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.075+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f67c0011bb0 con 0x7f67d005d050 2022-07-20T02:34:19.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.075+0000 7f67ccd99700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67b4053980 0x7f67b4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.075+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f67c006c280 con 0x7f67d005d050 2022-07-20T02:34:19.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.075+0000 7f67c659c700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67b4053980 0x7f67b4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.075+0000 7f67c659c700 1 --2- 172.21.15.26:0/735016016 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f67b4053980 0x7f67b4055e40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f67b800c000 tx=0x7f67b8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.083+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67c0042fe0 con 0x7f67d005d050 2022-07-20T02:34:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-07-20T02:34:19.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.251+0000 7f67d504d700 1 -- 172.21.15.26:0/735016016 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f67d00ae520 con 0x7f67d005d050 2022-07-20T02:34:19.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.251+0000 7f67ccd99700 1 -- 172.21.15.26:0/735016016 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=-22 pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. v63) v1 ==== 209+0+0 (secure 0 0 0) 0x7f67c0046600 con 0x7f67d005d050 2022-07-20T02:34:19.305 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. 2022-07-20T02:34:19.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:34:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 Processor -- start 2022-07-20T02:34:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 -- start start 2022-07-20T02:34:19.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f540b06d0 0x7f1f5413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f5413fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f540b1090 con 0x7f1f540a9520 2022-07-20T02:34:19.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f540b0bd0 con 0x7f1f540b06d0 2022-07-20T02:34:19.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f5a6d9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f540b12c0 con 0x7f1f540ac8e0 2022-07-20T02:34:19.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.423+0000 7f1f58ed6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f540b06d0 0x7f1f5413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f540b06d0 0x7f1f5413f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42478/0 (socket says 172.21.15.26:42478) 2022-07-20T02:34:19.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 -- 172.21.15.26:0/2735144172 learned_addr learned my addr 172.21.15.26:0/2735144172 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f59ed8700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f5413fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 -- 172.21.15.26:0/2735144172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 msgr2=0x7f1f5413fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:19.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f5413fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 -- 172.21.15.26:0/2735144172 >> v1:172.21.15.120:6789/0 conn(0x7f1f540a9520 legacy=0x7f1f5405c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 -- 172.21.15.26:0/2735144172 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540b1560 con 0x7f1f540b06d0 2022-07-20T02:34:19.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f5413f6e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f1f44002f70 tx=0x7f1f44006eb0).ready entity=mon.0 client_cookie=3a2b8ab218902e11 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f43fff700 1 -- 172.21.15.26:0/2735144172 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f4400d5c0 con 0x7f1f540b06d0 2022-07-20T02:34:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f43fff700 1 -- 172.21.15.26:0/2735144172 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f44000e90 con 0x7f1f540b06d0 2022-07-20T02:34:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f43fff700 1 -- 172.21.15.26:0/2735144172 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f4400d950 con 0x7f1f540b06d0 2022-07-20T02:34:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2735144172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 msgr2=0x7f1f5413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f5413f6e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f1f44002f70 tx=0x7f1f44006eb0).stop 2022-07-20T02:34:19.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2735144172 shutdown_connections 2022-07-20T02:34:19.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f5413fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2735144172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f5413f6e0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2735144172 >> 172.21.15.26:0/2735144172 conn(0x7f1f54060310 msgr2=0x7f1f540b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:19.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2735144172 shutdown_connections 2022-07-20T02:34:19.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2735144172 wait complete. 2022-07-20T02:34:19.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 Processor -- start 2022-07-20T02:34:19.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- start start 2022-07-20T02:34:19.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f540a9520 0x7f1f54143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f541535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f541556f0 con 0x7f1f540b06d0 2022-07-20T02:34:19.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54155990 con 0x7f1f540a9520 2022-07-20T02:34:19.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f5a6d9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f54155c30 con 0x7f1f540ac8e0 2022-07-20T02:34:19.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f59ed8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f59ed8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42486/0 (socket says 172.21.15.26:42486) 2022-07-20T02:34:19.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f59ed8700 1 -- 172.21.15.26:0/2147156260 learned_addr learned my addr 172.21.15.26:0/2147156260 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f58ed6700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f541535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.427+0000 7f1f596d7700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f540a9520 0x7f1f54143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 -- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 msgr2=0x7f1f541535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f541535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 -- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f540a9520 msgr2=0x7f1f54143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f540a9520 0x7f1f54143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 -- 172.21.15.26:0/2147156260 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4400b1e0 con 0x7f1f540b06d0 2022-07-20T02:34:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f59ed8700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f1f4c00b000 tx=0x7f1f4c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f4c0116f0 con 0x7f1f540b06d0 2022-07-20T02:34:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c011850 con 0x7f1f540b06d0 2022-07-20T02:34:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f54155ed0 con 0x7f1f540b06d0 2022-07-20T02:34:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f4c011bb0 con 0x7f1f540b06d0 2022-07-20T02:34:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f541432d0 con 0x7f1f540b06d0 2022-07-20T02:34:19.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.431+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f54156060 con 0x7f1f540b06d0 2022-07-20T02:34:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.435+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f4c017020 con 0x7f1f540b06d0 2022-07-20T02:34:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.435+0000 7f1f427fc700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 0x7f1f34055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.435+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f1f4c06c710 con 0x7f1f540b06d0 2022-07-20T02:34:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.435+0000 7f1f596d7700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 0x7f1f34055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.435+0000 7f1f596d7700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 0x7f1f34055e10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f1f5000d000 tx=0x7f1f50007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.439+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f4c043470 con 0x7f1f540b06d0 2022-07-20T02:34:19.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.607+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1f540574e0 con 0x7f1f540b06d0 2022-07-20T02:34:19.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:34:19.607+0000 7f558fb37700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:34:19.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.615+0000 7f1f427fc700 1 -- 172.21.15.26:0/2147156260 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 22 v64) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1f4c046a90 con 0x7f1f540b06d0 2022-07-20T02:34:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.615+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 msgr2=0x7f1f34055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.615+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 0x7f1f34055e10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f1f5000d000 tx=0x7f1f50007b90).stop 2022-07-20T02:34:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.615+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 msgr2=0x7f1f54153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.615+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f1f4c00b000 tx=0x7f1f4c008040).stop 2022-07-20T02:34:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 shutdown_connections 2022-07-20T02:34:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f540b06d0 0x7f1f54153ae0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f34053950 0x7f1f34055e10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f540ac8e0 0x7f1f541535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 --2- 172.21.15.26:0/2147156260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f540a9520 0x7f1f54143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 >> 172.21.15.26:0/2147156260 conn(0x7f1f54060310 msgr2=0x7f1f54066b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 shutdown_connections 2022-07-20T02:34:19.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.619+0000 7f1f5a6d9700 1 -- 172.21.15.26:0/2147156260 wait complete. 2022-07-20T02:34:19.734 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 23 and data pool 22 2022-07-20T02:34:19.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:34:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 Processor -- start 2022-07-20T02:34:19.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 -- start start 2022-07-20T02:34:19.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1bd40adf90 0x7f1bd40b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd40ae370 con 0x7f1bd4060e40 2022-07-20T02:34:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd40b28b0 con 0x7f1bd40adf90 2022-07-20T02:34:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bdc0cf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40b0610 con 0x7f1bd40b14b0 2022-07-20T02:34:19.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bd9e6b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1bd40adf90 0x7f1bd40b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bd9e6b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1bd40adf90 0x7f1bd40b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42490/0 (socket says 172.21.15.26:42490) 2022-07-20T02:34:19.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bda66c700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1bd4060e40 0x7f1bd40614b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:55986/0 (socket says 172.21.15.26:55986) 2022-07-20T02:34:19.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/148663515 learned_addr learned my addr 172.21.15.26:0/148663515 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.871+0000 7f1bd966a700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/148663515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 msgr2=0x7f1bd4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:19.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd8e69700 1 -- 172.21.15.26:0/148663515 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339508081 0 0) 0x7f1bd40ae370 con 0x7f1bd4060e40 2022-07-20T02:34:19.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd9e6b700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/148663515 >> v1:172.21.15.120:6789/0 conn(0x7f1bd4060e40 legacy=0x7f1bd40614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/148663515 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40b08b0 con 0x7f1bd40adf90 2022-07-20T02:34:19.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd9e6b700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd40adf90 0x7f1bd40b2c60 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0006eb0).ready entity=mon.0 client_cookie=4892ca5797fa94d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd8e69700 1 -- 172.21.15.26:0/148663515 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd000d570 con 0x7f1bd40adf90 2022-07-20T02:34:19.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd8e69700 1 -- 172.21.15.26:0/148663515 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0000e90 con 0x7f1bd40adf90 2022-07-20T02:34:19.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bd8e69700 1 -- 172.21.15.26:0/148663515 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd000d8c0 con 0x7f1bd40adf90 2022-07-20T02:34:19.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/148663515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd40adf90 msgr2=0x7f1bd40b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd40adf90 0x7f1bd40b2c60 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0006eb0).stop 2022-07-20T02:34:19.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/148663515 shutdown_connections 2022-07-20T02:34:19.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/148663515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd40adf90 0x7f1bd40b2c60 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/148663515 >> 172.21.15.26:0/148663515 conn(0x7f1bd40b66e0 msgr2=0x7f1bd40b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:19.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/148663515 shutdown_connections 2022-07-20T02:34:19.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/148663515 wait complete. 2022-07-20T02:34:19.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 Processor -- start 2022-07-20T02:34:19.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.875+0000 7f1bdc0cf700 1 -- start start 2022-07-20T02:34:19.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bd40adf90 0x7f1bd41501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd41537a0 con 0x7f1bd4060e40 2022-07-20T02:34:19.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4143b00 con 0x7f1bd40adf90 2022-07-20T02:34:19.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd41539d0 con 0x7f1bd40b14b0 2022-07-20T02:34:19.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42496/0 (socket says 172.21.15.26:42496) 2022-07-20T02:34:19.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/3278555546 learned_addr learned my addr 172.21.15.26:0/3278555546 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:19.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd966a700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bd40adf90 0x7f1bd41501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bda66c700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 msgr2=0x7f1bd4153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bd40adf90 msgr2=0x7f1bd41501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:19.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bd40adf90 0x7f1bd41501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:19.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 -- 172.21.15.26:0/3278555546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd000b1e0 con 0x7f1bd4060e40 2022-07-20T02:34:19.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bda66c700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:19.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bd9e6b700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd0011990 con 0x7f1bd4060e40 2022-07-20T02:34:19.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0011af0 con 0x7f1bd4060e40 2022-07-20T02:34:19.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd4143da0 con 0x7f1bd4060e40 2022-07-20T02:34:19.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd0011e50 con 0x7f1bd4060e40 2022-07-20T02:34:19.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd41435f0 con 0x7f1bd4060e40 2022-07-20T02:34:19.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.879+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd4153b90 con 0x7f1bd4060e40 2022-07-20T02:34:19.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.883+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1bd0025030 con 0x7f1bd4060e40 2022-07-20T02:34:19.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.883+0000 7f1bc2ffd700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 0x7f1bb4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:19.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.883+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f1bd006d200 con 0x7f1bd4060e40 2022-07-20T02:34:19.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.883+0000 7f1bd966a700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 0x7f1bb4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:19.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.883+0000 7f1bd966a700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 0x7f1bb4055e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f1bc400b000 tx=0x7f1bc4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:19.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:19.887+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1bd0047e60 con 0x7f1bd4060e40 2022-07-20T02:34:20.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.055+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1bd40574e0 con 0x7f1bd4060e40 2022-07-20T02:34:20.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.615+0000 7f1bc2ffd700 1 -- 172.21.15.26:0/3278555546 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1bd0043f70 con 0x7f1bd4060e40 2022-07-20T02:34:20.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 msgr2=0x7f1bb4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 0x7f1bb4055e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f1bc400b000 tx=0x7f1bc4008040).stop 2022-07-20T02:34:20.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 msgr2=0x7f1bd40b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0015000).stop 2022-07-20T02:34:20.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 shutdown_connections 2022-07-20T02:34:20.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1bb4053980 0x7f1bb4055e40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bd40b14b0 0x7f1bd4153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bd40adf90 0x7f1bd41501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 --2- 172.21.15.26:0/3278555546 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1bd4060e40 0x7f1bd40b54d0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 >> 172.21.15.26:0/3278555546 conn(0x7f1bd40b66e0 msgr2=0x7f1bd40b4870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:20.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 shutdown_connections 2022-07-20T02:34:20.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.619+0000 7f1bdc0cf700 1 -- 172.21.15.26:0/3278555546 wait complete. 2022-07-20T02:34:20.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:34:20.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 Processor -- start 2022-07-20T02:34:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 -- start start 2022-07-20T02:34:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 0x7f8d80143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8d80147000 0x7f8d800b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d801474e0 con 0x7f8d80060f00 2022-07-20T02:34:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8005d3b0 con 0x7f8d80147000 2022-07-20T02:34:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.759+0000 7f8d84fcc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801431e0 con 0x7f8d8005cfd0 2022-07-20T02:34:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8d80147000 0x7f8d800b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8d80060f00 0x7f8d8013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41788/0 (socket says 172.21.15.26:41788) 2022-07-20T02:34:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7ffff700 1 -- 172.21.15.26:0/291017417 learned_addr learned my addr 172.21.15.26:0/291017417 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7f7fe700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 0x7f8d80143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 -- 172.21.15.26:0/291017417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 msgr2=0x7f8d80143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 0x7f8d80143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 -- 172.21.15.26:0/291017417 >> v1:172.21.15.26:6789/0 conn(0x7f8d80060f00 legacy=0x7f8d8013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 -- 172.21.15.26:0/291017417 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d80143480 con 0x7f8d80147000 2022-07-20T02:34:20.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7e7fc700 1 -- 172.21.15.26:0/291017417 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450614047 0 0) 0x7f8d801474e0 con 0x7f8d80060f00 2022-07-20T02:34:20.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7f7fe700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 0x7f8d80143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7effd700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80147000 0x7f8d800b4730 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f8d68002f70 tx=0x7f8d68006df0).ready entity=mon.0 client_cookie=8f6f389ce021130e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7e7fc700 1 -- 172.21.15.26:0/291017417 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d6800d570 con 0x7f8d80147000 2022-07-20T02:34:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7e7fc700 1 -- 172.21.15.26:0/291017417 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68000e90 con 0x7f8d80147000 2022-07-20T02:34:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d7e7fc700 1 -- 172.21.15.26:0/291017417 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d6800d8c0 con 0x7f8d80147000 2022-07-20T02:34:20.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 -- 172.21.15.26:0/291017417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80147000 msgr2=0x7f8d800b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80147000 0x7f8d800b4730 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f8d68002f70 tx=0x7f8d68006df0).stop 2022-07-20T02:34:20.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 -- 172.21.15.26:0/291017417 shutdown_connections 2022-07-20T02:34:20.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80147000 0x7f8d800b4730 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/291017417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d8005cfd0 0x7f8d80143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 -- 172.21.15.26:0/291017417 >> 172.21.15.26:0/291017417 conn(0x7f8d800620c0 msgr2=0x7f8d8005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:20.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 -- 172.21.15.26:0/291017417 shutdown_connections 2022-07-20T02:34:20.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.763+0000 7f8d84fcc700 1 -- 172.21.15.26:0/291017417 wait complete. 2022-07-20T02:34:20.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 Processor -- start 2022-07-20T02:34:20.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 -- start start 2022-07-20T02:34:20.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:20.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d80147000 0x7f8d800b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:20.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 0x7f8d800b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:20.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801429a0 con 0x7f8d80060f00 2022-07-20T02:34:20.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80142ea0 con 0x7f8d80141340 2022-07-20T02:34:20.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d84fcc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80142c00 con 0x7f8d80147000 2022-07-20T02:34:20.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:20.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42510/0 (socket says 172.21.15.26:42510) 2022-07-20T02:34:20.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d7f7fe700 1 -- 172.21.15.26:0/2917624399 learned_addr learned my addr 172.21.15.26:0/2917624399 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:20.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d7ffff700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 0x7f8d800b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:20.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.767+0000 7f8d7effd700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d80147000 0x7f8d800b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:20.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 -- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d80147000 msgr2=0x7f8d800b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d80147000 0x7f8d800b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 -- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 msgr2=0x7f8d800b89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 0x7f8d800b89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 -- 172.21.15.26:0/2917624399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6800b180 con 0x7f8d80060f00 2022-07-20T02:34:20.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7ffff700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 0x7f8d800b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:20.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7f7fe700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f8d70006940 tx=0x7f8d70008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:20.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d70011780 con 0x7f8d80060f00 2022-07-20T02:34:20.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d700118e0 con 0x7f8d80060f00 2022-07-20T02:34:21.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d800b37f0 con 0x7f8d80060f00 2022-07-20T02:34:21.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d70011c40 con 0x7f8d80060f00 2022-07-20T02:34:21.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d80145b50 con 0x7f8d80060f00 2022-07-20T02:34:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80143720 con 0x7f8d80060f00 2022-07-20T02:34:21.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8d70012420 con 0x7f8d80060f00 2022-07-20T02:34:21.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 0x7f8d64055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f8d7006bec0 con 0x7f8d80060f00 2022-07-20T02:34:21.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.771+0000 7f8d7effd700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 0x7f8d64055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.775+0000 7f8d7effd700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 0x7f8d64055fc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f8d6801c000 tx=0x7f8d68015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:21.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.779+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8d70042bd0 con 0x7f8d80060f00 2022-07-20T02:34:21.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:20.907+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d800574e0 con 0x7f8d80060f00 2022-07-20T02:34:21.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.619+0000 7f8d7cff9700 1 -- 172.21.15.26:0/2917624399 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v129) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8d700461f0 con 0x7f8d80060f00 2022-07-20T02:34:21.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 msgr2=0x7f8d64055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 0x7f8d64055fc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f8d6801c000 tx=0x7f8d68015000).stop 2022-07-20T02:34:21.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 msgr2=0x7f8d801554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f8d70006940 tx=0x7f8d70008040).stop 2022-07-20T02:34:21.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 shutdown_connections 2022-07-20T02:34:21.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8d64053b00 0x7f8d64055fc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d80141340 0x7f8d800b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d80147000 0x7f8d800b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 --2- 172.21.15.26:0/2917624399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d80060f00 0x7f8d801554a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 >> 172.21.15.26:0/2917624399 conn(0x7f8d8005cfd0 msgr2=0x7f8d800b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:21.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 shutdown_connections 2022-07-20T02:34:21.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.627+0000 7f8d84fcc700 1 -- 172.21.15.26:0/2917624399 wait complete. 2022-07-20T02:34:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:34:21.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:34:21.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.803+0000 7f39eff4f700 1 Processor -- start 2022-07-20T02:34:21.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.803+0000 7f39eff4f700 1 -- start start 2022-07-20T02:34:21.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39eff4f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e805cfd0 0x7f39e81407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39eff4f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f39e8146e80 0x7f39e81472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39eff4f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e805d3b0 con 0x7f39e80b1160 2022-07-20T02:34:21.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39eff4f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e81435b0 con 0x7f39e8146e80 2022-07-20T02:34:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39eff4f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80b4bc0 con 0x7f39e805cfd0 2022-07-20T02:34:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ed4ea700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f39e80b1160 0x7f39e8140cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41798/0 (socket says 172.21.15.26:41798) 2022-07-20T02:34:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ed4ea700 1 -- 172.21.15.26:0/974372115 learned_addr learned my addr 172.21.15.26:0/974372115 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 --2- 172.21.15.26:0/974372115 >> v2:172.21.15.120:3300/0 conn(0x7f39e8146e80 0x7f39e81472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39edceb700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e805cfd0 0x7f39e81407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ecce9700 1 -- 172.21.15.26:0/974372115 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431275284 0 0) 0x7f39e805d3b0 con 0x7f39e80b1160 2022-07-20T02:34:21.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ecce9700 1 -- 172.21.15.26:0/974372115 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d0003260 con 0x7f39e80b1160 2022-07-20T02:34:21.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 -- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e805cfd0 msgr2=0x7f39e81407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:21.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e805cfd0 0x7f39e81407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 -- 172.21.15.26:0/974372115 >> v1:172.21.15.26:6789/0 conn(0x7f39e80b1160 legacy=0x7f39e8140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 -- 172.21.15.26:0/974372115 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80b4e60 con 0x7f39e8146e80 2022-07-20T02:34:21.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ee4ec700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e81472f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f39e00032f0 tx=0x7f39e00091d0).ready entity=mon.1 client_cookie=2b34e925109a0350 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:21.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ecce9700 1 -- 172.21.15.26:0/974372115 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39e000b600 con 0x7f39e8146e80 2022-07-20T02:34:21.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ecce9700 1 -- 172.21.15.26:0/974372115 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e000b760 con 0x7f39e8146e80 2022-07-20T02:34:21.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.807+0000 7f39ecce9700 1 -- 172.21.15.26:0/974372115 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39e000b8d0 con 0x7f39e8146e80 2022-07-20T02:34:21.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 msgr2=0x7f39e81472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e81472f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f39e00032f0 tx=0x7f39e00091d0).stop 2022-07-20T02:34:21.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- 172.21.15.26:0/974372115 shutdown_connections 2022-07-20T02:34:21.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e81472f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- 172.21.15.26:0/974372115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e805cfd0 0x7f39e81407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- 172.21.15.26:0/974372115 >> 172.21.15.26:0/974372115 conn(0x7f39e8066f70 msgr2=0x7f39e805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:21.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- 172.21.15.26:0/974372115 shutdown_connections 2022-07-20T02:34:21.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- 172.21.15.26:0/974372115 wait complete. 2022-07-20T02:34:21.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 Processor -- start 2022-07-20T02:34:21.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- start start 2022-07-20T02:34:21.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e80b1160 0x7f39e8152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e8153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8137060 con 0x7f39e805cfd0 2022-07-20T02:34:21.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8141730 con 0x7f39e8146e80 2022-07-20T02:34:21.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39eff4f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80b9230 con 0x7f39e80b1160 2022-07-20T02:34:21.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39edceb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39edceb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42524/0 (socket says 172.21.15.26:42524) 2022-07-20T02:34:21.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39edceb700 1 -- 172.21.15.26:0/2275393884 learned_addr learned my addr 172.21.15.26:0/2275393884 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:21.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39ee4ec700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e8153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.811+0000 7f39ed4ea700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e80b1160 0x7f39e8152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 -- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e80b1160 msgr2=0x7f39e8152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e80b1160 0x7f39e8152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 -- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 msgr2=0x7f39e8153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e8153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 -- 172.21.15.26:0/2275393884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009470 con 0x7f39e805cfd0 2022-07-20T02:34:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39edceb700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f39e400c000 tx=0x7f39e4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39e4010780 con 0x7f39e805cfd0 2022-07-20T02:34:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e40108e0 con 0x7f39e805cfd0 2022-07-20T02:34:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e80b94d0 con 0x7f39e805cfd0 2022-07-20T02:34:21.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39e4010c40 con 0x7f39e805cfd0 2022-07-20T02:34:21.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e81420c0 con 0x7f39e805cfd0 2022-07-20T02:34:21.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e81418f0 con 0x7f39e805cfd0 2022-07-20T02:34:21.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f39e401c460 con 0x7f39e805cfd0 2022-07-20T02:34:21.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 0x7f39c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:21.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f39e406c320 con 0x7f39e805cfd0 2022-07-20T02:34:21.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.815+0000 7f39ed4ea700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 0x7f39c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:21.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.819+0000 7f39ed4ea700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 0x7f39c8055f30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f39d800d000 tx=0x7f39d8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:21.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.827+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39e4043140 con 0x7f39e805cfd0 2022-07-20T02:34:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:21.951+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f39e80574e0 con 0x7f39e805cfd0 2022-07-20T02:34:22.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.631+0000 7f39d6ffd700 1 -- 172.21.15.26:0/2275393884 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v130) v1 ==== 133+0+0 (secure 0 0 0) 0x7f39e4046760 con 0x7f39e805cfd0 2022-07-20T02:34:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 msgr2=0x7f39c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 0x7f39c8055f30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f39d800d000 tx=0x7f39d8009040).stop 2022-07-20T02:34:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 msgr2=0x7f39e80b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f39e400c000 tx=0x7f39e4006df0).stop 2022-07-20T02:34:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 shutdown_connections 2022-07-20T02:34:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f39c8053a70 0x7f39c8055f30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39e8146e80 0x7f39e8153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39e80b1160 0x7f39e8152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 --2- 172.21.15.26:0/2275393884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39e805cfd0 0x7f39e80b2e50 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 >> 172.21.15.26:0/2275393884 conn(0x7f39e8066f70 msgr2=0x7f39e813c390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 shutdown_connections 2022-07-20T02:34:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.635+0000 7f39eff4f700 1 -- 172.21.15.26:0/2275393884 wait complete. 2022-07-20T02:34:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:34:22.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-20T02:34:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 Processor -- start 2022-07-20T02:34:22.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 -- start start 2022-07-20T02:34:22.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f37c4061780 0x7f37c4061b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4140360 0x7f37c41407a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c4062680 con 0x7f37c4146930 2022-07-20T02:34:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c40628b0 con 0x7f37c4061780 2022-07-20T02:34:22.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37cb0bf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40b35a0 con 0x7f37c4140360 2022-07-20T02:34:22.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f37c4061780 0x7f37c4061b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f37c4061780 0x7f37c4061b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42532/0 (socket says 172.21.15.26:42532) 2022-07-20T02:34:22.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 -- 172.21.15.26:0/60234269 learned_addr learned my addr 172.21.15.26:0/60234269 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:22.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37bbfff700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4140360 0x7f37c41407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 -- 172.21.15.26:0/60234269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4140360 msgr2=0x7f37c41407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:22.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4140360 0x7f37c41407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 -- 172.21.15.26:0/60234269 >> v1:172.21.15.120:6789/0 conn(0x7f37c4146930 legacy=0x7f37c4146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 -- 172.21.15.26:0/60234269 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40b3840 con 0x7f37c4061780 2022-07-20T02:34:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37c8e5b700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4061780 0x7f37c4061b60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f37c0002f70 tx=0x7f37c0006eb0).ready entity=mon.0 client_cookie=7c71730506a89656 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.779+0000 7f37bb7fe700 1 -- 172.21.15.26:0/60234269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37c000d570 con 0x7f37c4061780 2022-07-20T02:34:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37bb7fe700 1 -- 172.21.15.26:0/60234269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c0000e90 con 0x7f37c4061780 2022-07-20T02:34:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37bb7fe700 1 -- 172.21.15.26:0/60234269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37c000d900 con 0x7f37c4061780 2022-07-20T02:34:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 -- 172.21.15.26:0/60234269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4061780 msgr2=0x7f37c4061b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4061780 0x7f37c4061b60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f37c0002f70 tx=0x7f37c0006eb0).stop 2022-07-20T02:34:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 -- 172.21.15.26:0/60234269 shutdown_connections 2022-07-20T02:34:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4140360 0x7f37c41407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/60234269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4061780 0x7f37c4061b60 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 -- 172.21.15.26:0/60234269 >> 172.21.15.26:0/60234269 conn(0x7f37c405c580 msgr2=0x7f37c405c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 -- 172.21.15.26:0/60234269 shutdown_connections 2022-07-20T02:34:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 -- 172.21.15.26:0/60234269 wait complete. 2022-07-20T02:34:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.783+0000 7f37cb0bf700 1 Processor -- start 2022-07-20T02:34:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- start start 2022-07-20T02:34:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4061780 0x7f37c4155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 0x7f37c40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40aedc0 con 0x7f37c4140360 2022-07-20T02:34:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4066590 con 0x7f37c4146930 2022-07-20T02:34:22.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40aefb0 con 0x7f37c4061780 2022-07-20T02:34:22.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42540/0 (socket says 172.21.15.26:42540) 2022-07-20T02:34:22.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 -- 172.21.15.26:0/4082038866 learned_addr learned my addr 172.21.15.26:0/4082038866 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:22.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37c8e5b700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4061780 0x7f37c4155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37c965c700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 0x7f37c40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 -- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4061780 msgr2=0x7f37c4155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4061780 0x7f37c4155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 -- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 msgr2=0x7f37c40b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 0x7f37c40b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 -- 172.21.15.26:0/4082038866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c000b1e0 con 0x7f37c4140360 2022-07-20T02:34:22.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37c965c700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 0x7f37c40b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:22.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37bbfff700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f37b400c000 tx=0x7f37b4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:22.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37b4010780 con 0x7f37c4140360 2022-07-20T02:34:22.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b40108e0 con 0x7f37c4140360 2022-07-20T02:34:22.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c40af250 con 0x7f37c4140360 2022-07-20T02:34:22.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37b4010c40 con 0x7f37c4140360 2022-07-20T02:34:22.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.787+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c40b19f0 con 0x7f37c4140360 2022-07-20T02:34:22.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c40574e0 con 0x7f37c4140360 2022-07-20T02:34:22.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f37b401c460 con 0x7f37c4140360 2022-07-20T02:34:22.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37b9ffb700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 0x7f37a4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:22.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f37b406c540 con 0x7f37c4140360 2022-07-20T02:34:22.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37c8e5b700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 0x7f37a4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:22.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.791+0000 7f37c8e5b700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 0x7f37a4055f30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f37c001c000 tx=0x7f37c0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:22.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.803+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37b4043200 con 0x7f37c4140360 2022-07-20T02:34:22.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.959+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f37c405db80 con 0x7f37c4140360 2022-07-20T02:34:22.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:34:22.959+0000 7f558fb37700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:34:22.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.967+0000 7f37b9ffb700 1 -- 172.21.15.26:0/4082038866 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 22 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7f37b4046820 con 0x7f37c4140360 2022-07-20T02:34:22.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 msgr2=0x7f37a4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 0x7f37a4055f30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f37c001c000 tx=0x7f37c0015000).stop 2022-07-20T02:34:22.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 msgr2=0x7f37c40b6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f37b400c000 tx=0x7f37b4006df0).stop 2022-07-20T02:34:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 shutdown_connections 2022-07-20T02:34:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f37c4140360 0x7f37c40b6940 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f37a4053a70 0x7f37a4055f30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c4146930 0x7f37c40b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 --2- 172.21.15.26:0/4082038866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c4061780 0x7f37c4155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 >> 172.21.15.26:0/4082038866 conn(0x7f37c405c580 msgr2=0x7f37c4144650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:22.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 shutdown_connections 2022-07-20T02:34:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:22.971+0000 7f37cb0bf700 1 -- 172.21.15.26:0/4082038866 wait complete. 2022-07-20T02:34:22.999 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 24 and data pool 22 2022-07-20T02:34:23.003 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:34:23.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:23.039 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:23.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:23.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:23.182 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:34:23.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:23.186 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:23.196 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:23.222 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:34:23.223 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:23.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:23.240 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:23.261 INFO:teuthology.orchestra.run.smithi159.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:34:23.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:23.285 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:23.286 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:23.354 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:34:23.355 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:23.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:23.372 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:23.406 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:34:23.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:34:23.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 Processor -- start 2022-07-20T02:34:23.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- start start 2022-07-20T02:34:23.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f74a0061050 0x7f74a0148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a00b1bc0 con 0x7f74a0066a80 2022-07-20T02:34:23.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a014b040 con 0x7f74a0061050 2022-07-20T02:34:23.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00a8860 con 0x7f74a00b2920 2022-07-20T02:34:23.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f74a0061050 0x7f74a0148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f74a0061050 0x7f74a0148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42546/0 (socket says 172.21.15.26:42546) 2022-07-20T02:34:23.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 -- 172.21.15.26:0/1604718987 learned_addr learned my addr 172.21.15.26:0/1604718987 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749ffff700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 -- 172.21.15.26:0/1604718987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 msgr2=0x7f74a0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 -- 172.21.15.26:0/1604718987 >> v1:172.21.15.120:6789/0 conn(0x7f74a0066a80 legacy=0x7f74a0066ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 -- 172.21.15.26:0/1604718987 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0147700 con 0x7f74a0061050 2022-07-20T02:34:23.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749f7fe700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0061050 0x7f74a0148280 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f74900032f0 tx=0x7f7490009290).ready entity=mon.0 client_cookie=333525ea7fc4ad1c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:23.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749effd700 1 -- 172.21.15.26:0/1604718987 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f749000b8a0 con 0x7f74a0061050 2022-07-20T02:34:23.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749effd700 1 -- 172.21.15.26:0/1604718987 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749000ba00 con 0x7f74a0061050 2022-07-20T02:34:23.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f749effd700 1 -- 172.21.15.26:0/1604718987 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f749000bb70 con 0x7f74a0061050 2022-07-20T02:34:23.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- 172.21.15.26:0/1604718987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0061050 msgr2=0x7f74a0148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:23.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0061050 0x7f74a0148280 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f74900032f0 tx=0x7f7490009290).stop 2022-07-20T02:34:23.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- 172.21.15.26:0/1604718987 shutdown_connections 2022-07-20T02:34:23.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0061050 0x7f74a0148280 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:23.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 --2- 172.21.15.26:0/1604718987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:23.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.527+0000 7f74a62cd700 1 -- 172.21.15.26:0/1604718987 >> 172.21.15.26:0/1604718987 conn(0x7f74a00b60a0 msgr2=0x7f74a00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:23.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- 172.21.15.26:0/1604718987 shutdown_connections 2022-07-20T02:34:23.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- 172.21.15.26:0/1604718987 wait complete. 2022-07-20T02:34:23.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 Processor -- start 2022-07-20T02:34:23.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- start start 2022-07-20T02:34:23.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74a0061050 0x7f74a0149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00ace70 con 0x7f74a0066a80 2022-07-20T02:34:23.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00ada30 con 0x7f74a0061050 2022-07-20T02:34:23.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f74a62cd700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00ab540 con 0x7f74a00b2920 2022-07-20T02:34:23.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f749f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f749f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42554/0 (socket says 172.21.15.26:42554) 2022-07-20T02:34:23.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f749ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74a0061050 0x7f74a0149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.531+0000 7f749f7fe700 1 -- 172.21.15.26:0/2856671386 learned_addr learned my addr 172.21.15.26:0/2856671386 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f74a486a700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 -- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 msgr2=0x7f74a0156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 -- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74a0061050 msgr2=0x7f74a0149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74a0061050 0x7f74a0149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 -- 172.21.15.26:0/2856671386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490009530 con 0x7f74a0066a80 2022-07-20T02:34:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f74a486a700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749f7fe700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f74900032f0 tx=0x7f7490016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f749000b8a0 con 0x7f74a0066a80 2022-07-20T02:34:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490021440 con 0x7f74a0066a80 2022-07-20T02:34:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a00ad0a0 con 0x7f74a0066a80 2022-07-20T02:34:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74900217a0 con 0x7f74a0066a80 2022-07-20T02:34:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a0156f50 con 0x7f74a0066a80 2022-07-20T02:34:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a014ada0 con 0x7f74a0066a80 2022-07-20T02:34:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.535+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7490027020 con 0x7f74a0066a80 2022-07-20T02:34:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.539+0000 7f749d7fa700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 0x7f7484055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.539+0000 7f749ffff700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 0x7f7484055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.539+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f749006d430 con 0x7f74a0066a80 2022-07-20T02:34:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.539+0000 7f749ffff700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 0x7f7484055ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f748800b000 tx=0x7f7488006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.547+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74900441a0 con 0x7f74a0066a80 2022-07-20T02:34:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.675+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f74a00574e0 con 0x7f74a0066a80 2022-07-20T02:34:23.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.675+0000 7f749d7fa700 1 -- 172.21.15.26:0/2856671386 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v131) v1 ==== 81+0+34264 (secure 0 0 0) 0x7f7490015020 con 0x7f74a0066a80 2022-07-20T02:34:23.687 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:23.688 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:23.689 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 131, 2022-07-20T02:34:23.690 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:34:23.704 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:34:23.704 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:34:22.962490+0000", 2022-07-20T02:34:23.705 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:23.705 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:23.705 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:34:23.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:34:23.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:34:23.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:34:23.707 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:34:23.707 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:34:23.708 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:34:23.708 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:23.709 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:34:23.709 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:34:23.709 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:34:23.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:34:23.710 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:34:23.711 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 24, 2022-07-20T02:34:23.711 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:34:23.711 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:34:23.712 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:34:23.712 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:34:23.713 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:34:23.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:23.713 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:34:23.714 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:34:23.732 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:34:23.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:23.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:23.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:23.743 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:23.744 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:23.750 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:23.750 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:23.751 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:23.751 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:34:23.752 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:34:23.752 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:34:23.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:34:23.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:34:23.765 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:23.765 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:23.766 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:23.766 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:23.767 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:23.767 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:23.768 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:23.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:23.784 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:34:23.785 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:23.785 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:23.786 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:23.787 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:23.788 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:23.788 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:23.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:23.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:23.856 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:23.856 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:23.856 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:23.857 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:23.857 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:23.891 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:23.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:23.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:23.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:23.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:23.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:23.900 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:23.911 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:23.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:23.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:23.920 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:23.921 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:23.922 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:23.923 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:23.946 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:23.947 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:34:23.948 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:34:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:23.949 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:34:23.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:23.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:34:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:34:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:34:23.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:23.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:23.955 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:23.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:23.982 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:23.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:23.984 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:24.021 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:24.021 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:24.022 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:24.022 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:24.023 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:24.023 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:24.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:24.024 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:24.025 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:24.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.025 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:34:24.036 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:24.037 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:24.037 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:24.052 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:24.052 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:24.053 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:24.053 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:24.054 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:24.054 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:24.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:24.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:24.055 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:24.056 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:24.056 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:24.057 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:24.057 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:24.098 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:24.098 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:24.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:24.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:24.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:24.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:24.104 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:24.105 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:24.105 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:24.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:24.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.105 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:24.106 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:24.106 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:24.106 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:24.106 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:24.107 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:24.107 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:24.107 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:24.107 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:24.107 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:24.108 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:24.108 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:34:24.108 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:24.108 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:34:24.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 22, 2022-07-20T02:34:24.153 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:34:24.154 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:33:51.226718+0000", 2022-07-20T02:34:24.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:24.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:24.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:24.155 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:24.155 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:24.174 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:24.174 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:24.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:24.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:24.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:24.176 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:24.176 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:24.176 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:24.200 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:24.203 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:24.204 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:24.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:24.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:24.204 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:24.205 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:24.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "131", 2022-07-20T02:34:24.205 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:24.206 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:24.215 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:24.215 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:24.215 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:24.216 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:24.216 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:24.216 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:24.216 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:24.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:24.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:24.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:24.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:24.217 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:24.238 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:24.239 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:24.239 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:24.239 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:24.239 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:24.240 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:24.240 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:24.240 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:24.240 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:24.241 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:24.241 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:24.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:24.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.246 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:24.246 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:24.247 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:24.247 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:24.247 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:24.247 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:24.248 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:24.248 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:24.248 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:24.248 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:24.249 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:24.249 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:34:24.249 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:24.249 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:34:24.249 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:34:24.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.255 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 24, 2022-07-20T02:34:24.256 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:34:24.261 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:34:22.008641+0000", 2022-07-20T02:34:24.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 32769, 2022-07-20T02:34:24.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool,creating", 2022-07-20T02:34:24.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:24.268 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:24.268 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:24.268 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:24.268 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:24.269 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:24.269 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:24.269 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:24.269 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:24.269 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:24.270 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:24.270 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:24.270 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:24.270 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:24.271 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:24.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:24.279 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:24.279 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:24.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.280 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "131", 2022-07-20T02:34:24.281 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:24.281 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:24.281 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:24.290 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:24.291 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:24.291 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:24.291 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:24.292 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:24.292 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:24.293 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:24.293 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:24.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:24.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:24.295 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:24.295 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:24.295 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:24.296 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:24.296 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:24.297 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:24.297 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:24.298 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:24.298 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:24.298 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:24.299 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:24.299 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:24.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:24.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.309 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:24.310 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:24.310 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:24.311 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:24.312 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:24.321 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:24.322 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:34:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:34:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:34:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:34:24.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:24.325 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:34:24.340 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:34:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.342 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:24.343 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:34:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.344 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:34:24.345 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.346 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.347 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:24.393 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:24.394 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:34:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:34:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:24.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:34:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:34:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.424 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:24.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:34:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.434 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:34:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:34:24.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:34:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:24.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:34:24.457 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:34:24.464 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:34:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:34:24.465 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:24.467 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.469 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:24.470 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:34:24.471 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.472 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.473 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.474 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.475 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:24.476 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:24.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:34:24.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:34:24.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:24.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:34:24.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:34:24.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.510 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:34:24.512 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:34:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.516 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:34:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:34:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:24.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:34:24.526 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:34:24.526 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:34:24.527 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:34:24.527 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:24.527 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:34:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 125, 2022-07-20T02:34:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:34:24.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:34:24.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.574 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:24.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:34:24.577 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:34:24.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.581 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:24.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:34:24.583 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:34:24.585 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:34:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:34:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:34:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:34:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:34:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:34:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:34:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:34:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:34:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.721 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:24.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:34:24.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:34:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.764 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:24.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:34:24.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:34:24.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.775 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:24.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:34:24.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.780 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:34:24.780 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:24.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.794 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:34:24.794 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:34:24.794 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:34:24.795 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:34:24.795 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:24.795 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:24.796 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:24.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.797 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:24.800 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:34:24.800 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.801 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.801 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.801 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.801 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.802 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.803 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.803 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 125, 2022-07-20T02:34:24.803 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:24.804 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:24.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:24.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:34:24.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:34:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:34:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:34:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:34:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:34:24.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.862 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:24.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:24.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:34:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:34:24.911 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:24.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:24.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.971 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:34:24.971 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:34:24.974 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:34:24.975 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:34:24.975 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:24.976 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:24.987 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:24.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:24.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:24.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:24.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:24.990 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:34:24.990 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:24.994 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:24.994 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:24.995 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:24.995 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:24.995 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:24.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:24.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:24.996 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:25.000 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:25.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:25.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:34:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:34:25.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.041 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:25.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:34:25.047 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:34:25.050 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.051 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:25.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.053 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:34:25.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:34:25.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.060 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:25.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:34:25.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:34:25.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:25.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:34:25.063 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:34:25.063 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:34:25.063 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:34:25.064 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:25.064 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:25.065 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:25.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.066 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:25.066 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:34:25.066 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:25.078 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:25.119 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:25.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:25.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:25.126 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:25.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:25.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:34:25.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:34:25.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.135 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:25.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:34:25.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:34:25.141 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.143 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:25.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:34:25.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:34:25.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.152 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:25.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:34:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:34:25.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:25.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:34:25.220 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:34:25.220 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:34:25.221 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:34:25.221 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:25.222 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:25.222 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.224 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:25.224 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:34:25.225 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:25.225 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:25.255 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:25.255 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:25.257 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:25.258 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:25.259 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:25.269 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:25.269 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:25.270 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:25.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:25.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:34:25.288 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:34:25.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:34:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:34:25.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:34:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:34:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:25.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:25.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:34:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:34:25.384 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:34:25.386 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:34:25.387 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:34:25.393 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:34:25.425 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:25.426 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:25.426 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:25.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:25.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.428 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:25.428 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:34:25.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.429 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:25.430 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.431 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.431 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.432 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.432 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.440 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:34:25.441 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.441 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:25.442 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.448 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.448 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.448 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.449 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.449 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:34:25.449 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.451 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:25.451 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.451 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.451 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.452 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.452 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.452 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:34:25.452 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.453 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:25.453 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.454 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.454 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.454 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.454 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.455 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:34:25.455 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.455 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:25.456 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.456 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.456 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.457 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.457 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.457 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:34:25.457 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.473 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:25.473 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.474 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.474 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.474 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.475 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:34:25.475 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.476 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:25.477 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.477 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.477 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.478 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.478 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.478 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:34:25.478 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:25.479 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:25.479 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:25.479 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:25.480 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:25.480 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:25.480 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:25.480 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:34:25.481 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:25.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.481 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:25.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:34:25.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:34:25.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:34:25.482 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:34:25.482 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:34:25.482 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/126332545": "2022-07-21T02:34:08.449272+0000", 2022-07-20T02:34:25.483 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/126332545": "2022-07-21T02:34:08.449272+0000" 2022-07-20T02:34:25.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.483 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:34:25.484 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:34:25.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:34:25.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:25.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:34:25.485 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:34:25.485 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:25.485 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:34:25.486 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:25.486 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:25.487 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:34:25.487 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:34:25.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.487 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:34:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:34:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:34:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:34:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:34:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:34:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:34:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:34:25.491 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:34:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 msgr2=0x7f7484055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 0x7f7484055ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f748800b000 tx=0x7f7488006df0).stop 2022-07-20T02:34:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 msgr2=0x7f74a00ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f74900032f0 tx=0x7f7490016000).stop 2022-07-20T02:34:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 shutdown_connections 2022-07-20T02:34:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74a0066a80 0x7f74a00ad540 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f74840539e0 0x7f7484055ea0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74a00b2920 0x7f74a0156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 --2- 172.21.15.26:0/2856671386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74a0061050 0x7f74a0149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 >> 172.21.15.26:0/2856671386 conn(0x7f74a00b60a0 msgr2=0x7f74a00b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 shutdown_connections 2022-07-20T02:34:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:23.679+0000 7f74a62cd700 1 -- 172.21.15.26:0/2856671386 wait complete. 2022-07-20T02:34:25.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:25.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.699+0000 7f3d3ffff700 1 Processor -- start 2022-07-20T02:34:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.699+0000 7f3d3ffff700 1 -- start start 2022-07-20T02:34:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 0x7f3d400aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3d400a7fc0 0x7f3d400af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d400b62d0 con 0x7f3d400b5ef0 2022-07-20T02:34:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4014aac0 con 0x7f3d400a7fc0 2022-07-20T02:34:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4014ad60 con 0x7f3d40147be0 2022-07-20T02:34:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3d400a7fc0 0x7f3d400af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3d400a7fc0 0x7f3d400af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42564/0 (socket says 172.21.15.26:42564) 2022-07-20T02:34:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3e7fc700 1 --2- 172.21.15.26:0/4215200146 >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 0x7f3d400aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/4215200146 learned_addr learned my addr 172.21.15.26:0/4215200146 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/4215200146 >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 msgr2=0x7f3d400aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 --2- 172.21.15.26:0/4215200146 >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 0x7f3d400aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/4215200146 >> v1:172.21.15.120:6789/0 conn(0x7f3d400b5ef0 legacy=0x7f3d401474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/4215200146 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4014b000 con 0x7f3d400a7fc0 2022-07-20T02:34:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3e7fc700 1 --2- 172.21.15.26:0/4215200146 >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 0x7f3d400aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3f7fe700 1 --2- 172.21.15.26:0/4215200146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d400a7fc0 0x7f3d400af330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f3d380032f0 tx=0x7f3d38009290).ready entity=mon.0 client_cookie=2235f5a7804e5f4f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3dffb700 1 -- 172.21.15.26:0/4215200146 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3800b870 con 0x7f3d400a7fc0 2022-07-20T02:34:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3dffb700 1 -- 172.21.15.26:0/4215200146 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3800b9d0 con 0x7f3d400a7fc0 2022-07-20T02:34:25.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3dffb700 1 -- 172.21.15.26:0/4215200146 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3800bb40 con 0x7f3d400a7fc0 2022-07-20T02:34:25.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- 172.21.15.26:0/4215200146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d400a7fc0 msgr2=0x7f3d400af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/4215200146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d400a7fc0 0x7f3d400af330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f3d380032f0 tx=0x7f3d38009290).stop 2022-07-20T02:34:25.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- 172.21.15.26:0/4215200146 shutdown_connections 2022-07-20T02:34:25.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/4215200146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d400a7fc0 0x7f3d400af330 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/4215200146 >> v2:172.21.15.120:3300/0 conn(0x7f3d40147be0 0x7f3d400aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- 172.21.15.26:0/4215200146 >> 172.21.15.26:0/4215200146 conn(0x7f3d40060310 msgr2=0x7f3d400b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:25.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- 172.21.15.26:0/4215200146 shutdown_connections 2022-07-20T02:34:25.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.703+0000 7f3d3ffff700 1 -- 172.21.15.26:0/4215200146 wait complete. 2022-07-20T02:34:25.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 Processor -- start 2022-07-20T02:34:25.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- start start 2022-07-20T02:34:25.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d400a7fc0 0x7f3d400b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d400b5ef0 0x7f3d40159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d400a8570 con 0x7f3d40147be0 2022-07-20T02:34:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d400aab20 con 0x7f3d400a7fc0 2022-07-20T02:34:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d400abb20 con 0x7f3d400b5ef0 2022-07-20T02:34:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42570/0 (socket says 172.21.15.26:42570) 2022-07-20T02:34:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3effd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d400a7fc0 0x7f3d400b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/741302656 learned_addr learned my addr 172.21.15.26:0/741302656 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3e7fc700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d400b5ef0 0x7f3d40159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d400b5ef0 msgr2=0x7f3d40159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d400b5ef0 0x7f3d40159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d400a7fc0 msgr2=0x7f3d400b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d400a7fc0 0x7f3d400b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 -- 172.21.15.26:0/741302656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3000b180 con 0x7f3d40147be0 2022-07-20T02:34:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3f7fe700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f3d380032f0 tx=0x7f3d38016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3800b870 con 0x7f3d40147be0 2022-07-20T02:34:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d38021670 con 0x7f3d40147be0 2022-07-20T02:34:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38009530 con 0x7f3d40147be0 2022-07-20T02:34:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d380219d0 con 0x7f3d40147be0 2022-07-20T02:34:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.707+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d400aa680 con 0x7f3d40147be0 2022-07-20T02:34:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d400abd20 con 0x7f3d40147be0 2022-07-20T02:34:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3d38028070 con 0x7f3d40147be0 2022-07-20T02:34:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d2ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 0x7f3d1c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f3d3806cef0 con 0x7f3d40147be0 2022-07-20T02:34:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d3effd700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 0x7f3d1c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.711+0000 7f3d3effd700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 0x7f3d1c055ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f3d3400d000 tx=0x7f3d34000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.719+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d38043c60 con 0x7f3d40147be0 2022-07-20T02:34:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.883+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d400aad20 con 0x7f3d40147be0 2022-07-20T02:34:25.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.883+0000 7f3d2ffff700 1 -- 172.21.15.26:0/741302656 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f3d38015070 con 0x7f3d40147be0 2022-07-20T02:34:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:26.001 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":66,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:22.962666+0000","modified":"2022-07-20T02:34:22.962734+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-20T02:34:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 msgr2=0x7f3d1c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 0x7f3d1c055ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f3d3400d000 tx=0x7f3d34000bc0).stop 2022-07-20T02:34:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 msgr2=0x7f3d4015a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f3d380032f0 tx=0x7f3d38016000).stop 2022-07-20T02:34:26.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 shutdown_connections 2022-07-20T02:34:26.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3d40147be0 0x7f3d4015a4c0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3d1c053b30 0x7f3d1c055ff0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d400b5ef0 0x7f3d40159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 --2- 172.21.15.26:0/741302656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d400a7fc0 0x7f3d400b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 >> 172.21.15.26:0/741302656 conn(0x7f3d40060310 msgr2=0x7f3d400af150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:26.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 shutdown_connections 2022-07-20T02:34:26.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:25.887+0000 7f3d3ffff700 1 -- 172.21.15.26:0/741302656 wait complete. 2022-07-20T02:34:26.071 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 66 2022-07-20T02:34:26.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:34:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.203+0000 7fb6b1781700 1 Processor -- start 2022-07-20T02:34:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.203+0000 7fb6b1781700 1 -- start start 2022-07-20T02:34:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac0a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac0b2f10 con 0x7fb6ac147d70 2022-07-20T02:34:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac0a7500 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac0a8e30 con 0x7fb6ac060f10 2022-07-20T02:34:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6aa7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb6ac147d70 0x7fb6ac066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41856/0 (socket says 172.21.15.26:41856) 2022-07-20T02:34:26.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6aa7fc700 1 -- 172.21.15.26:0/2129385679 learned_addr learned my addr 172.21.15.26:0/2129385679 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6ab7fe700 1 --2- 172.21.15.26:0/2129385679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac0a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288060475 0 0) 0x7fb6ac0b2f10 con 0x7fb6ac147d70 2022-07-20T02:34:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb690003260 con 0x7fb6ac147d70 2022-07-20T02:34:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083089835 0 0) 0x7fb6ac0a7500 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ac0b2f10 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6ab7fe700 1 -- 172.21.15.26:0/2129385679 >> v1:172.21.15.120:6789/0 conn(0x7fb6ac0b2b30 legacy=0x7fb6ac147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6ab7fe700 1 -- 172.21.15.26:0/2129385679 >> v1:172.21.15.26:6789/0 conn(0x7fb6ac147d70 legacy=0x7fb6ac066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1811057609 0 0) 0x7fb690003260 con 0x7fb6ac147d70 2022-07-20T02:34:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6ab7fe700 1 -- 172.21.15.26:0/2129385679 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac0a77a0 con 0x7fb6ac060f10 2022-07-20T02:34:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6ab7fe700 1 --2- 172.21.15.26:0/2129385679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac0a7ee0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb69c001490 tx=0x7fb6a4006df0).ready entity=mon.2 client_cookie=864fade99909174f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb69c0019e0 con 0x7fb6ac060f10 2022-07-20T02:34:26.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a4000e90 con 0x7fb6ac060f10 2022-07-20T02:34:26.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6a9ffb700 1 -- 172.21.15.26:0/2129385679 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6a4012460 con 0x7fb6ac060f10 2022-07-20T02:34:26.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- 172.21.15.26:0/2129385679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 msgr2=0x7fb6ac0a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 --2- 172.21.15.26:0/2129385679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac0a7ee0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb69c001490 tx=0x7fb6a4006df0).stop 2022-07-20T02:34:26.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- 172.21.15.26:0/2129385679 shutdown_connections 2022-07-20T02:34:26.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 --2- 172.21.15.26:0/2129385679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac0a7ee0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- 172.21.15.26:0/2129385679 >> 172.21.15.26:0/2129385679 conn(0x7fb6ac0b6110 msgr2=0x7fb6ac0b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- 172.21.15.26:0/2129385679 shutdown_connections 2022-07-20T02:34:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.207+0000 7fb6b1781700 1 -- 172.21.15.26:0/2129385679 wait complete. 2022-07-20T02:34:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 Processor -- start 2022-07-20T02:34:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 -- start start 2022-07-20T02:34:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac15a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ac147d70 0x7fb6ac0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac15acf0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac0b0e50 con 0x7fb6ac147d70 2022-07-20T02:34:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6b1781700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac0b10f0 con 0x7fb6ac060f10 2022-07-20T02:34:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6aa7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6aa7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42582/0 (socket says 172.21.15.26:42582) 2022-07-20T02:34:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6aa7fc700 1 -- 172.21.15.26:0/3799000257 learned_addr learned my addr 172.21.15.26:0/3799000257 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6ab7fe700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ac147d70 0x7fb6ac0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.211+0000 7fb6aaffd700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac15a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 -- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 msgr2=0x7fb6ac15a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac15a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 -- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ac147d70 msgr2=0x7fb6ac0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ac147d70 0x7fb6ac0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 -- 172.21.15.26:0/3799000257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a400b180 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aa7fc700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb69c001bc0 tx=0x7fb69c001aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb69c004ab0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c004c10 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb69c007170 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac0ad240 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac0ad6e0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac11d5d0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb69c01b070 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 0x7fb688055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb69c06ba80 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.215+0000 7fb6aaffd700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 0x7fb688055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.219+0000 7fb6aaffd700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 0x7fb688055e40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fb6a000d000 tx=0x7fb6a0007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.223+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6ac11d5d0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.387+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb6ac11d5d0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.643+0000 7fb69bfff700 1 -- 172.21.15.26:0/3799000257 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb6ac11d5d0 con 0x7fb6ac0b2b30 2022-07-20T02:34:26.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 msgr2=0x7fb688055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 0x7fb688055e40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fb6a000d000 tx=0x7fb6a0007ad0).stop 2022-07-20T02:34:26.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 msgr2=0x7fb6ac14a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb69c001bc0 tx=0x7fb69c001aa0).stop 2022-07-20T02:34:26.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 shutdown_connections 2022-07-20T02:34:26.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb6ac0b2b30 0x7fb6ac14a400 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb688053980 0x7fb688055e40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6ac147d70 0x7fb6ac0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 --2- 172.21.15.26:0/3799000257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6ac060f10 0x7fb6ac15a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 >> 172.21.15.26:0/3799000257 conn(0x7fb6ac0b6110 msgr2=0x7fb6ac0aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:26.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 shutdown_connections 2022-07-20T02:34:26.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.647+0000 7fb6b1781700 1 -- 172.21.15.26:0/3799000257 wait complete. 2022-07-20T02:34:26.657 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:34:26.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.811+0000 7fb504310700 1 Processor -- start 2022-07-20T02:34:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.811+0000 7fb504310700 1 -- start start 2022-07-20T02:34:26.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc1468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc061e00 con 0x7fb4fc1432e0 2022-07-20T02:34:26.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc146db0 con 0x7fb4fc061a20 2022-07-20T02:34:26.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc147040 con 0x7fb4fc12c750 2022-07-20T02:34:26.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5018ab700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb4fc1432e0 0x7fb4fc0b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41870/0 (socket says 172.21.15.26:41870) 2022-07-20T02:34:26.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5018ab700 1 -- 172.21.15.26:0/562293802 learned_addr learned my addr 172.21.15.26:0/562293802 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:26.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5028ad700 1 --2- 172.21.15.26:0/562293802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc1468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675275053 0 0) 0x7fb4fc061e00 con 0x7fb4fc1432e0 2022-07-20T02:34:26.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e8003260 con 0x7fb4fc1432e0 2022-07-20T02:34:26.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141141478 0 0) 0x7fb4fc146db0 con 0x7fb4fc061a20 2022-07-20T02:34:26.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc061e00 con 0x7fb4fc061a20 2022-07-20T02:34:26.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5028ad700 1 -- 172.21.15.26:0/562293802 >> v1:172.21.15.120:6789/0 conn(0x7fb4fc061a20 legacy=0x7fb4fc12c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5028ad700 1 -- 172.21.15.26:0/562293802 >> v1:172.21.15.26:6789/0 conn(0x7fb4fc1432e0 legacy=0x7fb4fc0b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5028ad700 1 -- 172.21.15.26:0/562293802 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc1472e0 con 0x7fb4fc12c750 2022-07-20T02:34:26.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5028ad700 1 --2- 172.21.15.26:0/562293802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc1468c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4006df0).ready entity=mon.2 client_cookie=3d12ecf0d6b8929c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f400d5c0 con 0x7fb4fc12c750 2022-07-20T02:34:26.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f4000e90 con 0x7fb4fc12c750 2022-07-20T02:34:26.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb5010aa700 1 -- 172.21.15.26:0/562293802 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f400d910 con 0x7fb4fc12c750 2022-07-20T02:34:26.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- 172.21.15.26:0/562293802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 msgr2=0x7fb4fc1468c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 --2- 172.21.15.26:0/562293802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc1468c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4006df0).stop 2022-07-20T02:34:26.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- 172.21.15.26:0/562293802 shutdown_connections 2022-07-20T02:34:26.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 --2- 172.21.15.26:0/562293802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc1468c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- 172.21.15.26:0/562293802 >> 172.21.15.26:0/562293802 conn(0x7fb4fc05faf0 msgr2=0x7fb4fc066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:26.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- 172.21.15.26:0/562293802 shutdown_connections 2022-07-20T02:34:26.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.815+0000 7fb504310700 1 -- 172.21.15.26:0/562293802 wait complete. 2022-07-20T02:34:26.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 Processor -- start 2022-07-20T02:34:26.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- start start 2022-07-20T02:34:26.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc061a20 0x7fb4fc142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0b0910 con 0x7fb4fc1432e0 2022-07-20T02:34:26.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc126420 con 0x7fb4fc061a20 2022-07-20T02:34:26.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc126fc0 con 0x7fb4fc12c750 2022-07-20T02:34:26.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42594/0 (socket says 172.21.15.26:42594) 2022-07-20T02:34:26.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 -- 172.21.15.26:0/2494654991 learned_addr learned my addr 172.21.15.26:0/2494654991 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:26.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5018ab700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5020ac700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc061a20 0x7fb4fc142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 -- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 msgr2=0x7fb4fc144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 -- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc061a20 msgr2=0x7fb4fc142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:26.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc061a20 0x7fb4fc142c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:26.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 -- 172.21.15.26:0/2494654991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f400b180 con 0x7fb4fc1432e0 2022-07-20T02:34:26.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb5028ad700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f4013de0 con 0x7fb4fc1432e0 2022-07-20T02:34:26.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f400de70 con 0x7fb4fc1432e0 2022-07-20T02:34:26.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc129740 con 0x7fb4fc1432e0 2022-07-20T02:34:26.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f401e5f0 con 0x7fb4fc1432e0 2022-07-20T02:34:26.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.819+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc1268b0 con 0x7fb4fc1432e0 2022-07-20T02:34:26.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc126e20 con 0x7fb4fc1432e0 2022-07-20T02:34:26.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb4f4025030 con 0x7fb4fc1432e0 2022-07-20T02:34:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb4fb7fe700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb4f406ce20 con 0x7fb4fc1432e0 2022-07-20T02:34:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb5020ac700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.823+0000 7fb5020ac700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fb4ec00d000 tx=0x7fb4ec007eb0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.831+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4fc126e20 con 0x7fb4fc1432e0 2022-07-20T02:34:27.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.995+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4fc126e20 con 0x7fb4fc1432e0 2022-07-20T02:34:27.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.995+0000 7fb4fb7fe700 1 -- 172.21.15.26:0/2494654991 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1426 (secure 0 0 0) 0x7fb4fc126e20 con 0x7fb4fc1432e0 2022-07-20T02:34:27.008 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:27.014 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":67,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:22.962666+0000","modified":"2022-07-20T02:34:26.638919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-20T02:34:27.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 msgr2=0x7fb4dc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fb4ec00d000 tx=0x7fb4ec007eb0).stop 2022-07-20T02:34:27.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 msgr2=0x7fb4fc0ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4014000).stop 2022-07-20T02:34:27.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 shutdown_connections 2022-07-20T02:34:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc1432e0 0x7fb4fc0ad910 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc12c750 0x7fb4fc144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 --2- 172.21.15.26:0/2494654991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc061a20 0x7fb4fc142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 >> 172.21.15.26:0/2494654991 conn(0x7fb4fc05faf0 msgr2=0x7fb4fc0b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 shutdown_connections 2022-07-20T02:34:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:26.999+0000 7fb504310700 1 -- 172.21.15.26:0/2494654991 wait complete. 2022-07-20T02:34:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 67 2022-07-20T02:34:27.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:34:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.287+0000 7fbce3567700 1 Processor -- start 2022-07-20T02:34:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.287+0000 7fbce3567700 1 -- start start 2022-07-20T02:34:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbcdc0af4a0 0x7fbcdc149d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbcdc0aa070 0x7fbcdc14a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcdc067030 con 0x7fbcdc0aeae0 2022-07-20T02:34:27.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc0aeec0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc066af0 con 0x7fbcdc0aa070 2022-07-20T02:34:27.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbcdc0af4a0 0x7fbcdc149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbcdc0af4a0 0x7fbcdc149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42604/0 (socket says 172.21.15.26:42604) 2022-07-20T02:34:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 -- 172.21.15.26:0/1432666053 learned_addr learned my addr 172.21.15.26:0/1432666053 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce1b04700 1 --2- 172.21.15.26:0/1432666053 >> v2:172.21.15.120:3300/0 conn(0x7fbcdc0aa070 0x7fbcdc14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 -- 172.21.15.26:0/1432666053 >> v2:172.21.15.120:3300/0 conn(0x7fbcdc0aa070 msgr2=0x7fbcdc14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:27.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 --2- 172.21.15.26:0/1432666053 >> v2:172.21.15.120:3300/0 conn(0x7fbcdc0aa070 0x7fbcdc14a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 -- 172.21.15.26:0/1432666053 >> v1:172.21.15.120:6789/0 conn(0x7fbcdc0aeae0 legacy=0x7fbcdc061780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 -- 172.21.15.26:0/1432666053 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc066d90 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce0b02700 1 --2- 172.21.15.26:0/1432666053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc149d70 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7fbccc002f70 tx=0x7fbccc006eb0).ready entity=mon.0 client_cookie=c584ed9e9e082669 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbccbfff700 1 -- 172.21.15.26:0/1432666053 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbccc00d5c0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbccbfff700 1 -- 172.21.15.26:0/1432666053 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc000e90 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbccbfff700 1 -- 172.21.15.26:0/1432666053 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbccc00d950 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- 172.21.15.26:0/1432666053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 msgr2=0x7fbcdc149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 --2- 172.21.15.26:0/1432666053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc149d70 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7fbccc002f70 tx=0x7fbccc006eb0).stop 2022-07-20T02:34:27.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- 172.21.15.26:0/1432666053 shutdown_connections 2022-07-20T02:34:27.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 --2- 172.21.15.26:0/1432666053 >> v2:172.21.15.120:3300/0 conn(0x7fbcdc0aa070 0x7fbcdc14a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 --2- 172.21.15.26:0/1432666053 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc149d70 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- 172.21.15.26:0/1432666053 >> 172.21.15.26:0/1432666053 conn(0x7fbcdc0b66e0 msgr2=0x7fbcdc0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:27.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- 172.21.15.26:0/1432666053 shutdown_connections 2022-07-20T02:34:27.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.291+0000 7fbce3567700 1 -- 172.21.15.26:0/1432666053 wait complete. 2022-07-20T02:34:27.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 Processor -- start 2022-07-20T02:34:27.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 -- start start 2022-07-20T02:34:27.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcdc0aa070 0x7fbcdc0b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcdc0aeae0 0x7fbcdc145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc149160 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc144ec0 con 0x7fbcdc0aeae0 2022-07-20T02:34:27.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce3567700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc146890 con 0x7fbcdc0aa070 2022-07-20T02:34:27.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42612/0 (socket says 172.21.15.26:42612) 2022-07-20T02:34:27.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 -- 172.21.15.26:0/850621019 learned_addr learned my addr 172.21.15.26:0/850621019 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:27.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce0b02700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcdc0aeae0 0x7fbcdc145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1303700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcdc0aa070 0x7fbcdc0b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 -- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcdc0aa070 msgr2=0x7fbcdc0b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcdc0aa070 0x7fbcdc0b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 -- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcdc0aeae0 msgr2=0x7fbcdc145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcdc0aeae0 0x7fbcdc145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 -- 172.21.15.26:0/850621019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc00b1e0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.295+0000 7fbce1b04700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7fbcd400c000 tx=0x7fbcd40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcd4012600 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd4012760 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc146d60 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcd4012ac0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc146ac0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc157050 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbcd4022070 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 0x7fbcbc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fbcd406c6b0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.299+0000 7fbce1303700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 0x7fbcbc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.303+0000 7fbce1303700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 0x7fbcbc055f60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbcd800d000 tx=0x7fbcd8007b50).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.311+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbcd4043380 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.479+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbcdc0602b0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.651+0000 7fbcca7fc700 1 -- 172.21.15.26:0/850621019 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbcd40469a0 con 0x7fbcdc0af4a0 2022-07-20T02:34:27.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 msgr2=0x7fbcbc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 0x7fbcbc055f60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbcd800d000 tx=0x7fbcd8007b50).stop 2022-07-20T02:34:27.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 msgr2=0x7fbcdc1569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7fbcd400c000 tx=0x7fbcd40091d0).stop 2022-07-20T02:34:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 shutdown_connections 2022-07-20T02:34:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbcdc0af4a0 0x7fbcdc1569c0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbcbc053aa0 0x7fbcbc055f60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcdc0aeae0 0x7fbcdc145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 --2- 172.21.15.26:0/850621019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcdc0aa070 0x7fbcdc0b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 >> 172.21.15.26:0/850621019 conn(0x7fbcdc0b66e0 msgr2=0x7fbcdc0b4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:27.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 shutdown_connections 2022-07-20T02:34:27.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.655+0000 7fbce3567700 1 -- 172.21.15.26:0/850621019 wait complete. 2022-07-20T02:34:27.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:34:27.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 Processor -- start 2022-07-20T02:34:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 -- start start 2022-07-20T02:34:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02141468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214061e00 con 0x7f02141432e0 2022-07-20T02:34:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214146db0 con 0x7f0214061a20 2022-07-20T02:34:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.815+0000 7f021921d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0214147040 con 0x7f021412c750 2022-07-20T02:34:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021259c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f02141432e0 0x7f02140b2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41898/0 (socket says 172.21.15.26:41898) 2022-07-20T02:34:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021259c700 1 -- 172.21.15.26:0/976729400 learned_addr learned my addr 172.21.15.26:0/976729400 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021359e700 1 --2- 172.21.15.26:0/976729400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02141468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126318326 0 0) 0x7f0214061e00 con 0x7f02141432e0 2022-07-20T02:34:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f4003260 con 0x7f02141432e0 2022-07-20T02:34:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952594234 0 0) 0x7f0214146db0 con 0x7f0214061a20 2022-07-20T02:34:27.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214061e00 con 0x7f0214061a20 2022-07-20T02:34:27.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021359e700 1 -- 172.21.15.26:0/976729400 >> v1:172.21.15.120:6789/0 conn(0x7f0214061a20 legacy=0x7f021412c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021359e700 1 -- 172.21.15.26:0/976729400 >> v1:172.21.15.26:6789/0 conn(0x7f02141432e0 legacy=0x7f02140b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870777724 0 0) 0x7f01f4003260 con 0x7f02141432e0 2022-07-20T02:34:27.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021359e700 1 -- 172.21.15.26:0/976729400 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02141472e0 con 0x7f021412c750 2022-07-20T02:34:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021359e700 1 --2- 172.21.15.26:0/976729400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02141468c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f01fc001110 tx=0x7f020c0091d0).ready entity=mon.2 client_cookie=e26af8ebb05030f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc001660 con 0x7f021412c750 2022-07-20T02:34:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c00f510 con 0x7f021412c750 2022-07-20T02:34:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f0211d9b700 1 -- 172.21.15.26:0/976729400 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f020c00f680 con 0x7f021412c750 2022-07-20T02:34:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 -- 172.21.15.26:0/976729400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 msgr2=0x7f02141468c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 --2- 172.21.15.26:0/976729400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02141468c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f01fc001110 tx=0x7f020c0091d0).stop 2022-07-20T02:34:27.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 -- 172.21.15.26:0/976729400 shutdown_connections 2022-07-20T02:34:27.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 --2- 172.21.15.26:0/976729400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02141468c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 -- 172.21.15.26:0/976729400 >> 172.21.15.26:0/976729400 conn(0x7f021405faf0 msgr2=0x7f0214066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:27.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 -- 172.21.15.26:0/976729400 shutdown_connections 2022-07-20T02:34:27.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.819+0000 7f021921d700 1 -- 172.21.15.26:0/976729400 wait complete. 2022-07-20T02:34:27.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 Processor -- start 2022-07-20T02:34:27.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- start start 2022-07-20T02:34:27.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0214061a20 0x7f02140b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02140b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02140b5620 con 0x7f02141432e0 2022-07-20T02:34:27.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02140b5850 con 0x7f0214061a20 2022-07-20T02:34:27.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02140ae960 con 0x7f021412c750 2022-07-20T02:34:27.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42626/0 (socket says 172.21.15.26:42626) 2022-07-20T02:34:27.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 -- 172.21.15.26:0/3592102223 learned_addr learned my addr 172.21.15.26:0/3592102223 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:27.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f0212d9d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0214061a20 0x7f02140b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021259c700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02140b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 -- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 msgr2=0x7f02140b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02140b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 -- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0214061a20 msgr2=0x7f02140b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0214061a20 0x7f02140b5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 -- 172.21.15.26:0/3592102223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c009470 con 0x7f02141432e0 2022-07-20T02:34:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021359e700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f01fc001110 tx=0x7f020c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f020c0032d0 con 0x7f02141432e0 2022-07-20T02:34:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c011b80 con 0x7f02141432e0 2022-07-20T02:34:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140ae670 con 0x7f02141432e0 2022-07-20T02:34:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f020c021440 con 0x7f02141432e0 2022-07-20T02:34:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.823+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02140ba6d0 con 0x7f02141432e0 2022-07-20T02:34:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214147580 con 0x7f02141432e0 2022-07-20T02:34:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f020c027020 con 0x7f02141432e0 2022-07-20T02:34:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f020bfff700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 0x7f01f8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f020c06d260 con 0x7f02141432e0 2022-07-20T02:34:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f0212d9d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 0x7f01f8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.827+0000 7f0212d9d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 0x7f01f8055f30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f020400d000 tx=0x7f0204007ad0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:27.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.835+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f020c043ec0 con 0x7f02141432e0 2022-07-20T02:34:27.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:27.959+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f02140574e0 con 0x7f02141432e0 2022-07-20T02:34:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.663+0000 7f020bfff700 1 -- 172.21.15.26:0/3592102223 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 185+0+0 (secure 0 0 0) 0x7f020c0150e0 con 0x7f02141432e0 2022-07-20T02:34:28.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 msgr2=0x7f01f8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 0x7f01f8055f30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f020400d000 tx=0x7f0204007ad0).stop 2022-07-20T02:34:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 msgr2=0x7f0214148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f01fc001110 tx=0x7f020c016000).stop 2022-07-20T02:34:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 shutdown_connections 2022-07-20T02:34:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02141432e0 0x7f0214148040 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01f8053a70 0x7f01f8055f30 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021412c750 0x7f02140b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 --2- 172.21.15.26:0/3592102223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0214061a20 0x7f02140b5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 >> 172.21.15.26:0/3592102223 conn(0x7f021405faf0 msgr2=0x7f0214142590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 shutdown_connections 2022-07-20T02:34:28.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.667+0000 7f021921d700 1 -- 172.21.15.26:0/3592102223 wait complete. 2022-07-20T02:34:28.721 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:34:28.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:34:28.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.839+0000 7f47312e5700 1 Processor -- start 2022-07-20T02:34:28.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.839+0000 7f47312e5700 1 -- start start 2022-07-20T02:34:28.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47312e5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f472c1465b0 0x7f472c066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47312e5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f472c0672a0 0x7f472c146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47312e5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0b1ae0 con 0x7f472c0b24c0 2022-07-20T02:34:28.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47312e5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0b2e20 con 0x7f472c0672a0 2022-07-20T02:34:28.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47312e5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0b1d80 con 0x7f472c1465b0 2022-07-20T02:34:28.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f472c0672a0 0x7f472c146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47237fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f472c0b24c0 0x7f472c0604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41914/0 (socket says 172.21.15.26:41914) 2022-07-20T02:34:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f47237fe700 1 -- 172.21.15.26:0/2764676961 learned_addr learned my addr 172.21.15.26:0/2764676961 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f472bfff700 1 --2- 172.21.15.26:0/2764676961 >> v2:172.21.15.120:3300/0 conn(0x7f472c1465b0 0x7f472c066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 -- 172.21.15.26:0/2764676961 >> v2:172.21.15.120:3300/0 conn(0x7f472c1465b0 msgr2=0x7f472c066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:28.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 --2- 172.21.15.26:0/2764676961 >> v2:172.21.15.120:3300/0 conn(0x7f472c1465b0 0x7f472c066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 -- 172.21.15.26:0/2764676961 >> v1:172.21.15.26:6789/0 conn(0x7f472c0b24c0 legacy=0x7f472c0604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 -- 172.21.15.26:0/2764676961 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0b2020 con 0x7f472c0672a0 2022-07-20T02:34:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f472b7fe700 1 -- 172.21.15.26:0/2764676961 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324167285 0 0) 0x7f472c0b1ae0 con 0x7f472c0b24c0 2022-07-20T02:34:28.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f4730ae4700 1 --2- 172.21.15.26:0/2764676961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c146990 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f47240032f0 tx=0x7f47240091d0).ready entity=mon.0 client_cookie=9418898bdd7cc9ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f472b7fe700 1 -- 172.21.15.26:0/2764676961 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472400b680 con 0x7f472c0672a0 2022-07-20T02:34:28.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f472b7fe700 1 -- 172.21.15.26:0/2764676961 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472400b7e0 con 0x7f472c0672a0 2022-07-20T02:34:28.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.843+0000 7f472b7fe700 1 -- 172.21.15.26:0/2764676961 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472400b950 con 0x7f472c0672a0 2022-07-20T02:34:28.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- 172.21.15.26:0/2764676961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 msgr2=0x7f472c146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 --2- 172.21.15.26:0/2764676961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c146990 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f47240032f0 tx=0x7f47240091d0).stop 2022-07-20T02:34:28.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- 172.21.15.26:0/2764676961 shutdown_connections 2022-07-20T02:34:28.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 --2- 172.21.15.26:0/2764676961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c146990 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 --2- 172.21.15.26:0/2764676961 >> v2:172.21.15.120:3300/0 conn(0x7f472c1465b0 0x7f472c066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- 172.21.15.26:0/2764676961 >> 172.21.15.26:0/2764676961 conn(0x7f472c0b3050 msgr2=0x7f472c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:28.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- 172.21.15.26:0/2764676961 shutdown_connections 2022-07-20T02:34:28.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- 172.21.15.26:0/2764676961 wait complete. 2022-07-20T02:34:28.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 Processor -- start 2022-07-20T02:34:28.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.847+0000 7f47312e5700 1 -- start start 2022-07-20T02:34:28.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c1465b0 0x7f472c065a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 0x7f472c065ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c156890 con 0x7f472c0672a0 2022-07-20T02:34:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c155980 con 0x7f472c140e80 2022-07-20T02:34:28.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0ad4f0 con 0x7f472c1465b0 2022-07-20T02:34:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42636/0 (socket says 172.21.15.26:42636) 2022-07-20T02:34:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 -- 172.21.15.26:0/3467701502 learned_addr learned my addr 172.21.15.26:0/3467701502 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47237fe700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c1465b0 0x7f472c065a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4730ae4700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 0x7f472c065ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 -- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c1465b0 msgr2=0x7f472c065a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c1465b0 0x7f472c065a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 -- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 msgr2=0x7f472c065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 0x7f472c065ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 -- 172.21.15.26:0/3467701502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009470 con 0x7f472c0672a0 2022-07-20T02:34:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4730ae4700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 0x7f472c065ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:28.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f472bfff700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f471c00c000 tx=0x7f471c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:28.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c010780 con 0x7f472c0672a0 2022-07-20T02:34:28.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c0108e0 con 0x7f472c0672a0 2022-07-20T02:34:28.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c010c40 con 0x7f472c0672a0 2022-07-20T02:34:28.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c155bb0 con 0x7f472c0672a0 2022-07-20T02:34:28.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c0663e0 con 0x7f472c0672a0 2022-07-20T02:34:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c0574e0 con 0x7f472c0672a0 2022-07-20T02:34:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.851+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f471c01c460 con 0x7f472c0672a0 2022-07-20T02:34:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.855+0000 7f4729ffb700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 0x7f4710055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.855+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f471c06c320 con 0x7f472c0672a0 2022-07-20T02:34:28.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.855+0000 7f47237fe700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 0x7f4710055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:28.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.855+0000 7f47237fe700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 0x7f4710055ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f471400d000 tx=0x7f4714009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:28.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.859+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f471c043140 con 0x7f472c0672a0 2022-07-20T02:34:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:28.983+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f472c0b6390 con 0x7f472c0672a0 2022-07-20T02:34:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.683+0000 7f4729ffb700 1 -- 172.21.15.26:0/3467701502 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 173+0+0 (secure 0 0 0) 0x7f471c046760 con 0x7f472c0672a0 2022-07-20T02:34:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 msgr2=0x7f4710055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 0x7f4710055ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f471400d000 tx=0x7f4714009040).stop 2022-07-20T02:34:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 msgr2=0x7f472c143120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f471c00c000 tx=0x7f471c006df0).stop 2022-07-20T02:34:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 shutdown_connections 2022-07-20T02:34:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4710053a10 0x7f4710055ed0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c140e80 0x7f472c065ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c1465b0 0x7f472c065a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 --2- 172.21.15.26:0/3467701502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f472c0672a0 0x7f472c143120 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 >> 172.21.15.26:0/3467701502 conn(0x7f472c0b3050 msgr2=0x7f472c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:29.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 shutdown_connections 2022-07-20T02:34:29.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.691+0000 7f47312e5700 1 -- 172.21.15.26:0/3467701502 wait complete. 2022-07-20T02:34:29.701 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:34:29.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:34:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.875+0000 7f6db8f2d700 1 Processor -- start 2022-07-20T02:34:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.875+0000 7f6db8f2d700 1 -- start start 2022-07-20T02:34:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6db41465b0 0x7f6db4066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db40b1ae0 con 0x7f6db40b24c0 2022-07-20T02:34:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db40b2e20 con 0x7f6db41465b0 2022-07-20T02:34:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40b1d80 con 0x7f6db40672a0 2022-07-20T02:34:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6db41465b0 0x7f6db4066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db2ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6db40b24c0 0x7f6db40604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41924/0 (socket says 172.21.15.26:41924) 2022-07-20T02:34:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db2ffd700 1 -- 172.21.15.26:0/383168836 learned_addr learned my addr 172.21.15.26:0/383168836 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db3fff700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db27fc700 1 -- 172.21.15.26:0/383168836 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744564655 0 0) 0x7f6db40b1ae0 con 0x7f6db40b24c0 2022-07-20T02:34:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db27fc700 1 -- 172.21.15.26:0/383168836 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da0003260 con 0x7f6db40b24c0 2022-07-20T02:34:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 -- 172.21.15.26:0/383168836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 msgr2=0x7f6db4146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 -- 172.21.15.26:0/383168836 >> v1:172.21.15.26:6789/0 conn(0x7f6db40b24c0 legacy=0x7f6db40604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 -- 172.21.15.26:0/383168836 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40b2020 con 0x7f6db41465b0 2022-07-20T02:34:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db37fe700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4066db0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f6da4002f70 tx=0x7f6da4006df0).ready entity=mon.0 client_cookie=6e981aa14d9a10df server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db27fc700 1 -- 172.21.15.26:0/383168836 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da400d4a0 con 0x7f6db41465b0 2022-07-20T02:34:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db27fc700 1 -- 172.21.15.26:0/383168836 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da4000e90 con 0x7f6db41465b0 2022-07-20T02:34:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db27fc700 1 -- 172.21.15.26:0/383168836 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da400d7f0 con 0x7f6db41465b0 2022-07-20T02:34:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- 172.21.15.26:0/383168836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 msgr2=0x7f6db4066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4066db0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f6da4002f70 tx=0x7f6da4006df0).stop 2022-07-20T02:34:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- 172.21.15.26:0/383168836 shutdown_connections 2022-07-20T02:34:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/383168836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4066db0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- 172.21.15.26:0/383168836 >> 172.21.15.26:0/383168836 conn(0x7f6db40b3050 msgr2=0x7f6db40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- 172.21.15.26:0/383168836 shutdown_connections 2022-07-20T02:34:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.879+0000 7f6db8f2d700 1 -- 172.21.15.26:0/383168836 wait complete. 2022-07-20T02:34:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 Processor -- start 2022-07-20T02:34:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 -- start start 2022-07-20T02:34:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4064260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db40aab90 0x7f6db40abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40ab220 con 0x7f6db41465b0 2022-07-20T02:34:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.883+0000 7f6db8f2d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40ab480 con 0x7f6db40aab90 2022-07-20T02:34:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db8f2d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40ab720 con 0x7f6db40672a0 2022-07-20T02:34:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42652/0 (socket says 172.21.15.26:42652) 2022-07-20T02:34:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 -- 172.21.15.26:0/2774940679 learned_addr learned my addr 172.21.15.26:0/2774940679 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db37fe700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4064260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db3fff700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db40aab90 0x7f6db40abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 -- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 msgr2=0x7f6db4064260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4064260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 -- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db40aab90 msgr2=0x7f6db40abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db40aab90 0x7f6db40abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 -- 172.21.15.26:0/2774940679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da400b180 con 0x7f6db41465b0 2022-07-20T02:34:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db2ffd700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f6da8001110 tx=0x7f6da8007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da80070d0 con 0x7f6db41465b0 2022-07-20T02:34:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8007230 con 0x7f6db41465b0 2022-07-20T02:34:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db40ab9c0 con 0x7f6db41465b0 2022-07-20T02:34:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da8005610 con 0x7f6db41465b0 2022-07-20T02:34:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.887+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db4156650 con 0x7f6db41465b0 2022-07-20T02:34:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db40abb50 con 0x7f6db41465b0 2022-07-20T02:34:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6da801a070 con 0x7f6db41465b0 2022-07-20T02:34:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db0ff9700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 0x7f6d94055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6da806ad80 con 0x7f6db41465b0 2022-07-20T02:34:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db37fe700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 0x7f6d94055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:29.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.891+0000 7f6db37fe700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 0x7f6d94055e10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f6da401a000 tx=0x7f6da4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:29.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:29.895+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6da8041dc0 con 0x7f6db41465b0 2022-07-20T02:34:30.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.023+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6db4060bf0 con 0x7f6db41465b0 2022-07-20T02:34:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db0ff9700 1 -- 172.21.15.26:0/2774940679 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6da806c020 con 0x7f6db41465b0 2022-07-20T02:34:30.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 msgr2=0x7f6d94055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 0x7f6d94055e10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f6da401a000 tx=0x7f6da4014000).stop 2022-07-20T02:34:30.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 msgr2=0x7f6db4156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f6da8001110 tx=0x7f6da8007b20).stop 2022-07-20T02:34:30.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 shutdown_connections 2022-07-20T02:34:30.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6db41465b0 0x7f6db4156160 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d94053950 0x7f6d94055e10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6db40aab90 0x7f6db40abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 --2- 172.21.15.26:0/2774940679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6db40672a0 0x7f6db4064260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 >> 172.21.15.26:0/2774940679 conn(0x7f6db40b3050 msgr2=0x7f6db413f890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 shutdown_connections 2022-07-20T02:34:30.050 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.031+0000 7f6db8f2d700 1 -- 172.21.15.26:0/2774940679 wait complete. 2022-07-20T02:34:30.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-20T02:34:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.163+0000 7f8fd86dc700 1 Processor -- start 2022-07-20T02:34:30.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- start start 2022-07-20T02:34:30.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 0x7f8fd0061bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00620b0 con 0x7f8fd00adf90 2022-07-20T02:34:30.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00ae370 con 0x7f8fd00b14b0 2022-07-20T02:34:30.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00b28b0 con 0x7f8fd00b2c60 2022-07-20T02:34:30.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6478700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8fd00adf90 0x7f8fd0060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41942/0 (socket says 172.21.15.26:41942) 2022-07-20T02:34:30.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6478700 1 -- 172.21.15.26:0/2972532500 learned_addr learned my addr 172.21.15.26:0/2972532500 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:30.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6c79700 1 --2- 172.21.15.26:0/2972532500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 0x7f8fd0061bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350873327 0 0) 0x7f8fd00620b0 con 0x7f8fd00adf90 2022-07-20T02:34:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb8003260 con 0x7f8fd00adf90 2022-07-20T02:34:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244260093 0 0) 0x7f8fd00ae370 con 0x7f8fd00b14b0 2022-07-20T02:34:30.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00620b0 con 0x7f8fd00b14b0 2022-07-20T02:34:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6c79700 1 -- 172.21.15.26:0/2972532500 >> v1:172.21.15.120:6789/0 conn(0x7f8fd00b14b0 legacy=0x7f8fd00614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6c79700 1 -- 172.21.15.26:0/2972532500 >> v1:172.21.15.26:6789/0 conn(0x7f8fd00adf90 legacy=0x7f8fd0060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6c79700 1 -- 172.21.15.26:0/2972532500 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00b0610 con 0x7f8fd00b2c60 2022-07-20T02:34:30.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd6c79700 1 --2- 172.21.15.26:0/2972532500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 0x7f8fd0061bc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8fcc001820 tx=0x7f8fc80091d0).ready entity=mon.2 client_cookie=61a8ee5caccfe8a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fcc001680 con 0x7f8fd00b2c60 2022-07-20T02:34:30.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc800b500 con 0x7f8fd00b2c60 2022-07-20T02:34:30.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd5476700 1 -- 172.21.15.26:0/2972532500 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc800b670 con 0x7f8fd00b2c60 2022-07-20T02:34:30.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- 172.21.15.26:0/2972532500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 msgr2=0x7f8fd0061bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/2972532500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 0x7f8fd0061bc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8fcc001820 tx=0x7f8fc80091d0).stop 2022-07-20T02:34:30.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- 172.21.15.26:0/2972532500 shutdown_connections 2022-07-20T02:34:30.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/2972532500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00b2c60 0x7f8fd0061bc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- 172.21.15.26:0/2972532500 >> 172.21.15.26:0/2972532500 conn(0x7f8fd00b66e0 msgr2=0x7f8fd00b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:30.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- 172.21.15.26:0/2972532500 shutdown_connections 2022-07-20T02:34:30.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.167+0000 7f8fd86dc700 1 -- 172.21.15.26:0/2972532500 wait complete. 2022-07-20T02:34:30.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 Processor -- start 2022-07-20T02:34:30.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 -- start start 2022-07-20T02:34:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00adf90 0x7f8fd0146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd00b2c60 0x7f8fd01500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd009f010 con 0x7f8fd00b14b0 2022-07-20T02:34:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd0064aa0 con 0x7f8fd00b2c60 2022-07-20T02:34:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd86dc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd009f240 con 0x7f8fd00adf90 2022-07-20T02:34:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42666/0 (socket says 172.21.15.26:42666) 2022-07-20T02:34:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 -- 172.21.15.26:0/1807439041 learned_addr learned my addr 172.21.15.26:0/1807439041 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd6c79700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd00b2c60 0x7f8fd01500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd6478700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00adf90 0x7f8fd0146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 -- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00adf90 msgr2=0x7f8fd0146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00adf90 0x7f8fd0146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 -- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd00b2c60 msgr2=0x7f8fd01500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd00b2c60 0x7f8fd01500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 -- 172.21.15.26:0/1807439041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8009470 con 0x7f8fd00b14b0 2022-07-20T02:34:30.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fd5c77700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f8fc0001110 tx=0x7f8fc0007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.171+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc000ce50 con 0x7f8fd00b14b0 2022-07-20T02:34:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc000f470 con 0x7f8fd00b14b0 2022-07-20T02:34:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd0064cd0 con 0x7f8fd00b14b0 2022-07-20T02:34:30.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc000f7d0 con 0x7f8fd00b14b0 2022-07-20T02:34:30.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd0065900 con 0x7f8fd00b14b0 2022-07-20T02:34:30.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd00574e0 con 0x7f8fd00b14b0 2022-07-20T02:34:30.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8fc0019030 con 0x7f8fd00b14b0 2022-07-20T02:34:30.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fbf7fe700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 0x7f8fb0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fd6478700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 0x7f8fb0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f8fc006b170 con 0x7f8fd00b14b0 2022-07-20T02:34:30.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.175+0000 7f8fd6478700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 0x7f8fb0055ed0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f8fcc00d000 tx=0x7f8fcc001a00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.183+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fc0042210 con 0x7f8fd00b14b0 2022-07-20T02:34:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.347+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f8fd00683d0 con 0x7f8fd00b14b0 2022-07-20T02:34:30.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.679+0000 7f8fbf7fe700 1 -- 172.21.15.26:0/1807439041 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v415) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8fc0045830 con 0x7f8fd00b14b0 2022-07-20T02:34:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 msgr2=0x7f8fb0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 0x7f8fb0055ed0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f8fcc00d000 tx=0x7f8fcc001a00).stop 2022-07-20T02:34:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 msgr2=0x7f8fd0065410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f8fc0001110 tx=0x7f8fc0007b60).stop 2022-07-20T02:34:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 shutdown_connections 2022-07-20T02:34:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8fd00b14b0 0x7f8fd0065410 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8fb0053a10 0x7f8fb0055ed0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd00b2c60 0x7f8fd01500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 --2- 172.21.15.26:0/1807439041 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fd00adf90 0x7f8fd0146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 >> 172.21.15.26:0/1807439041 conn(0x7f8fd00b66e0 msgr2=0x7f8fd0066b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 shutdown_connections 2022-07-20T02:34:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.683+0000 7f8fd86dc700 1 -- 172.21.15.26:0/1807439041 wait complete. 2022-07-20T02:34:30.708 INFO:tasks.cephfs_test_runner:test_fs_new (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:34:30.709 INFO:tasks.cephfs_test_runner:Starting test: test_getattr_caps (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:34:30.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-20T02:34:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.819+0000 7f77568bb700 1 Processor -- start 2022-07-20T02:34:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.819+0000 7f77568bb700 1 -- start start 2022-07-20T02:34:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77500b5ef0 0x7f77501474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f77501479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500b62d0 con 0x7f77500a8c40 2022-07-20T02:34:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7750147eb0 con 0x7f77500b5ef0 2022-07-20T02:34:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775014aac0 con 0x7f77500a7fc0 2022-07-20T02:34:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f77500b5ef0 0x7f77501474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77560ba700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f77500a8c40 0x7f77500aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41956/0 (socket says 172.21.15.26:41956) 2022-07-20T02:34:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77560ba700 1 -- 172.21.15.26:0/3136977613 learned_addr learned my addr 172.21.15.26:0/3136977613 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77550b8700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f77501479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77548b7700 1 -- 172.21.15.26:0/3136977613 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868934147 0 0) 0x7f77500b62d0 con 0x7f77500a8c40 2022-07-20T02:34:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77548b7700 1 -- 172.21.15.26:0/3136977613 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7738003260 con 0x7f77500a8c40 2022-07-20T02:34:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 -- 172.21.15.26:0/3136977613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 msgr2=0x7f77501479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f77501479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 -- 172.21.15.26:0/3136977613 >> v1:172.21.15.26:6789/0 conn(0x7f77500a8c40 legacy=0x7f77500aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 -- 172.21.15.26:0/3136977613 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775014ad60 con 0x7f77500b5ef0 2022-07-20T02:34:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77558b9700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500b5ef0 0x7f77501474d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f7740002f70 tx=0x7f7740006df0).ready entity=mon.0 client_cookie=58c3a75d47349b2d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77548b7700 1 -- 172.21.15.26:0/3136977613 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f774000d4a0 con 0x7f77500b5ef0 2022-07-20T02:34:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77548b7700 1 -- 172.21.15.26:0/3136977613 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7740000e90 con 0x7f77500b5ef0 2022-07-20T02:34:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77548b7700 1 -- 172.21.15.26:0/3136977613 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f774000d7f0 con 0x7f77500b5ef0 2022-07-20T02:34:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- 172.21.15.26:0/3136977613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500b5ef0 msgr2=0x7f77501474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500b5ef0 0x7f77501474d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f7740002f70 tx=0x7f7740006df0).stop 2022-07-20T02:34:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- 172.21.15.26:0/3136977613 shutdown_connections 2022-07-20T02:34:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f77501479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 --2- 172.21.15.26:0/3136977613 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500b5ef0 0x7f77501474d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- 172.21.15.26:0/3136977613 >> 172.21.15.26:0/3136977613 conn(0x7f7750060310 msgr2=0x7f77500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- 172.21.15.26:0/3136977613 shutdown_connections 2022-07-20T02:34:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.823+0000 7f77568bb700 1 -- 172.21.15.26:0/3136977613 wait complete. 2022-07-20T02:34:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 Processor -- start 2022-07-20T02:34:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- start start 2022-07-20T02:34:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f775015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77500b5ef0 0x7f775011d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7750157420 con 0x7f77500a8c40 2022-07-20T02:34:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500b3340 con 0x7f77500b5ef0 2022-07-20T02:34:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775015a930 con 0x7f77500a7fc0 2022-07-20T02:34:30.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42678/0 (socket says 172.21.15.26:42678) 2022-07-20T02:34:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 -- 172.21.15.26:0/3030092443 learned_addr learned my addr 172.21.15.26:0/3030092443 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77560ba700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77500b5ef0 0x7f775011d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77558b9700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f775015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 -- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 msgr2=0x7f775015a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f775015a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 -- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77500b5ef0 msgr2=0x7f775011d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77500b5ef0 0x7f775011d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 -- 172.21.15.26:0/3030092443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774000b180 con 0x7f77500a8c40 2022-07-20T02:34:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77550b8700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f774c00c000 tx=0x7f774c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f774c00c820 con 0x7f77500a8c40 2022-07-20T02:34:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c00c980 con 0x7f77500a8c40 2022-07-20T02:34:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f774c00cce0 con 0x7f77500a8c40 2022-07-20T02:34:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7750064b40 con 0x7f77500a8c40 2022-07-20T02:34:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.827+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7750065010 con 0x7f77500a8c40 2022-07-20T02:34:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77500b36d0 con 0x7f77500a8c40 2022-07-20T02:34:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f774c018020 con 0x7f77500a8c40 2022-07-20T02:34:30.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f773effd700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 0x7f7730055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:30.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f774c06c4c0 con 0x7f77500a8c40 2022-07-20T02:34:30.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f77558b9700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 0x7f7730055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:30.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.831+0000 7f77558b9700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 0x7f7730055ea0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f7740018000 tx=0x7f7740014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:30.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:30.843+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f774c043460 con 0x7f77500a8c40 2022-07-20T02:34:31.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.011+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f775014b000 con 0x7f77500a8c40 2022-07-20T02:34:31.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.683+0000 7f773effd700 1 -- 172.21.15.26:0/3030092443 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v416) v1 ==== 131+0+0 (secure 0 0 0) 0x7f774c0212a0 con 0x7f77500a8c40 2022-07-20T02:34:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 msgr2=0x7f7730055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 0x7f7730055ea0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f7740018000 tx=0x7f7740014000).stop 2022-07-20T02:34:31.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 msgr2=0x7f77500641c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f774c00c000 tx=0x7f774c0091d0).stop 2022-07-20T02:34:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 shutdown_connections 2022-07-20T02:34:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f77500a8c40 0x7f77500641c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f77300539e0 0x7f7730055ea0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77500b5ef0 0x7f775011d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 --2- 172.21.15.26:0/3030092443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77500a7fc0 0x7f775015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 >> 172.21.15.26:0/3030092443 conn(0x7f7750060310 msgr2=0x7f77500ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 shutdown_connections 2022-07-20T02:34:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.687+0000 7f77568bb700 1 -- 172.21.15.26:0/3030092443 wait complete. 2022-07-20T02:34:31.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:34:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 Processor -- start 2022-07-20T02:34:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 -- start start 2022-07-20T02:34:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff8dc0b31b0 0x7ff8dc0614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:31.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc0608d0 con 0x7ff8dc0b2c70 2022-07-20T02:34:31.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc0b7d00 con 0x7ff8dc0b31b0 2022-07-20T02:34:31.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.875+0000 7ff8e1e81700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0b4120 con 0x7ff8dc0b7920 2022-07-20T02:34:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8dbfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff8dc0b31b0 0x7ff8dc0614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1680700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff8dc0b2c70 0x7ff8dc0b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56182/0 (socket says 172.21.15.26:56182) 2022-07-20T02:34:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1680700 1 -- 172.21.15.26:0/597368899 learned_addr learned my addr 172.21.15.26:0/597368899 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8db7fe700 1 -- 172.21.15.26:0/597368899 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554636677 0 0) 0x7ff8dc0608d0 con 0x7ff8dc0b2c70 2022-07-20T02:34:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8db7fe700 1 -- 172.21.15.26:0/597368899 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c8003260 con 0x7ff8dc0b2c70 2022-07-20T02:34:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 -- 172.21.15.26:0/597368899 >> v2:172.21.15.120:3300/0 conn(0x7ff8dc0b31b0 msgr2=0x7ff8dc0614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 --2- 172.21.15.26:0/597368899 >> v2:172.21.15.120:3300/0 conn(0x7ff8dc0b31b0 0x7ff8dc0614e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 -- 172.21.15.26:0/597368899 >> v1:172.21.15.120:6789/0 conn(0x7ff8dc0b2c70 legacy=0x7ff8dc0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 -- 172.21.15.26:0/597368899 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc060b00 con 0x7ff8dc0b7920 2022-07-20T02:34:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e0e7f700 1 --2- 172.21.15.26:0/597368899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc0b3c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8cc002f70 tx=0x7ff8cc006d90).ready entity=mon.2 client_cookie=6cea9a1f08b021ed server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8db7fe700 1 -- 172.21.15.26:0/597368899 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8cc00e590 con 0x7ff8dc0b7920 2022-07-20T02:34:31.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8db7fe700 1 -- 172.21.15.26:0/597368899 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc000e90 con 0x7ff8dc0b7920 2022-07-20T02:34:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8db7fe700 1 -- 172.21.15.26:0/597368899 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8cc00e8e0 con 0x7ff8dc0b7920 2022-07-20T02:34:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 -- 172.21.15.26:0/597368899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 msgr2=0x7ff8dc0b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/597368899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc0b3c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8cc002f70 tx=0x7ff8cc006d90).stop 2022-07-20T02:34:31.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 -- 172.21.15.26:0/597368899 shutdown_connections 2022-07-20T02:34:31.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/597368899 >> v2:172.21.15.120:3300/0 conn(0x7ff8dc0b31b0 0x7ff8dc0614e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/597368899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc0b3c30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 -- 172.21.15.26:0/597368899 >> 172.21.15.26:0/597368899 conn(0x7ff8dc05f840 msgr2=0x7ff8dc05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 -- 172.21.15.26:0/597368899 shutdown_connections 2022-07-20T02:34:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.879+0000 7ff8e1e81700 1 -- 172.21.15.26:0/597368899 wait complete. 2022-07-20T02:34:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 Processor -- start 2022-07-20T02:34:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 -- start start 2022-07-20T02:34:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8dc0b2c70 0x7ff8dc0af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0ad5f0 con 0x7ff8dc0b31b0 2022-07-20T02:34:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc153110 con 0x7ff8dc0b2c70 2022-07-20T02:34:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1533b0 con 0x7ff8dc0b7920 2022-07-20T02:34:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42694/0 (socket says 172.21.15.26:42694) 2022-07-20T02:34:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 -- 172.21.15.26:0/1235486158 learned_addr learned my addr 172.21.15.26:0/1235486158 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e0e7f700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8dc0b2c70 0x7ff8dc0af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1680700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 -- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 msgr2=0x7ff8dc1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 -- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8dc0b2c70 msgr2=0x7ff8dc0af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8dc0b2c70 0x7ff8dc0af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 -- 172.21.15.26:0/1235486158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc00b120 con 0x7ff8dc0b31b0 2022-07-20T02:34:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8dbfff700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7ff8d000c000 tx=0x7ff8d0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8d0010780 con 0x7ff8dc0b31b0 2022-07-20T02:34:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d00108e0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8d0010c40 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.883+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc0aceb0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc0ad7c0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc060da0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff8d001c460 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8d9ffb700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 0x7ff8bc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7ff8d006c2b0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8e0e7f700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 0x7ff8bc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.887+0000 7ff8e0e7f700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 0x7ff8bc055e40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7ff8cc01a000 tx=0x7ff8cc014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:31.895+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8d0043250 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.023+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff8dc0574e0 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8d9ffb700 1 -- 172.21.15.26:0/1235486158 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff8d0046870 con 0x7ff8dc0b31b0 2022-07-20T02:34:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 msgr2=0x7ff8bc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 0x7ff8bc055e40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7ff8cc01a000 tx=0x7ff8cc014000).stop 2022-07-20T02:34:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 msgr2=0x7ff8dc0b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7ff8d000c000 tx=0x7ff8d0006df0).stop 2022-07-20T02:34:32.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 shutdown_connections 2022-07-20T02:34:32.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8dc0b31b0 0x7ff8dc0b0f40 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8bc053980 0x7ff8bc055e40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8dc0b7920 0x7ff8dc1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 --2- 172.21.15.26:0/1235486158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8dc0b2c70 0x7ff8dc0af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 >> 172.21.15.26:0/1235486158 conn(0x7ff8dc05f840 msgr2=0x7ff8dc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 shutdown_connections 2022-07-20T02:34:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.031+0000 7ff8e1e81700 1 -- 172.21.15.26:0/1235486158 wait complete. 2022-07-20T02:34:32.084 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:34:32.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:32.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:32.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:32.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:32.310 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:34:32.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:32.311 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:32.311 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:32.340 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:34:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:32.344 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:32.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:32.346 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:32.346 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:34:32.383 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:34:32.384 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:32.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:34:32.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 Processor -- start 2022-07-20T02:34:32.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 -- start start 2022-07-20T02:34:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6674067350 0x7f66741473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b13f0 con 0x7f66740b0f50 2022-07-20T02:34:32.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b1690 con 0x7f667405f9e0 2022-07-20T02:34:32.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.503+0000 7f667a729700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66740b1930 con 0x7f6674067350 2022-07-20T02:34:32.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6678f26700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f66740b0f50 0x7f6674066c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41980/0 (socket says 172.21.15.26:41980) 2022-07-20T02:34:32.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6678f26700 1 -- 172.21.15.26:0/4287237407 learned_addr learned my addr 172.21.15.26:0/4287237407 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:32.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 --2- 172.21.15.26:0/4287237407 >> v2:172.21.15.120:3300/0 conn(0x7f6674067350 0x7f66741473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971823182 0 0) 0x7f66740b13f0 con 0x7f66740b0f50 2022-07-20T02:34:32.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003260 con 0x7f66740b0f50 2022-07-20T02:34:32.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486221923 0 0) 0x7f6660003260 con 0x7f66740b0f50 2022-07-20T02:34:32.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 >> v2:172.21.15.120:3300/0 conn(0x7f6674067350 msgr2=0x7f66741473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:32.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 --2- 172.21.15.26:0/4287237407 >> v2:172.21.15.120:3300/0 conn(0x7f6674067350 0x7f66741473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 >> v1:172.21.15.120:6789/0 conn(0x7f667405f9e0 legacy=0x7f66740b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740b1bd0 con 0x7f66740b0f50 2022-07-20T02:34:32.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f66700019e0 con 0x7f66740b0f50 2022-07-20T02:34:32.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 >> v1:172.21.15.26:6789/0 conn(0x7f66740b0f50 legacy=0x7f6674066c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6660003750 0x7f6660003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004500 0x7f6660004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 0x7f666000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66740b1930 con 0x7f6660004e70 2022-07-20T02:34:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 0x7f666000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6678f26700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004500 0x7f6660004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679727700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6660003750 0x7f6660003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666000bd20 con 0x7f6660004500 2022-07-20T02:34:32.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666000bee0 con 0x7f6660003750 2022-07-20T02:34:32.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 -- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6660003750 msgr2=0x7f6660003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:32.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6660003750 0x7f6660003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 -- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004500 msgr2=0x7f6660004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:32.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004500 0x7f6660004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 -- 172.21.15.26:0/4287237407 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660003260 con 0x7f6660004e70 2022-07-20T02:34:32.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 -- 172.21.15.26:0/4287237407 reap_dead start 2022-07-20T02:34:32.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f6679f28700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 0x7f666000b750 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7f666c00c000 tx=0x7f666c006df0).ready entity=mon.0 client_cookie=2126c4b4ebdddefa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:32.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c016070 con 0x7f6660004e70 2022-07-20T02:34:32.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c00e5f0 con 0x7f6660004e70 2022-07-20T02:34:32.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.507+0000 7f666bfff700 1 -- 172.21.15.26:0/4287237407 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c00e950 con 0x7f6660004e70 2022-07-20T02:34:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 msgr2=0x7f666000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 0x7f666000b750 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7f666c00c000 tx=0x7f666c006df0).stop 2022-07-20T02:34:32.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- 172.21.15.26:0/4287237407 shutdown_connections 2022-07-20T02:34:32.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 --2- 172.21.15.26:0/4287237407 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6660004e70 0x7f666000b750 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- 172.21.15.26:0/4287237407 >> 172.21.15.26:0/4287237407 conn(0x7f66740b3050 msgr2=0x7f66740b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:32.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- 172.21.15.26:0/4287237407 shutdown_connections 2022-07-20T02:34:32.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- 172.21.15.26:0/4287237407 wait complete. 2022-07-20T02:34:32.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 Processor -- start 2022-07-20T02:34:32.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- start start 2022-07-20T02:34:32.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 0x7f66740b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 0x7f6674147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66740ac880 0x7f6674147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66740acd90 con 0x7f6674063b10 2022-07-20T02:34:32.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667415aa70 con 0x7f6660004e70 2022-07-20T02:34:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f667a729700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66740acfc0 con 0x7f66740ac880 2022-07-20T02:34:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6678f26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 0x7f6674147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 0x7f66740b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6678f26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 0x7f6674147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42714/0 (socket says 172.21.15.26:42714) 2022-07-20T02:34:32.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6678f26700 1 -- 172.21.15.26:0/4099017945 learned_addr learned my addr 172.21.15.26:0/4099017945 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:32.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679f28700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66740ac880 0x7f6674147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 -- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66740ac880 msgr2=0x7f6674147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66740ac880 0x7f6674147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 -- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 msgr2=0x7f6674147600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 0x7f6674147600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.511+0000 7f6679727700 1 -- 172.21.15.26:0/4099017945 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667415a530 con 0x7f6660004e70 2022-07-20T02:34:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f6679727700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 0x7f66740b0570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f666400d000 tx=0x7f6664007a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66640133f0 con 0x7f6660004e70 2022-07-20T02:34:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6664013550 con 0x7f6660004e70 2022-07-20T02:34:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667415a7d0 con 0x7f6660004e70 2022-07-20T02:34:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66640138b0 con 0x7f6660004e70 2022-07-20T02:34:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66740665f0 con 0x7f6660004e70 2022-07-20T02:34:32.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66740aa0e0 con 0x7f6660004e70 2022-07-20T02:34:32.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f666401a030 con 0x7f6660004e70 2022-07-20T02:34:32.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 0x7f6654055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:32.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f666406b880 con 0x7f6660004e70 2022-07-20T02:34:32.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f6678f26700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 0x7f6654055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:32.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.515+0000 7f6678f26700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 0x7f6654055fc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f6670005000 tx=0x7f66740b1d60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:32.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.523+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6664042840 con 0x7f6660004e70 2022-07-20T02:34:32.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.647+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f66740574e0 con 0x7f6660004e70 2022-07-20T02:34:32.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.651+0000 7f666a7fc700 1 -- 172.21.15.26:0/4099017945 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v134) v1 ==== 81+0+29275 (secure 0 0 0) 0x7f6664029270 con 0x7f6660004e70 2022-07-20T02:34:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:32.664 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:32.664 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 134, 2022-07-20T02:34:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:34:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:34:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:34:29.672246+0000", 2022-07-20T02:34:32.665 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:34:32.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:34:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:34:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:34:32.699 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:34:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:34:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:34:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:34:32.700 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:34:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:34:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:34:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:34:32.701 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 24, 2022-07-20T02:34:32.714 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:34:32.715 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:34:32.730 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:34:32.730 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:34:32.730 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:34:32.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:32.731 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:34:32.731 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:34:32.731 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:34:32.732 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:32.732 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:32.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:32.732 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:32.733 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:32.733 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:32.733 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:32.733 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:32.733 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:34:32.734 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:34:32.734 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:34:32.735 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:34:32.735 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:34:32.735 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:32.735 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:32.735 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:32.736 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:32.736 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:32.736 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:32.736 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:32.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:32.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:34:32.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:32.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:32.738 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:32.752 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:32.753 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:32.754 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:32.754 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:32.758 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:32.759 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:32.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:32.760 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:32.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:32.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:32.761 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:32.761 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:32.761 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:32.767 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:32.767 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:32.768 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:32.768 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:32.769 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:32.787 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:32.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:32.788 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:32.799 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:32.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:32.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:32.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:32.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:32.801 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:32.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:32.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:34:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:34:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:34:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:34:32.864 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:34:32.864 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:34:32.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:32.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:32.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:32.877 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:32.891 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:32.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:32.892 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:32.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:32.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:32.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:32.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:32.962 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:32.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:32.976 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:32.977 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:34:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:33.005 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:33.006 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:33.008 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:33.013 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:33.015 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:33.016 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:33.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.026 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:33.051 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:33.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:33.053 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:33.054 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:33.055 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:33.056 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:34:33.057 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:34:33.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.067 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:34:33.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:33.069 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:34:33.085 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.087 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.088 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.088 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.089 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.090 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:34:33.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:34:33.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.095 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:34:33.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:34:33.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:34:33.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.177 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:34:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:34:33.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:33.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.199 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:34:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:34:33.200 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:34:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:34:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.201 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.209 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:34:33.210 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.211 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.213 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.214 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:34:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:34:33.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:34:33.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:34:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:34:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:34:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.295 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:34:33.295 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.297 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:34:33.297 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:33.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.298 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.299 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:34:33.299 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:34:33.300 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:34:33.303 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:34:33.304 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.304 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.305 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.306 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:33.307 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:34:33.308 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.308 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.309 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.309 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.309 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.310 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.310 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.311 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 125, 2022-07-20T02:34:33.312 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.313 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.313 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:34:33.315 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.317 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:34:33.317 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.327 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:34:33.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:34:33.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.356 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:34:33.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:34:33.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.375 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:34:33.389 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:34:33.392 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:33.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:34:33.396 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:34:33.396 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:34:33.397 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:34:33.397 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.398 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.398 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.400 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:33.400 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:34:33.401 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.401 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.401 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.402 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.402 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.403 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.403 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.404 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.404 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.405 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:34:33.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:34:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:34:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:34:33.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.453 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:34:33.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:34:33.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.475 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:34:33.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:34:33.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:33.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:34:33.494 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:34:33.502 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:34:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:34:33.508 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.509 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:33.512 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:34:33.513 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.519 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.520 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 125, 2022-07-20T02:34:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:34:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:34:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:34:33.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:34:33.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:34:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:34:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:34:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:34:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:34:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:34:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:34:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:34:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:34:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:34:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:34:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:34:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:34:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:34:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:34:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:34:33.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:34:33.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:33.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:34:33.737 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:34:33.738 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:34:33.738 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:34:33.739 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.740 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.741 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:33.748 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:34:33.748 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.749 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.749 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.750 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:34:33.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:34:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.781 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:34:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:34:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.807 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:34:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:34:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.814 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:34:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:34:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:34:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:34:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:34:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:34:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:34:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:34:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:34:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:34:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:34:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:33.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:34:33.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:34:33.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.919 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:33.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:33.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:33.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:34:33.922 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:33.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:34:33.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:33.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:34:33.964 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:34:33.964 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:34:33.965 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:34:33.965 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:33.971 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:33.971 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:33.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:33.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:33.973 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:33.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:34:33.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:33.974 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:33.983 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:33.984 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:33.984 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:33.985 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:33.985 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:34:33.985 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:33.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:33.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:33.996 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:33.997 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:33.997 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:33.997 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:33.998 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:33.998 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:33.999 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:34:33.999 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:34.002 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.003 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.003 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.004 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.005 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:34:34.005 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:34.007 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.008 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.008 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.009 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.009 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.010 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:34:34.010 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.011 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:34.012 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.012 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.013 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.015 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.015 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:34:34.016 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.017 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:34.017 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:34:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:34:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:34:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:34:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:34:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:34:34.070 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:34:34.071 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:34:34.072 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/126332545": "2022-07-21T02:34:08.449272+0000", 2022-07-20T02:34:34.072 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/126332545": "2022-07-21T02:34:08.449272+0000" 2022-07-20T02:34:34.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.073 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:34:34.073 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:34:34.074 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:34:34.074 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:34.075 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:34:34.075 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:34:34.076 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:34.076 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:34:34.076 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:34.077 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:34.077 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:34:34.083 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:34:34.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.084 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:34:34.085 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:34.085 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:34.086 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:34:34.086 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:34.086 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:34.087 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:34:34.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:34.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:34.088 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:34:34.089 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:34:34.089 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:34:34.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:34:34.090 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:34:34.091 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:34:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.651+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 msgr2=0x7f6654055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.651+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 0x7f6654055fc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f6670005000 tx=0x7f66740b1d60).stop 2022-07-20T02:34:34.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.651+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 msgr2=0x7f66740b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.651+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 0x7f66740b0570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f666400d000 tx=0x7f6664007a00).stop 2022-07-20T02:34:34.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 shutdown_connections 2022-07-20T02:34:34.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6674063b10 0x7f6674147600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6654053b00 0x7f6654055fc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66740ac880 0x7f6674147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 --2- 172.21.15.26:0/4099017945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6660004e70 0x7f66740b0570 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 >> 172.21.15.26:0/4099017945 conn(0x7f66740b3050 msgr2=0x7f66740b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:34.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 shutdown_connections 2022-07-20T02:34:34.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:32.655+0000 7f667a729700 1 -- 172.21.15.26:0/4099017945 wait complete. 2022-07-20T02:34:34.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:34.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 Processor -- start 2022-07-20T02:34:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- start start 2022-07-20T02:34:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 0x7fab5c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab5c061050 0x7fab5c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5c0b1bc0 con 0x7fab5c066a80 2022-07-20T02:34:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5c14b040 con 0x7fab5c061050 2022-07-20T02:34:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5c0a8860 con 0x7fab5c0b2920 2022-07-20T02:34:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab5c061050 0x7fab5c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab5c061050 0x7fab5c148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42722/0 (socket says 172.21.15.26:42722) 2022-07-20T02:34:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 -- 172.21.15.26:0/664247785 learned_addr learned my addr 172.21.15.26:0/664247785 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab61be9700 1 --2- 172.21.15.26:0/664247785 >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 0x7fab5c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 -- 172.21.15.26:0/664247785 >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 msgr2=0x7fab5c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:34.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 --2- 172.21.15.26:0/664247785 >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 0x7fab5c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 -- 172.21.15.26:0/664247785 >> v1:172.21.15.120:6789/0 conn(0x7fab5c066a80 legacy=0x7fab5c066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:34.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 -- 172.21.15.26:0/664247785 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c147700 con 0x7fab5c061050 2022-07-20T02:34:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab61be9700 1 --2- 172.21.15.26:0/664247785 >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 0x7fab5c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab613e8700 1 --2- 172.21.15.26:0/664247785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c061050 0x7fab5c148280 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fab5c147c70 tx=0x7fab58006eb0).ready entity=mon.0 client_cookie=5e0c30016c180c00 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab60be7700 1 -- 172.21.15.26:0/664247785 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5800d5c0 con 0x7fab5c061050 2022-07-20T02:34:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab60be7700 1 -- 172.21.15.26:0/664247785 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58000e90 con 0x7fab5c061050 2022-07-20T02:34:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab60be7700 1 -- 172.21.15.26:0/664247785 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5800d910 con 0x7fab5c061050 2022-07-20T02:34:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- 172.21.15.26:0/664247785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c061050 msgr2=0x7fab5c148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 --2- 172.21.15.26:0/664247785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c061050 0x7fab5c148280 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fab5c147c70 tx=0x7fab58006eb0).stop 2022-07-20T02:34:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- 172.21.15.26:0/664247785 shutdown_connections 2022-07-20T02:34:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 --2- 172.21.15.26:0/664247785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c061050 0x7fab5c148280 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 --2- 172.21.15.26:0/664247785 >> v2:172.21.15.120:3300/0 conn(0x7fab5c0b2920 0x7fab5c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- 172.21.15.26:0/664247785 >> 172.21.15.26:0/664247785 conn(0x7fab5c0b60a0 msgr2=0x7fab5c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- 172.21.15.26:0/664247785 shutdown_connections 2022-07-20T02:34:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.371+0000 7fab63e4d700 1 -- 172.21.15.26:0/664247785 wait complete. 2022-07-20T02:34:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 Processor -- start 2022-07-20T02:34:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- start start 2022-07-20T02:34:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 0x7fab5c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab5c0b2920 0x7fab5c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0ab5a0 con 0x7fab5c066a80 2022-07-20T02:34:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0632c0 con 0x7fab5c061050 2022-07-20T02:34:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c063560 con 0x7fab5c0b2920 2022-07-20T02:34:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42728/0 (socket says 172.21.15.26:42728) 2022-07-20T02:34:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 -- 172.21.15.26:0/1818181727 learned_addr learned my addr 172.21.15.26:0/1818181727 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab61be9700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 0x7fab5c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab623ea700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab5c0b2920 0x7fab5c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 -- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab5c0b2920 msgr2=0x7fab5c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab5c0b2920 0x7fab5c1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 -- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 msgr2=0x7fab5c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 0x7fab5c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 -- 172.21.15.26:0/1818181727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5800b1e0 con 0x7fab5c066a80 2022-07-20T02:34:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab61be9700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 0x7fab5c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab613e8700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fab5c147c70 tx=0x7fab58015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab580068b0 con 0x7fab5c066a80 2022-07-20T02:34:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5800d940 con 0x7fab5c066a80 2022-07-20T02:34:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5c0aac10 con 0x7fab5c066a80 2022-07-20T02:34:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab58011e20 con 0x7fab5c066a80 2022-07-20T02:34:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.375+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5c0641b0 con 0x7fab5c066a80 2022-07-20T02:34:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5c063720 con 0x7fab5c066a80 2022-07-20T02:34:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fab5801a040 con 0x7fab5c066a80 2022-07-20T02:34:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab4affd700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 0x7fab3c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fab5806cd20 con 0x7fab5c066a80 2022-07-20T02:34:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab61be9700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 0x7fab3c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.379+0000 7fab61be9700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 0x7fab3c055e10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fab4c00b000 tx=0x7fab4c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.387+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab58043dc0 con 0x7fab5c066a80 2022-07-20T02:34:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.547+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab5c14ada0 con 0x7fab5c066a80 2022-07-20T02:34:34.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.547+0000 7fab4affd700 1 -- 172.21.15.26:0/1818181727 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+504 (secure 0 0 0) 0x7fab58023020 con 0x7fab5c066a80 2022-07-20T02:34:34.553 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:34.553 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-20T02:34:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 msgr2=0x7fab3c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 0x7fab3c055e10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fab4c00b000 tx=0x7fab4c008040).stop 2022-07-20T02:34:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 msgr2=0x7fab5c0ab0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fab5c147c70 tx=0x7fab58015000).stop 2022-07-20T02:34:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 shutdown_connections 2022-07-20T02:34:34.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab5c066a80 0x7fab5c0ab0b0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab3c053950 0x7fab3c055e10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab5c0b2920 0x7fab5c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 --2- 172.21.15.26:0/1818181727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab5c061050 0x7fab5c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 >> 172.21.15.26:0/1818181727 conn(0x7fab5c0b60a0 msgr2=0x7fab5c149650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 shutdown_connections 2022-07-20T02:34:34.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.551+0000 7fab63e4d700 1 -- 172.21.15.26:0/1818181727 wait complete. 2022-07-20T02:34:34.572 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 68 2022-07-20T02:34:34.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:34.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 Processor -- start 2022-07-20T02:34:34.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 -- start start 2022-07-20T02:34:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd708060f00 0x7fd708143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7081474e0 con 0x7fd708147000 2022-07-20T02:34:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7080b4e40 con 0x7fd70805cfd0 2022-07-20T02:34:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70f2ba700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70805d3b0 con 0x7fd708060f00 2022-07-20T02:34:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70dab7700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd708147000 0x7fd70813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42016/0 (socket says 172.21.15.26:42016) 2022-07-20T02:34:34.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70dab7700 1 -- 172.21.15.26:0/2267960265 learned_addr learned my addr 172.21.15.26:0/2267960265 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70eab9700 1 --2- 172.21.15.26:0/2267960265 >> v2:172.21.15.120:3300/0 conn(0x7fd708060f00 0x7fd708143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186683735 0 0) 0x7fd7081474e0 con 0x7fd708147000 2022-07-20T02:34:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f0003260 con 0x7fd708147000 2022-07-20T02:34:34.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570053786 0 0) 0x7fd6f0003260 con 0x7fd708147000 2022-07-20T02:34:34.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 >> v2:172.21.15.120:3300/0 conn(0x7fd708060f00 msgr2=0x7fd708143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:34.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 --2- 172.21.15.26:0/2267960265 >> v2:172.21.15.120:3300/0 conn(0x7fd708060f00 0x7fd708143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 >> v1:172.21.15.120:6789/0 conn(0x7fd70805cfd0 legacy=0x7fd7080b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7081431e0 con 0x7fd708147000 2022-07-20T02:34:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fd704001660 con 0x7fd708147000 2022-07-20T02:34:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 >> v1:172.21.15.26:6789/0 conn(0x7fd708147000 legacy=0x7fd70813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 0x7fd6f0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0004500 0x7fd6f0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f0004e70 0x7fd6f000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70805d3b0 con 0x7fd6f0003750 2022-07-20T02:34:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f000bd20 con 0x7fd6f0004500 2022-07-20T02:34:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f000bee0 con 0x7fd6f0004e70 2022-07-20T02:34:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.695+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 0x7fd6f0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70dab7700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0004500 0x7fd6f0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70eab9700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f0004e70 0x7fd6f000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 -- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f0004e70 msgr2=0x7fd6f000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f0004e70 0x7fd6f000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 -- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0004500 msgr2=0x7fd6f0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0004500 0x7fd6f0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 -- 172.21.15.26:0/2267960265 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f0003260 con 0x7fd6f0003750 2022-07-20T02:34:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70eab9700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f0004e70 0x7fd6f000b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70eab9700 1 -- 172.21.15.26:0/2267960265 reap_dead start 2022-07-20T02:34:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 0x7fd6f0003b70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7fd708068700 tx=0x7fd6fc007a00).ready entity=mon.0 client_cookie=3f5f1c27fceaf70 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc007d10 con 0x7fd6f0003750 2022-07-20T02:34:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc007030 con 0x7fd6f0003750 2022-07-20T02:34:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70d2b6700 1 -- 172.21.15.26:0/2267960265 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc007390 con 0x7fd6f0003750 2022-07-20T02:34:34.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 msgr2=0x7fd6f0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 0x7fd6f0003b70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7fd708068700 tx=0x7fd6fc007a00).stop 2022-07-20T02:34:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- 172.21.15.26:0/2267960265 shutdown_connections 2022-07-20T02:34:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/2267960265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd6f0003750 0x7fd6f0003b70 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- 172.21.15.26:0/2267960265 >> 172.21.15.26:0/2267960265 conn(0x7fd7080620c0 msgr2=0x7fd70805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- 172.21.15.26:0/2267960265 shutdown_connections 2022-07-20T02:34:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- 172.21.15.26:0/2267960265 wait complete. 2022-07-20T02:34:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 Processor -- start 2022-07-20T02:34:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.699+0000 7fd70f2ba700 1 -- start start 2022-07-20T02:34:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 0x7fd708152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7080b7a10 0x7fd708152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd708142ac0 con 0x7fd708152ef0 2022-07-20T02:34:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080b8a10 con 0x7fd7080b8630 2022-07-20T02:34:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080b9370 con 0x7fd7080b7a10 2022-07-20T02:34:34.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70e2b8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 0x7fd708152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42750/0 (socket says 172.21.15.26:42750) 2022-07-20T02:34:34.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 -- 172.21.15.26:0/248024498 learned_addr learned my addr 172.21.15.26:0/248024498 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:34.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70e2b8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 0x7fd708152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:34206/0 (socket says 172.21.15.26:34206) 2022-07-20T02:34:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70dab7700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7080b7a10 0x7fd708152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 -- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7080b7a10 msgr2=0x7fd708152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7080b7a10 0x7fd708152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 -- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 msgr2=0x7fd708152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 0x7fd708152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 -- 172.21.15.26:0/248024498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd708145f80 con 0x7fd708152ef0 2022-07-20T02:34:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70eab9700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7fd70000b000 tx=0x7fd700008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd700018070 con 0x7fd708152ef0 2022-07-20T02:34:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7000127a0 con 0x7fd708152ef0 2022-07-20T02:34:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd708146220 con 0x7fd708152ef0 2022-07-20T02:34:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd700012b00 con 0x7fd708152ef0 2022-07-20T02:34:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.703+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7080b7680 con 0x7fd708152ef0 2022-07-20T02:34:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd708143480 con 0x7fd708152ef0 2022-07-20T02:34:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd700012040 con 0x7fd708152ef0 2022-07-20T02:34:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd6fb7fe700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 0x7fd6e8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd70006be00 con 0x7fd708152ef0 2022-07-20T02:34:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 0x7fd6e8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.707+0000 7fd70e2b8700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 0x7fd6e8055ff0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fd6fc01b000 tx=0x7fd6fc015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.711+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd700042e00 con 0x7fd708152ef0 2022-07-20T02:34:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.875+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7080574e0 con 0x7fd708152ef0 2022-07-20T02:34:34.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.875+0000 7fd6fb7fe700 1 -- 172.21.15.26:0/248024498 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+504 (secure 0 0 0) 0x7fd700046420 con 0x7fd708152ef0 2022-07-20T02:34:34.887 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:34.887 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-20T02:34:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 msgr2=0x7fd6e8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 0x7fd6e8055ff0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fd6fc01b000 tx=0x7fd6fc015000).stop 2022-07-20T02:34:34.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 msgr2=0x7fd708155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7fd70000b000 tx=0x7fd700008040).stop 2022-07-20T02:34:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 shutdown_connections 2022-07-20T02:34:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd708152ef0 0x7fd708155860 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd6e8053b30 0x7fd6e8055ff0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7080b7a10 0x7fd708152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 --2- 172.21.15.26:0/248024498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7080b8630 0x7fd708152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 >> 172.21.15.26:0/248024498 conn(0x7fd6f0003750 msgr2=0x7fd7080b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 shutdown_connections 2022-07-20T02:34:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:34.879+0000 7fd70f2ba700 1 -- 172.21.15.26:0/248024498 wait complete. 2022-07-20T02:34:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 68 2022-07-20T02:34:34.937 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:34:34.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:34:34.968 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:34:34.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:34:34.969 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:34:34.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:34:34.972 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:34:34.973 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:34.973 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:34:34.974 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:35.007 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:34:35.007 DEBUG:teuthology.parallel:result is None 2022-07-20T02:34:35.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:34:35.055 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:34:35.056 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:34:35.065 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:34:35.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 Processor -- start 2022-07-20T02:34:35.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 -- start start 2022-07-20T02:34:35.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1850064450 0x7f1850064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1850064880 con 0x7f18500670c0 2022-07-20T02:34:35.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18500b35a0 con 0x7f18501444a0 2022-07-20T02:34:35.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184ffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18500b85f0 con 0x7f1850064450 2022-07-20T02:34:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184effd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f18500670c0 0x7f18500b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42058/0 (socket says 172.21.15.26:42058) 2022-07-20T02:34:35.128 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184effd700 1 -- 172.21.15.26:0/916235529 learned_addr learned my addr 172.21.15.26:0/916235529 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:35.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184e7fc700 1 --2- 172.21.15.26:0/916235529 >> v2:172.21.15.120:3300/0 conn(0x7f1850064450 0x7f1850064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.129 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169283138 0 0) 0x7f1850064880 con 0x7f18500670c0 2022-07-20T02:34:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834003260 con 0x7f18500670c0 2022-07-20T02:34:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267535393 0 0) 0x7f1834003260 con 0x7f18500670c0 2022-07-20T02:34:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 >> v2:172.21.15.120:3300/0 conn(0x7f1850064450 msgr2=0x7f1850064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 --2- 172.21.15.26:0/916235529 >> v2:172.21.15.120:3300/0 conn(0x7f1850064450 0x7f1850064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 >> v1:172.21.15.120:6789/0 conn(0x7f18501444a0 legacy=0x7f185014b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500b80b0 con 0x7f18500670c0 2022-07-20T02:34:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f18480019e0 con 0x7f18500670c0 2022-07-20T02:34:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 >> v1:172.21.15.26:6789/0 conn(0x7f18500670c0 legacy=0x7f18500b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 0x7f1834003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1834004500 0x7f1834004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834004e70 0x7f183400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184effd700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 0x7f1834003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.174 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183400bcf0 con 0x7f1834003750 2022-07-20T02:34:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183400be60 con 0x7f1834004e70 2022-07-20T02:34:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183400c020 con 0x7f1834004500 2022-07-20T02:34:35.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184e7fc700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1834004500 0x7f1834004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.119+0000 7f184f7fe700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834004e70 0x7f183400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 -- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1834004500 msgr2=0x7f1834004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:35.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1834004500 0x7f1834004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 -- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834004e70 msgr2=0x7f183400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834004e70 0x7f183400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 -- 172.21.15.26:0/916235529 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834003260 con 0x7f1834003750 2022-07-20T02:34:35.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184f7fe700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834004e70 0x7f183400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:35.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184f7fe700 1 -- 172.21.15.26:0/916235529 reap_dead start 2022-07-20T02:34:35.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184effd700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 0x7f1834003b70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f18480016f0 tx=0x7f1850140ed0).ready entity=mon.0 client_cookie=83fd62fe50e8ea5a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f184800cca0 con 0x7f1834003750 2022-07-20T02:34:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184800ce00 con 0x7f1834003750 2022-07-20T02:34:35.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184dffb700 1 -- 172.21.15.26:0/916235529 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1848005b80 con 0x7f1834003750 2022-07-20T02:34:35.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 msgr2=0x7f1834003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:35.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 0x7f1834003b70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f18480016f0 tx=0x7f1850140ed0).stop 2022-07-20T02:34:35.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- 172.21.15.26:0/916235529 shutdown_connections 2022-07-20T02:34:35.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 --2- 172.21.15.26:0/916235529 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1834003750 0x7f1834003b70 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- 172.21.15.26:0/916235529 >> 172.21.15.26:0/916235529 conn(0x7f185005f840 msgr2=0x7f185005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:35.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- 172.21.15.26:0/916235529 shutdown_connections 2022-07-20T02:34:35.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- 172.21.15.26:0/916235529 wait complete. 2022-07-20T02:34:35.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 Processor -- start 2022-07-20T02:34:35.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- start start 2022-07-20T02:34:35.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834003750 0x7f18500ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18500ad9e0 0x7f1850152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18500abf60 con 0x7f18500ae180 2022-07-20T02:34:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1850152250 con 0x7f1834003750 2022-07-20T02:34:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.123+0000 7f184ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1850060c90 con 0x7f18500ad9e0 2022-07-20T02:34:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42794/0 (socket says 172.21.15.26:42794) 2022-07-20T02:34:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 -- 172.21.15.26:0/3260110387 learned_addr learned my addr 172.21.15.26:0/3260110387 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184f7fe700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18500ad9e0 0x7f1850152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184effd700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834003750 0x7f18500ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 -- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18500ad9e0 msgr2=0x7f1850152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18500ad9e0 0x7f1850152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 -- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834003750 msgr2=0x7f18500ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834003750 0x7f18500ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 -- 172.21.15.26:0/3260110387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500609f0 con 0x7f18500ae180 2022-07-20T02:34:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184e7fc700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f184400b000 tx=0x7f1844008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1844014680 con 0x7f18500ae180 2022-07-20T02:34:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440147e0 con 0x7f18500ae180 2022-07-20T02:34:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18501524b0 con 0x7f18500ae180 2022-07-20T02:34:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1844014b40 con 0x7f18500ae180 2022-07-20T02:34:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1850152e40 con 0x7f18500ae180 2022-07-20T02:34:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.127+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18500b8350 con 0x7f18500ae180 2022-07-20T02:34:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.131+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1844024070 con 0x7f18500ae180 2022-07-20T02:34:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.131+0000 7f183ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 0x7f182c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.131+0000 7f184effd700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 0x7f182c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.131+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f184406cba0 con 0x7f18500ae180 2022-07-20T02:34:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.131+0000 7f184effd700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 0x7f182c055fc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f1848019000 tx=0x7f1848015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:35.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.135+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1844043b30 con 0x7f18500ae180 2022-07-20T02:34:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:35.271+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f18500574e0 con 0x7f18500ae180 2022-07-20T02:34:36.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.091+0000 7f183ffff700 1 -- 172.21.15.26:0/3260110387 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7f184402b080 con 0x7f18500ae180 2022-07-20T02:34:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 msgr2=0x7f182c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 0x7f182c055fc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f1848019000 tx=0x7f1848015000).stop 2022-07-20T02:34:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 msgr2=0x7f1850061360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f184400b000 tx=0x7f1844008040).stop 2022-07-20T02:34:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 shutdown_connections 2022-07-20T02:34:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18500ae180 0x7f1850061360 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f182c053b00 0x7f182c055fc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18500ad9e0 0x7f1850152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 --2- 172.21.15.26:0/3260110387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1834003750 0x7f18500ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 >> 172.21.15.26:0/3260110387 conn(0x7f185005f840 msgr2=0x7f1850143030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 shutdown_connections 2022-07-20T02:34:36.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.095+0000 7f184ffff700 1 -- 172.21.15.26:0/3260110387 wait complete. 2022-07-20T02:34:36.134 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:34:36.135 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:34:36.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:34:36.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 Processor -- start 2022-07-20T02:34:36.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- start start 2022-07-20T02:34:36.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc1881484a0 0x7fc188066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc188061050 0x7fc188066f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1880b1bc0 con 0x7fc1880b2920 2022-07-20T02:34:36.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc18814b040 con 0x7fc1881484a0 2022-07-20T02:34:36.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1880a8860 con 0x7fc188061050 2022-07-20T02:34:36.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc1881484a0 0x7fc188066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc1881484a0 0x7fc188066a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42800/0 (socket says 172.21.15.26:42800) 2022-07-20T02:34:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d9d6700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc1880b2920 0x7fc188147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42080/0 (socket says 172.21.15.26:42080) 2022-07-20T02:34:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 -- 172.21.15.26:0/401293293 learned_addr learned my addr 172.21.15.26:0/401293293 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18e1d7700 1 --2- 172.21.15.26:0/401293293 >> v2:172.21.15.120:3300/0 conn(0x7fc188061050 0x7fc188066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 -- 172.21.15.26:0/401293293 >> v2:172.21.15.120:3300/0 conn(0x7fc188061050 msgr2=0x7fc188066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:36.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 --2- 172.21.15.26:0/401293293 >> v2:172.21.15.120:3300/0 conn(0x7fc188061050 0x7fc188066f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 -- 172.21.15.26:0/401293293 >> v1:172.21.15.26:6789/0 conn(0x7fc1880b2920 legacy=0x7fc188147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 -- 172.21.15.26:0/401293293 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc188147700 con 0x7fc1881484a0 2022-07-20T02:34:36.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18d1d5700 1 --2- 172.21.15.26:0/401293293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1881484a0 0x7fc188066a80 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fc184002f70 tx=0x7fc184006eb0).ready entity=mon.0 client_cookie=c3c074451124443a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18c9d4700 1 -- 172.21.15.26:0/401293293 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18400d610 con 0x7fc1881484a0 2022-07-20T02:34:36.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18c9d4700 1 -- 172.21.15.26:0/401293293 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc184000e90 con 0x7fc1881484a0 2022-07-20T02:34:36.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18c9d4700 1 -- 172.21.15.26:0/401293293 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18400d9a0 con 0x7fc1881484a0 2022-07-20T02:34:36.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- 172.21.15.26:0/401293293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1881484a0 msgr2=0x7fc188066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/401293293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1881484a0 0x7fc188066a80 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fc184002f70 tx=0x7fc184006eb0).stop 2022-07-20T02:34:36.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- 172.21.15.26:0/401293293 shutdown_connections 2022-07-20T02:34:36.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/401293293 >> v2:172.21.15.120:3300/0 conn(0x7fc188061050 0x7fc188066f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/401293293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1881484a0 0x7fc188066a80 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- 172.21.15.26:0/401293293 >> 172.21.15.26:0/401293293 conn(0x7fc1880b60a0 msgr2=0x7fc1880b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:36.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- 172.21.15.26:0/401293293 shutdown_connections 2022-07-20T02:34:36.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.283+0000 7fc18fc3a700 1 -- 172.21.15.26:0/401293293 wait complete. 2022-07-20T02:34:36.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.287+0000 7fc18fc3a700 1 Processor -- start 2022-07-20T02:34:36.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.287+0000 7fc18fc3a700 1 -- start start 2022-07-20T02:34:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.287+0000 7fc18fc3a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 0x7fc188062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 0x7fc188142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188065ae0 con 0x7fc1880b2920 2022-07-20T02:34:36.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188065cb0 con 0x7fc188061050 2022-07-20T02:34:36.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc188065f50 con 0x7fc1881484a0 2022-07-20T02:34:36.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42808/0 (socket says 172.21.15.26:42808) 2022-07-20T02:34:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 -- 172.21.15.26:0/277243760 learned_addr learned my addr 172.21.15.26:0/277243760 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d9d6700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 0x7fc188062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18e1d7700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 0x7fc188142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 -- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 msgr2=0x7fc188142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 0x7fc188142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 -- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 msgr2=0x7fc188062b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 0x7fc188062b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 -- 172.21.15.26:0/277243760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18400b1e0 con 0x7fc1880b2920 2022-07-20T02:34:36.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18e1d7700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 0x7fc188142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:36.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d9d6700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 0x7fc188062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:36.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d1d5700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fc184002f70 tx=0x7fc184015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc184006b40 con 0x7fc1880b2920 2022-07-20T02:34:36.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1840068b0 con 0x7fc1880b2920 2022-07-20T02:34:36.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1880661f0 con 0x7fc1880b2920 2022-07-20T02:34:36.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18400d970 con 0x7fc1880b2920 2022-07-20T02:34:36.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc188065640 con 0x7fc1880b2920 2022-07-20T02:34:36.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc188065800 con 0x7fc1880b2920 2022-07-20T02:34:36.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc184025030 con 0x7fc1880b2920 2022-07-20T02:34:36.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 0x7fc168055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fc18406cfe0 con 0x7fc1880b2920 2022-07-20T02:34:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.291+0000 7fc18d9d6700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 0x7fc168055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.295+0000 7fc18d9d6700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 0x7fc168055ea0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fc17800b000 tx=0x7fc1780070e0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.295+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc184043fd0 con 0x7fc1880b2920 2022-07-20T02:34:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.463+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc18814ada0 con 0x7fc1880b2920 2022-07-20T02:34:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.463+0000 7fc176ffd700 1 -- 172.21.15.26:0/277243760 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fc1840475f0 con 0x7fc1880b2920 2022-07-20T02:34:36.470 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:36.471 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:36.471 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:34:36.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.473 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:34:36.473 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:34:36.474 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.475 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:34:36.475 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.476 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.476 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.516 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:34:36.516 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:34:36.518 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.518 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:34:36.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.520 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.564 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:34:36.565 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:34:36.565 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:34:36.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.567 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.569 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:34:36.569 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:34:36.570 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.574 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.575 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.577 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:34:36.577 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:34:36.578 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.578 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.579 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.588 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.589 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:34:36.590 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:34:36.590 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.590 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.590 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.591 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.592 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:34:36.592 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:34:36.592 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.592 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.592 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.596 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.597 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:34:36.597 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:34:36.597 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.598 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.598 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.601 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:34:36.601 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:34:36.601 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.602 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.602 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.602 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.603 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:34:36.603 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:34:36.604 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.607 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.608 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:34:36.608 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:34:36.608 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.609 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.609 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.620 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:34:36.630 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:34:36.631 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.631 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:34:36.631 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:34:36.632 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:34:36.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.634 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:34:36.634 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:34:36.635 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.635 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:34:36.636 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:34:36.636 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:34:36.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.638 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:34:36.638 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:34:36.639 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.639 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:34:36.639 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:34:36.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:34:36.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.642 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:34:36.655 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:34:36.656 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.656 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:34:36.657 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:34:36.657 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:36.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:36.660 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:34:36.660 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:34:36.661 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:34:36.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:34:36.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:34:36.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:34:36.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:36.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:36.664 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:34:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 msgr2=0x7fc168055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 0x7fc168055ea0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fc17800b000 tx=0x7fc1780070e0).stop 2022-07-20T02:34:36.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 msgr2=0x7fc1881427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fc184002f70 tx=0x7fc184015000).stop 2022-07-20T02:34:36.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 shutdown_connections 2022-07-20T02:34:36.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1880b2920 0x7fc1881427d0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc1680539e0 0x7fc168055ea0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1881484a0 0x7fc188142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 --2- 172.21.15.26:0/277243760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188061050 0x7fc188062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 >> 172.21.15.26:0/277243760 conn(0x7fc1880b60a0 msgr2=0x7fc1880b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:36.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 shutdown_connections 2022-07-20T02:34:36.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.467+0000 7fc18fc3a700 1 -- 172.21.15.26:0/277243760 wait complete. 2022-07-20T02:34:36.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 Processor -- start 2022-07-20T02:34:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 -- start start 2022-07-20T02:34:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f801c0a7fc0 0x7f801c1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0b62d0 con 0x7f801c0a8c40 2022-07-20T02:34:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c147eb0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.799+0000 7f8022d2a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c14aac0 con 0x7f801c0b5ef0 2022-07-20T02:34:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f801c0a7fc0 0x7f801c1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f801c0a7fc0 0x7f801c1479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42816/0 (socket says 172.21.15.26:42816) 2022-07-20T02:34:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 -- 172.21.15.26:0/2635590075 learned_addr learned my addr 172.21.15.26:0/2635590075 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021d28700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 -- 172.21.15.26:0/2635590075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 msgr2=0x7f801c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c1474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 -- 172.21.15.26:0/2635590075 >> v1:172.21.15.26:6789/0 conn(0x7f801c0a8c40 legacy=0x7f801c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 -- 172.21.15.26:0/2635590075 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c14ad60 con 0x7f801c0a7fc0 2022-07-20T02:34:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021d28700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c1474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8021527700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c1479c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f80180032f0 tx=0x7f8018009290).ready entity=mon.0 client_cookie=4563578d6485c387 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8020d26700 1 -- 172.21.15.26:0/2635590075 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f801800b8a0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8020d26700 1 -- 172.21.15.26:0/2635590075 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801800ba00 con 0x7f801c0a7fc0 2022-07-20T02:34:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8020d26700 1 -- 172.21.15.26:0/2635590075 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f801800bb70 con 0x7f801c0a7fc0 2022-07-20T02:34:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- 172.21.15.26:0/2635590075 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 msgr2=0x7f801c1479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c1479c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f80180032f0 tx=0x7f8018009290).stop 2022-07-20T02:34:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- 172.21.15.26:0/2635590075 shutdown_connections 2022-07-20T02:34:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c1479c0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 --2- 172.21.15.26:0/2635590075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c1474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- 172.21.15.26:0/2635590075 >> 172.21.15.26:0/2635590075 conn(0x7f801c060310 msgr2=0x7f801c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- 172.21.15.26:0/2635590075 shutdown_connections 2022-07-20T02:34:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- 172.21.15.26:0/2635590075 wait complete. 2022-07-20T02:34:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 Processor -- start 2022-07-20T02:34:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.803+0000 7f8022d2a700 1 -- start start 2022-07-20T02:34:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a8c40 0x7f801c149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c067ae0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c067180 con 0x7f801c0a8c40 2022-07-20T02:34:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c067420 con 0x7f801c0b5ef0 2022-07-20T02:34:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42822/0 (socket says 172.21.15.26:42822) 2022-07-20T02:34:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 -- 172.21.15.26:0/25938292 learned_addr learned my addr 172.21.15.26:0/25938292 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022529700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021527700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a8c40 0x7f801c149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 -- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 msgr2=0x7f801c156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 -- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a8c40 msgr2=0x7f801c149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a8c40 0x7f801c149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 -- 172.21.15.26:0/25938292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018009530 con 0x7f801c0a7fc0 2022-07-20T02:34:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8021d28700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f800c007000 tx=0x7f800c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f800c00ce80 con 0x7f801c0a7fc0 2022-07-20T02:34:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c010580 con 0x7f801c0a7fc0 2022-07-20T02:34:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c0676c0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f800c0108e0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.807+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c0626f0 con 0x7f801c0a7fc0 2022-07-20T02:34:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c066f40 con 0x7f801c0a7fc0 2022-07-20T02:34:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f800c010a40 con 0x7f801c0a7fc0 2022-07-20T02:34:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f800affd700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 0x7f7ffc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f800c06ba50 con 0x7f801c0a7fc0 2022-07-20T02:34:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f8021527700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 0x7f7ffc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.811+0000 7f8021527700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 0x7f7ffc055ea0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f801801b000 tx=0x7f8018016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.815+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f801c066f40 con 0x7f801c0a7fc0 2022-07-20T02:34:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.943+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f801c066f40 con 0x7f801c0a7fc0 2022-07-20T02:34:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f800affd700 1 -- 172.21.15.26:0/25938292 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+13472 (secure 0 0 0) 0x7f801c066f40 con 0x7f801c0a7fc0 2022-07-20T02:34:36.954 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:36.955 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":135,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:36.081949+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:36.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 msgr2=0x7f7ffc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 0x7f7ffc055ea0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f801801b000 tx=0x7f8018016000).stop 2022-07-20T02:34:36.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 msgr2=0x7f801c063c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:36.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f800c007000 tx=0x7f800c008040).stop 2022-07-20T02:34:36.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 shutdown_connections 2022-07-20T02:34:36.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7ffc0539e0 0x7f7ffc055ea0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c0b5ef0 0x7f801c156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a8c40 0x7f801c149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 --2- 172.21.15.26:0/25938292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f801c0a7fc0 0x7f801c063c10 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 >> 172.21.15.26:0/25938292 conn(0x7f801c060310 msgr2=0x7f801c0af380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 shutdown_connections 2022-07-20T02:34:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:36.947+0000 7f8022d2a700 1 -- 172.21.15.26:0/25938292 wait complete. 2022-07-20T02:34:36.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:34:36.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:34:37.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 Processor -- start 2022-07-20T02:34:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 -- start start 2022-07-20T02:34:37.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7304147000 0x7f7304143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73041474e0 con 0x7f730405cfd0 2022-07-20T02:34:37.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73040b4e40 con 0x7f7304060f00 2022-07-20T02:34:37.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.119+0000 7f7309a76700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730405d3b0 con 0x7f7304147000 2022-07-20T02:34:37.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f7304147000 0x7f7304143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7308a74700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f730405cfd0 0x7f73040b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42110/0 (socket says 172.21.15.26:42110) 2022-07-20T02:34:37.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7308a74700 1 -- 172.21.15.26:0/2522672897 learned_addr learned my addr 172.21.15.26:0/2522672897 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:37.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 -- 172.21.15.26:0/2522672897 >> v1:172.21.15.120:6789/0 conn(0x7f7304060f00 legacy=0x7f730413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 -- 172.21.15.26:0/2522672897 >> v1:172.21.15.26:6789/0 conn(0x7f730405cfd0 legacy=0x7f73040b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 -- 172.21.15.26:0/2522672897 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73041431e0 con 0x7f7304147000 2022-07-20T02:34:37.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 --2- 172.21.15.26:0/2522672897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304143bc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f72f4002f70 tx=0x7f72f4006df0).ready entity=mon.0 client_cookie=aabf6183ca3b36d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f73037fe700 1 -- 172.21.15.26:0/2522672897 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f400d610 con 0x7f7304147000 2022-07-20T02:34:37.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f73037fe700 1 -- 172.21.15.26:0/2522672897 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f4000e90 con 0x7f7304147000 2022-07-20T02:34:37.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f73037fe700 1 -- 172.21.15.26:0/2522672897 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f400d960 con 0x7f7304147000 2022-07-20T02:34:37.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- 172.21.15.26:0/2522672897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 msgr2=0x7f7304143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 --2- 172.21.15.26:0/2522672897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304143bc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f72f4002f70 tx=0x7f72f4006df0).stop 2022-07-20T02:34:37.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- 172.21.15.26:0/2522672897 shutdown_connections 2022-07-20T02:34:37.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 --2- 172.21.15.26:0/2522672897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304143bc0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- 172.21.15.26:0/2522672897 >> 172.21.15.26:0/2522672897 conn(0x7f73040620c0 msgr2=0x7f730405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- 172.21.15.26:0/2522672897 shutdown_connections 2022-07-20T02:34:37.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- 172.21.15.26:0/2522672897 wait complete. 2022-07-20T02:34:37.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 Processor -- start 2022-07-20T02:34:37.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- start start 2022-07-20T02:34:37.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7304060f00 0x7f73040afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7304140a40 0x7f7304137600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040af790 con 0x7f7304147000 2022-07-20T02:34:37.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040af2d0 con 0x7f7304060f00 2022-07-20T02:34:37.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309a76700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73040af4f0 con 0x7f7304140a40 2022-07-20T02:34:37.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42838/0 (socket says 172.21.15.26:42838) 2022-07-20T02:34:37.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7303fff700 1 -- 172.21.15.26:0/2939484421 learned_addr learned my addr 172.21.15.26:0/2939484421 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:37.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7308a74700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7304060f00 0x7f73040afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.123+0000 7f7309275700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7304140a40 0x7f7304137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 -- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7304140a40 msgr2=0x7f7304137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7304140a40 0x7f7304137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 -- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7304060f00 msgr2=0x7f73040afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7304060f00 0x7f73040afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 -- 172.21.15.26:0/2939484421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f400b180 con 0x7f7304147000 2022-07-20T02:34:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7303fff700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7f72f4002f70 tx=0x7f72f4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f400d990 con 0x7f7304147000 2022-07-20T02:34:37.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f4011ca0 con 0x7f7304147000 2022-07-20T02:34:37.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f401e460 con 0x7f7304147000 2022-07-20T02:34:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7304142190 con 0x7f7304147000 2022-07-20T02:34:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73041426d0 con 0x7f7304147000 2022-07-20T02:34:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f72f4025030 con 0x7f7304147000 2022-07-20T02:34:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7304142890 con 0x7f7304147000 2022-07-20T02:34:37.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 0x7f72e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f72f406d150 con 0x7f7304147000 2022-07-20T02:34:37.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7308a74700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 0x7f72e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.127+0000 7f7308a74700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 0x7f72e8055f30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f72ec00b000 tx=0x7f72ec009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.135+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72f4044250 con 0x7f7304147000 2022-07-20T02:34:37.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.263+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7304143480 con 0x7f7304147000 2022-07-20T02:34:37.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.263+0000 7f7301ffb700 1 -- 172.21.15.26:0/2939484421 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v135) v1 ==== 110+0+10 (secure 0 0 0) 0x7f72f4047870 con 0x7f7304147000 2022-07-20T02:34:37.268 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:34:37.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 msgr2=0x7f72e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 0x7f72e8055f30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f72ec00b000 tx=0x7f72ec009040).stop 2022-07-20T02:34:37.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 msgr2=0x7f7304137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7f72f4002f70 tx=0x7f72f4015000).stop 2022-07-20T02:34:37.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 shutdown_connections 2022-07-20T02:34:37.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7304147000 0x7f7304137110 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f72e8053a70 0x7f72e8055f30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7304140a40 0x7f7304137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 --2- 172.21.15.26:0/2939484421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7304060f00 0x7f73040afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 >> 172.21.15.26:0/2939484421 conn(0x7f730405cfd0 msgr2=0x7f73041385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 shutdown_connections 2022-07-20T02:34:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.267+0000 7f7309a76700 1 -- 172.21.15.26:0/2939484421 wait complete. 2022-07-20T02:34:37.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:34:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 Processor -- start 2022-07-20T02:34:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 -- start start 2022-07-20T02:34:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0618a0 0x7f17ec144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17ec14ac80 0x7f17ec144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec061c80 con 0x7f17ec0b4240 2022-07-20T02:34:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec0b1ab0 con 0x7f17ec14ac80 2022-07-20T02:34:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f2b53700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec14b190 con 0x7f17ec0618a0 2022-07-20T02:34:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17ec14ac80 0x7f17ec144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17ec14ac80 0x7f17ec144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42844/0 (socket says 172.21.15.26:42844) 2022-07-20T02:34:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 -- 172.21.15.26:0/2826848890 learned_addr learned my addr 172.21.15.26:0/2826848890 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1b51700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0618a0 0x7f17ec144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 -- 172.21.15.26:0/2826848890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0618a0 msgr2=0x7f17ec144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0618a0 0x7f17ec144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 -- 172.21.15.26:0/2826848890 >> v1:172.21.15.120:6789/0 conn(0x7f17ec0b4240 legacy=0x7f17ec147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 -- 172.21.15.26:0/2826848890 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec14b430 con 0x7f17ec14ac80 2022-07-20T02:34:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f1350700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec144d40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f17e8002f70 tx=0x7f17e8006eb0).ready entity=mon.0 client_cookie=816bf039a5232d6b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f0b4f700 1 -- 172.21.15.26:0/2826848890 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17e800d5c0 con 0x7f17ec14ac80 2022-07-20T02:34:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f0b4f700 1 -- 172.21.15.26:0/2826848890 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e8000e90 con 0x7f17ec14ac80 2022-07-20T02:34:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.439+0000 7f17f0b4f700 1 -- 172.21.15.26:0/2826848890 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17e800d950 con 0x7f17ec14ac80 2022-07-20T02:34:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- 172.21.15.26:0/2826848890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 msgr2=0x7f17ec144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec144d40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f17e8002f70 tx=0x7f17e8006eb0).stop 2022-07-20T02:34:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- 172.21.15.26:0/2826848890 shutdown_connections 2022-07-20T02:34:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec144d40 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2826848890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0618a0 0x7f17ec144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- 172.21.15.26:0/2826848890 >> 172.21.15.26:0/2826848890 conn(0x7f17ec066f70 msgr2=0x7f17ec067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- 172.21.15.26:0/2826848890 shutdown_connections 2022-07-20T02:34:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- 172.21.15.26:0/2826848890 wait complete. 2022-07-20T02:34:37.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 Processor -- start 2022-07-20T02:34:37.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- start start 2022-07-20T02:34:37.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17ec0618a0 0x7f17ec15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0b4240 0x7f17ec14d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec15afe0 con 0x7f17ec14ac80 2022-07-20T02:34:37.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec13de80 con 0x7f17ec0618a0 2022-07-20T02:34:37.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.443+0000 7f17f2b53700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec13e120 con 0x7f17ec0b4240 2022-07-20T02:34:37.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42852/0 (socket says 172.21.15.26:42852) 2022-07-20T02:34:37.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 -- 172.21.15.26:0/2272694997 learned_addr learned my addr 172.21.15.26:0/2272694997 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:37.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f1b51700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17ec0618a0 0x7f17ec15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f1350700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0b4240 0x7f17ec14d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 -- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0b4240 msgr2=0x7f17ec14d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0b4240 0x7f17ec14d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 -- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17ec0618a0 msgr2=0x7f17ec15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17ec0618a0 0x7f17ec15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 -- 172.21.15.26:0/2272694997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e800b1e0 con 0x7f17ec14ac80 2022-07-20T02:34:37.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2352700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f17ec0b7420 tx=0x7f17e4007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17e400f470 con 0x7f17ec14ac80 2022-07-20T02:34:37.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400f5d0 con 0x7f17ec14ac80 2022-07-20T02:34:37.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec13e3c0 con 0x7f17ec14ac80 2022-07-20T02:34:37.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17e400f930 con 0x7f17ec14ac80 2022-07-20T02:34:37.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec13e900 con 0x7f17ec14ac80 2022-07-20T02:34:37.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f17e4019030 con 0x7f17ec14ac80 2022-07-20T02:34:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec13fac0 con 0x7f17ec14ac80 2022-07-20T02:34:37.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 0x7f17d4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f17e406b520 con 0x7f17ec14ac80 2022-07-20T02:34:37.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.447+0000 7f17f1b51700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 0x7f17d4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.451+0000 7f17f1b51700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 0x7f17d4055f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f17dc00c000 tx=0x7f17dc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.459+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17e4042580 con 0x7f17ec14ac80 2022-07-20T02:34:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.583+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f17ec14b6d0 con 0x7f17ec14ac80 2022-07-20T02:34:37.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17e2ffd700 1 -- 172.21.15.26:0/2272694997 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v135) v1 ==== 92+0+10 (secure 0 0 0) 0x7f17e4045ba0 con 0x7f17ec14ac80 2022-07-20T02:34:37.595 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:37.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 msgr2=0x7f17d4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 0x7f17d4055f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f17dc00c000 tx=0x7f17dc008040).stop 2022-07-20T02:34:37.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 msgr2=0x7f17ec1416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f17ec0b7420 tx=0x7f17e4007f10).stop 2022-07-20T02:34:37.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 shutdown_connections 2022-07-20T02:34:37.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17ec14ac80 0x7f17ec1416d0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17d4053aa0 0x7f17d4055f60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17ec0b4240 0x7f17ec14d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 --2- 172.21.15.26:0/2272694997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17ec0618a0 0x7f17ec15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 >> 172.21.15.26:0/2272694997 conn(0x7f17ec066f70 msgr2=0x7f17ec05bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 shutdown_connections 2022-07-20T02:34:37.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.587+0000 7f17f2b53700 1 -- 172.21.15.26:0/2272694997 wait complete. 2022-07-20T02:34:37.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:34:37.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 Processor -- start 2022-07-20T02:34:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 -- start start 2022-07-20T02:34:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f636005cfd0 0x7f6360143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6360060f00 0x7f636013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63601474e0 con 0x7f6360147000 2022-07-20T02:34:37.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600b4e40 con 0x7f6360060f00 2022-07-20T02:34:37.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6364fd3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636005d3b0 con 0x7f636005cfd0 2022-07-20T02:34:37.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6356ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6360147000 0x7f63600b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42138/0 (socket says 172.21.15.26:42138) 2022-07-20T02:34:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f6356ffd700 1 -- 172.21.15.26:0/3470108356 learned_addr learned my addr 172.21.15.26:0/3470108356 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.739+0000 7f635ffff700 1 --2- 172.21.15.26:0/3470108356 >> v2:172.21.15.26:3300/0 conn(0x7f6360060f00 0x7f636013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635f7fe700 1 --2- 172.21.15.26:0/3470108356 >> v2:172.21.15.120:3300/0 conn(0x7f636005cfd0 0x7f6360143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635effd700 1 -- 172.21.15.26:0/3470108356 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360212890 0 0) 0x7f63601474e0 con 0x7f6360147000 2022-07-20T02:34:37.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635effd700 1 -- 172.21.15.26:0/3470108356 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003260 con 0x7f6360147000 2022-07-20T02:34:37.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635ffff700 1 -- 172.21.15.26:0/3470108356 >> v2:172.21.15.120:3300/0 conn(0x7f636005cfd0 msgr2=0x7f6360143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:37.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635ffff700 1 --2- 172.21.15.26:0/3470108356 >> v2:172.21.15.120:3300/0 conn(0x7f636005cfd0 0x7f6360143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635ffff700 1 -- 172.21.15.26:0/3470108356 >> v1:172.21.15.26:6789/0 conn(0x7f6360147000 legacy=0x7f63600b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635ffff700 1 -- 172.21.15.26:0/3470108356 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63601431e0 con 0x7f6360060f00 2022-07-20T02:34:37.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635ffff700 1 --2- 172.21.15.26:0/3470108356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360060f00 0x7f636013c9b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f63580032f0 tx=0x7f63580091d0).ready entity=mon.0 client_cookie=779bf65ee2299765 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635effd700 1 -- 172.21.15.26:0/3470108356 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f635800b600 con 0x7f6360060f00 2022-07-20T02:34:37.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635effd700 1 -- 172.21.15.26:0/3470108356 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800b760 con 0x7f6360060f00 2022-07-20T02:34:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f635effd700 1 -- 172.21.15.26:0/3470108356 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f635800b8d0 con 0x7f6360060f00 2022-07-20T02:34:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 -- 172.21.15.26:0/3470108356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360060f00 msgr2=0x7f636013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 --2- 172.21.15.26:0/3470108356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360060f00 0x7f636013c9b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f63580032f0 tx=0x7f63580091d0).stop 2022-07-20T02:34:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 -- 172.21.15.26:0/3470108356 shutdown_connections 2022-07-20T02:34:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 --2- 172.21.15.26:0/3470108356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360060f00 0x7f636013c9b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 --2- 172.21.15.26:0/3470108356 >> v2:172.21.15.120:3300/0 conn(0x7f636005cfd0 0x7f6360143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 -- 172.21.15.26:0/3470108356 >> 172.21.15.26:0/3470108356 conn(0x7f63600620c0 msgr2=0x7f636005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 -- 172.21.15.26:0/3470108356 shutdown_connections 2022-07-20T02:34:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.743+0000 7f6364fd3700 1 -- 172.21.15.26:0/3470108356 wait complete. 2022-07-20T02:34:37.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 Processor -- start 2022-07-20T02:34:37.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- start start 2022-07-20T02:34:37.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6360060f00 0x7f63600b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360147000 0x7f63601526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360156740 con 0x7f6360146050 2022-07-20T02:34:37.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600af330 con 0x7f6360147000 2022-07-20T02:34:37.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6360155790 con 0x7f6360060f00 2022-07-20T02:34:37.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42866/0 (socket says 172.21.15.26:42866) 2022-07-20T02:34:37.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 -- 172.21.15.26:0/2261251397 learned_addr learned my addr 172.21.15.26:0/2261251397 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:37.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6356ffd700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360147000 0x7f63601526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635f7fe700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6360060f00 0x7f63600b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 -- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6360060f00 msgr2=0x7f63600b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6360060f00 0x7f63600b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 -- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360147000 msgr2=0x7f63601526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360147000 0x7f63601526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 -- 172.21.15.26:0/2261251397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358009470 con 0x7f6360146050 2022-07-20T02:34:37.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635ffff700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f6360145510 tx=0x7f6358016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6358011b20 con 0x7f6360146050 2022-07-20T02:34:37.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6358011c80 con 0x7f6360146050 2022-07-20T02:34:37.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63600ad690 con 0x7f6360146050 2022-07-20T02:34:37.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f635801f3f0 con 0x7f6360146050 2022-07-20T02:34:37.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.747+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360155980 con 0x7f6360146050 2022-07-20T02:34:37.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63601523e0 con 0x7f6360146050 2022-07-20T02:34:37.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6358026030 con 0x7f6360146050 2022-07-20T02:34:37.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f635d7fa700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 0x7f6344055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:37.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f635806c760 con 0x7f6360146050 2022-07-20T02:34:37.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f635f7fe700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 0x7f6344055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:37.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.751+0000 7f635f7fe700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 0x7f6344055ed0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f635000b000 tx=0x7f6350006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:37.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.759+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6358043860 con 0x7f6360146050 2022-07-20T02:34:37.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6360143480 con 0x7f6360146050 2022-07-20T02:34:37.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f635d7fa700 1 -- 172.21.15.26:0/2261251397 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+143 (secure 0 0 0) 0x7f63580240a0 con 0x7f6360146050 2022-07-20T02:34:37.958 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:37.958 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:34:37.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:37.958 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:34:37.959 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:34:37.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:37.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:37.960 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:34:37.960 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:34:37.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:37.961 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:34:37.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 msgr2=0x7f6344055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 0x7f6344055ed0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f635000b000 tx=0x7f6350006df0).stop 2022-07-20T02:34:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 msgr2=0x7f6360152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f6360145510 tx=0x7f6358016000).stop 2022-07-20T02:34:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 shutdown_connections 2022-07-20T02:34:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6360146050 0x7f6360152bd0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6344053a10 0x7f6344055ed0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360147000 0x7f63601526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 --2- 172.21.15.26:0/2261251397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6360060f00 0x7f63600b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:37.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 >> 172.21.15.26:0/2261251397 conn(0x7f636005cfd0 msgr2=0x7f6360139d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:37.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 shutdown_connections 2022-07-20T02:34:37.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:37.887+0000 7f6364fd3700 1 -- 172.21.15.26:0/2261251397 wait complete. 2022-07-20T02:34:37.968 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:34:37.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:34:38.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 Processor -- start 2022-07-20T02:34:38.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 -- start start 2022-07-20T02:34:38.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0c3805cfd0 0x7f0c381407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c380b1160 0x7f0c38140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0c38146e80 0x7f0c381472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c381435b0 con 0x7f0c38146e80 2022-07-20T02:34:38.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3805d3b0 con 0x7f0c3805cfd0 2022-07-20T02:34:38.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.095+0000 7f0c407f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b4bc0 con 0x7f0c380b1160 2022-07-20T02:34:38.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0c38146e80 0x7f0c381472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3e58e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0c3805cfd0 0x7f0c381407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0c38146e80 0x7f0c381472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42874/0 (socket says 172.21.15.26:42874) 2022-07-20T02:34:38.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3dd8d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c380b1160 0x7f0c38140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 -- 172.21.15.26:0/438306948 learned_addr learned my addr 172.21.15.26:0/438306948 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:38.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 -- 172.21.15.26:0/438306948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c380b1160 msgr2=0x7f0c38140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:38.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 --2- 172.21.15.26:0/438306948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c380b1160 0x7f0c38140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 -- 172.21.15.26:0/438306948 >> v2:172.21.15.120:3300/0 conn(0x7f0c3805cfd0 msgr2=0x7f0c381407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:38.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 --2- 172.21.15.26:0/438306948 >> v2:172.21.15.120:3300/0 conn(0x7f0c3805cfd0 0x7f0c381407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 -- 172.21.15.26:0/438306948 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380b4e60 con 0x7f0c38146e80 2022-07-20T02:34:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3ed8f700 1 --2- 172.21.15.26:0/438306948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c38146e80 0x7f0c381472f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f0c300032f0 tx=0x7f0c30009290).ready entity=mon.0 client_cookie=3e349d0d67941182 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3d58c700 1 -- 172.21.15.26:0/438306948 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c3000b680 con 0x7f0c38146e80 2022-07-20T02:34:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3d58c700 1 -- 172.21.15.26:0/438306948 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3000b7e0 con 0x7f0c38146e80 2022-07-20T02:34:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c3d58c700 1 -- 172.21.15.26:0/438306948 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c3000b950 con 0x7f0c38146e80 2022-07-20T02:34:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 -- 172.21.15.26:0/438306948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c38146e80 msgr2=0x7f0c381472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 --2- 172.21.15.26:0/438306948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c38146e80 0x7f0c381472f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f0c300032f0 tx=0x7f0c30009290).stop 2022-07-20T02:34:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 -- 172.21.15.26:0/438306948 shutdown_connections 2022-07-20T02:34:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 --2- 172.21.15.26:0/438306948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c38146e80 0x7f0c381472f0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 --2- 172.21.15.26:0/438306948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c380b1160 0x7f0c38140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 --2- 172.21.15.26:0/438306948 >> v2:172.21.15.120:3300/0 conn(0x7f0c3805cfd0 0x7f0c381407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 -- 172.21.15.26:0/438306948 >> 172.21.15.26:0/438306948 conn(0x7f0c38066f70 msgr2=0x7f0c3805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:38.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 -- 172.21.15.26:0/438306948 shutdown_connections 2022-07-20T02:34:38.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.099+0000 7f0c407f2700 1 -- 172.21.15.26:0/438306948 wait complete. 2022-07-20T02:34:38.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 Processor -- start 2022-07-20T02:34:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- start start 2022-07-20T02:34:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 0x7f0c3813fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c38146e80 0x7f0c38145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38143b50 con 0x7f0c380b1160 2022-07-20T02:34:38.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380aedc0 con 0x7f0c38146e80 2022-07-20T02:34:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38146990 con 0x7f0c3805cfd0 2022-07-20T02:34:38.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42878/0 (socket says 172.21.15.26:42878) 2022-07-20T02:34:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 -- 172.21.15.26:0/2427149893 learned_addr learned my addr 172.21.15.26:0/2427149893 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3ed8f700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c38146e80 0x7f0c38145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3e58e700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 0x7f0c3813fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 -- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 msgr2=0x7f0c3813fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 0x7f0c3813fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 -- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c38146e80 msgr2=0x7f0c38145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c38146e80 0x7f0c38145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 -- 172.21.15.26:0/2427149893 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30009530 con 0x7f0c380b1160 2022-07-20T02:34:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3e58e700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 0x7f0c3813fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:38.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c3dd8d700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f0c3400c000 tx=0x7f0c34006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:38.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c34010780 con 0x7f0c380b1160 2022-07-20T02:34:38.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c340108e0 con 0x7f0c380b1160 2022-07-20T02:34:38.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c380ae4f0 con 0x7f0c380b1160 2022-07-20T02:34:38.163 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c34010c40 con 0x7f0c380b1160 2022-07-20T02:34:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.103+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c380ae2c0 con 0x7f0c380b1160 2022-07-20T02:34:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0c3401c460 con 0x7f0c380b1160 2022-07-20T02:34:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38143e20 con 0x7f0c380b1160 2022-07-20T02:34:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c277fe700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 0x7f0c18055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f0c3406c240 con 0x7f0c380b1160 2022-07-20T02:34:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c3e58e700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 0x7f0c18055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.107+0000 7f0c3e58e700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 0x7f0c18055f60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f0c2800b000 tx=0x7f0c28008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.111+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c340431d0 con 0x7f0c380b1160 2022-07-20T02:34:38.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.239+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f0c380574e0 con 0x7f0c380b1160 2022-07-20T02:34:38.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.727+0000 7f0c277fe700 1 -- 172.21.15.26:0/2427149893 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v136) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0c340467f0 con 0x7f0c380b1160 2022-07-20T02:34:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 msgr2=0x7f0c18055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 0x7f0c18055f60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f0c2800b000 tx=0x7f0c28008040).stop 2022-07-20T02:34:38.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 msgr2=0x7f0c38145450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f0c3400c000 tx=0x7f0c34006df0).stop 2022-07-20T02:34:38.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 shutdown_connections 2022-07-20T02:34:38.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0c380b1160 0x7f0c38145450 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0c18053aa0 0x7f0c18055f60 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c38146e80 0x7f0c38145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 --2- 172.21.15.26:0/2427149893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c3805cfd0 0x7f0c3813fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:38.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 >> 172.21.15.26:0/2427149893 conn(0x7f0c38066f70 msgr2=0x7f0c380b6940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:38.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 shutdown_connections 2022-07-20T02:34:38.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.731+0000 7f0c407f2700 1 -- 172.21.15.26:0/2427149893 wait complete. 2022-07-20T02:34:38.761 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:34:38.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:34:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.959+0000 7fcd85aa3700 1 Processor -- start 2022-07-20T02:34:38.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.959+0000 7fcd85aa3700 1 -- start start 2022-07-20T02:34:38.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd80066c00 0x7fcd800ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:38.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd80146bb0 con 0x7fcd80060c80 2022-07-20T02:34:38.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd80146e50 con 0x7fcd8005ce30 2022-07-20T02:34:38.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd801470f0 con 0x7fcd80066c00 2022-07-20T02:34:38.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd852a2700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcd80060c80 0x7fcd80061110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42168/0 (socket says 172.21.15.26:42168) 2022-07-20T02:34:38.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd852a2700 1 -- 172.21.15.26:0/3671823422 learned_addr learned my addr 172.21.15.26:0/3671823422 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:38.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7ffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcd80066c00 0x7fcd800ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:38.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7f7fe700 1 -- 172.21.15.26:0/3671823422 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729668867 0 0) 0x7fcd80146bb0 con 0x7fcd80060c80 2022-07-20T02:34:38.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7f7fe700 1 -- 172.21.15.26:0/3671823422 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd64003260 con 0x7fcd80060c80 2022-07-20T02:34:38.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7ffff700 1 -- 172.21.15.26:0/3671823422 >> v1:172.21.15.120:6789/0 conn(0x7fcd8005ce30 legacy=0x7fcd800ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7ffff700 1 -- 172.21.15.26:0/3671823422 >> v1:172.21.15.26:6789/0 conn(0x7fcd80060c80 legacy=0x7fcd80061110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7ffff700 1 -- 172.21.15.26:0/3671823422 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80147390 con 0x7fcd80066c00 2022-07-20T02:34:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7ffff700 1 --2- 172.21.15.26:0/3671823422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd80066c00 0x7fcd800ace00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fcd6c002f70 tx=0x7fcd6c006df0).ready entity=mon.0 client_cookie=106e3a3427a5c189 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:38.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7f7fe700 1 -- 172.21.15.26:0/3671823422 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd6c00d610 con 0x7fcd80066c00 2022-07-20T02:34:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7f7fe700 1 -- 172.21.15.26:0/3671823422 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c000e90 con 0x7fcd80066c00 2022-07-20T02:34:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd7f7fe700 1 -- 172.21.15.26:0/3671823422 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd6c00d960 con 0x7fcd80066c00 2022-07-20T02:34:39.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- 172.21.15.26:0/3671823422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd80066c00 msgr2=0x7fcd800ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/3671823422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd80066c00 0x7fcd800ace00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fcd6c002f70 tx=0x7fcd6c006df0).stop 2022-07-20T02:34:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- 172.21.15.26:0/3671823422 shutdown_connections 2022-07-20T02:34:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/3671823422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd80066c00 0x7fcd800ace00 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- 172.21.15.26:0/3671823422 >> 172.21.15.26:0/3671823422 conn(0x7fcd800b7090 msgr2=0x7fcd8005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- 172.21.15.26:0/3671823422 shutdown_connections 2022-07-20T02:34:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.963+0000 7fcd85aa3700 1 -- 172.21.15.26:0/3671823422 wait complete. 2022-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.967+0000 7fcd85aa3700 1 Processor -- start 2022-07-20T02:34:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.967+0000 7fcd85aa3700 1 -- start start 2022-07-20T02:34:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd80060c80 0x7fcd80148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 0x7fcd800661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8013eb40 con 0x7fcd8005ce30 2022-07-20T02:34:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80148520 con 0x7fcd80066c00 2022-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801531a0 con 0x7fcd80060c80 2022-07-20T02:34:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42890/0 (socket says 172.21.15.26:42890) 2022-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 -- 172.21.15.26:0/1023416057 learned_addr learned my addr 172.21.15.26:0/1023416057 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd7ffff700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd80060c80 0x7fcd80148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd852a2700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 0x7fcd800661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 -- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd80060c80 msgr2=0x7fcd80148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd80060c80 0x7fcd80148750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 -- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 msgr2=0x7fcd800661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 0x7fcd800661c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 -- 172.21.15.26:0/1023416057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c00b180 con 0x7fcd8005ce30 2022-07-20T02:34:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd852a2700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 0x7fcd800661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd84aa1700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fcd80063cc0 tx=0x7fcd74007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd74012400 con 0x7fcd8005ce30 2022-07-20T02:34:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd74012560 con 0x7fcd8005ce30 2022-07-20T02:34:39.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd80147aa0 con 0x7fcd8005ce30 2022-07-20T02:34:39.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd740128c0 con 0x7fcd8005ce30 2022-07-20T02:34:39.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd80148280 con 0x7fcd8005ce30 2022-07-20T02:34:39.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.971+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd80061820 con 0x7fcd8005ce30 2022-07-20T02:34:39.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.975+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd74019030 con 0x7fcd8005ce30 2022-07-20T02:34:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.975+0000 7fcd7dffb700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 0x7fcd60055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.975+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5954+0+0 (secure 0 0 0) 0x7fcd7406b260 con 0x7fcd8005ce30 2022-07-20T02:34:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.975+0000 7fcd7ffff700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 0x7fcd60055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.975+0000 7fcd7ffff700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 0x7fcd60055fc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fcd6c01a000 tx=0x7fcd6c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:38.983+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd740421f0 con 0x7fcd8005ce30 2022-07-20T02:34:39.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.115+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fcd800574e0 con 0x7fcd8005ce30 2022-07-20T02:34:39.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.727+0000 7fcd7dffb700 1 -- 172.21.15.26:0/1023416057 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v137) v1 ==== 125+0+0 (secure 0 0 0) 0x7fcd74045810 con 0x7fcd8005ce30 2022-07-20T02:34:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 msgr2=0x7fcd60055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 0x7fcd60055fc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fcd6c01a000 tx=0x7fcd6c014000).stop 2022-07-20T02:34:39.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 msgr2=0x7fcd80152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fcd80063cc0 tx=0x7fcd74007c50).stop 2022-07-20T02:34:39.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 shutdown_connections 2022-07-20T02:34:39.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd60053b00 0x7fcd60055fc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd80066c00 0x7fcd800661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd80060c80 0x7fcd80148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 --2- 172.21.15.26:0/1023416057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd8005ce30 0x7fcd80152cb0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 >> 172.21.15.26:0/1023416057 conn(0x7fcd800b7090 msgr2=0x7fcd800b56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:39.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 shutdown_connections 2022-07-20T02:34:39.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.735+0000 7fcd85aa3700 1 -- 172.21.15.26:0/1023416057 wait complete. 2022-07-20T02:34:39.787 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:34:39.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:34:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.927+0000 7f1e87fff700 1 Processor -- start 2022-07-20T02:34:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.927+0000 7f1e87fff700 1 -- start start 2022-07-20T02:34:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e87fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1e88147000 0x7f1e88143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e87fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e881474e0 con 0x7f1e88060f00 2022-07-20T02:34:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e87fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e880b4e40 con 0x7f1e8805cfd0 2022-07-20T02:34:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e87fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8805d3b0 con 0x7f1e88147000 2022-07-20T02:34:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e877fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1e88060f00 0x7f1e8813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42182/0 (socket says 172.21.15.26:42182) 2022-07-20T02:34:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e877fe700 1 -- 172.21.15.26:0/1019531830 learned_addr learned my addr 172.21.15.26:0/1019531830 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e867fc700 1 --2- 172.21.15.26:0/1019531830 >> v2:172.21.15.120:3300/0 conn(0x7f1e88147000 0x7f1e88143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691453013 0 0) 0x7f1e881474e0 con 0x7f1e88060f00 2022-07-20T02:34:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e68003260 con 0x7f1e88060f00 2022-07-20T02:34:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430203806 0 0) 0x7f1e880b4e40 con 0x7f1e8805cfd0 2022-07-20T02:34:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e881474e0 con 0x7f1e8805cfd0 2022-07-20T02:34:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e867fc700 1 -- 172.21.15.26:0/1019531830 >> v1:172.21.15.120:6789/0 conn(0x7f1e8805cfd0 legacy=0x7f1e880b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e867fc700 1 -- 172.21.15.26:0/1019531830 >> v1:172.21.15.26:6789/0 conn(0x7f1e88060f00 legacy=0x7f1e8813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e867fc700 1 -- 172.21.15.26:0/1019531830 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e881431e0 con 0x7f1e88147000 2022-07-20T02:34:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e867fc700 1 --2- 172.21.15.26:0/1019531830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e88147000 0x7f1e88143bc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f1e70002f70 tx=0x7f1e70006df0).ready entity=mon.1 client_cookie=f0ffda5816b3eec7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e7000d540 con 0x7f1e88147000 2022-07-20T02:34:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70000e90 con 0x7f1e88147000 2022-07-20T02:34:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.931+0000 7f1e85ffb700 1 -- 172.21.15.26:0/1019531830 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e7000d890 con 0x7f1e88147000 2022-07-20T02:34:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- 172.21.15.26:0/1019531830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e88147000 msgr2=0x7f1e88143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 --2- 172.21.15.26:0/1019531830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e88147000 0x7f1e88143bc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f1e70002f70 tx=0x7f1e70006df0).stop 2022-07-20T02:34:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- 172.21.15.26:0/1019531830 shutdown_connections 2022-07-20T02:34:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 --2- 172.21.15.26:0/1019531830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e88147000 0x7f1e88143bc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- 172.21.15.26:0/1019531830 >> 172.21.15.26:0/1019531830 conn(0x7f1e880620c0 msgr2=0x7f1e8805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- 172.21.15.26:0/1019531830 shutdown_connections 2022-07-20T02:34:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- 172.21.15.26:0/1019531830 wait complete. 2022-07-20T02:34:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 Processor -- start 2022-07-20T02:34:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- start start 2022-07-20T02:34:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e88060f00 0x7f1e88145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8814a570 0x7f1e88155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e88156280 con 0x7f1e88147000 2022-07-20T02:34:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e880b2ec0 con 0x7f1e8814a570 2022-07-20T02:34:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e87fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e881532d0 con 0x7f1e88060f00 2022-07-20T02:34:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e867fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e867fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42908/0 (socket says 172.21.15.26:42908) 2022-07-20T02:34:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e867fc700 1 -- 172.21.15.26:0/3352569901 learned_addr learned my addr 172.21.15.26:0/3352569901 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e86ffd700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e88060f00 0x7f1e88145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.935+0000 7f1e877fe700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8814a570 0x7f1e88155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 -- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e88060f00 msgr2=0x7f1e88145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e88060f00 0x7f1e88145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 -- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8814a570 msgr2=0x7f1e88155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8814a570 0x7f1e88155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 -- 172.21.15.26:0/3352569901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7000b180 con 0x7f1e88147000 2022-07-20T02:34:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e867fc700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7f1e881563c0 tx=0x7f1e70014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e70016070 con 0x7f1e88147000 2022-07-20T02:34:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70013850 con 0x7f1e88147000 2022-07-20T02:34:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88153030 con 0x7f1e88147000 2022-07-20T02:34:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e70013bb0 con 0x7f1e88147000 2022-07-20T02:34:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e880b3320 con 0x7f1e88147000 2022-07-20T02:34:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e70025030 con 0x7f1e88147000 2022-07-20T02:34:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e880b3d40 con 0x7f1e88147000 2022-07-20T02:34:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 0x7f1e6c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e86ffd700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 0x7f1e6c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.939+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f1e7006cee0 con 0x7f1e88147000 2022-07-20T02:34:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.943+0000 7f1e86ffd700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 0x7f1e6c055f30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f1e7800d000 tx=0x7f1e78007a60).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:39.947+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e70043c10 con 0x7f1e88147000 2022-07-20T02:34:40.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.115+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1e88143480 con 0x7f1e88147000 2022-07-20T02:34:40.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.127+0000 7f1e7ffff700 1 -- 172.21.15.26:0/3352569901 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v71) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1e700230d0 con 0x7f1e88147000 2022-07-20T02:34:40.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 msgr2=0x7f1e6c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 0x7f1e6c055f30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f1e7800d000 tx=0x7f1e78007a60).stop 2022-07-20T02:34:40.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 msgr2=0x7f1e881558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7f1e881563c0 tx=0x7f1e70014000).stop 2022-07-20T02:34:40.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 shutdown_connections 2022-07-20T02:34:40.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e88147000 0x7f1e881558a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1e6c053a70 0x7f1e6c055f30 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0x7f1e7800d000 tx=0x7f1e78007a60).stop 2022-07-20T02:34:40.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8814a570 0x7f1e88155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 --2- 172.21.15.26:0/3352569901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e88060f00 0x7f1e88145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.131+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 >> 172.21.15.26:0/3352569901 conn(0x7f1e8805cfd0 msgr2=0x7f1e8813bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:40.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.139+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 shutdown_connections 2022-07-20T02:34:40.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.139+0000 7f1e87fff700 1 -- 172.21.15.26:0/3352569901 wait complete. 2022-07-20T02:34:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 25 and data pool 26 2022-07-20T02:34:40.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:40.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 Processor -- start 2022-07-20T02:34:40.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 -- start start 2022-07-20T02:34:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7380066b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73800b0f50 con 0x7f7380060250 2022-07-20T02:34:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738005f8e0 con 0x7f73800621f0 2022-07-20T02:34:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f73849c9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7380060df0 con 0x7f73800aff40 2022-07-20T02:34:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f737f59e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7380060250 0x7f73800606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42208/0 (socket says 172.21.15.26:42208) 2022-07-20T02:34:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f737f59e700 1 -- 172.21.15.26:0/3313504705 learned_addr learned my addr 172.21.15.26:0/3313504705 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f737ed9d700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7380066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467554042 0 0) 0x7f73800b0f50 con 0x7f7380060250 2022-07-20T02:34:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.291+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003260 con 0x7f7380060250 2022-07-20T02:34:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349761966 0 0) 0x7f738005f8e0 con 0x7f73800621f0 2022-07-20T02:34:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73800b0f50 con 0x7f73800621f0 2022-07-20T02:34:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366892824 0 0) 0x7f7364003260 con 0x7f7380060250 2022-07-20T02:34:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 msgr2=0x7f7380066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7380066b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 >> v1:172.21.15.120:6789/0 conn(0x7f73800621f0 legacy=0x7f7380067020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738013e7f0 con 0x7f7380060250 2022-07-20T02:34:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f73780019e0 con 0x7f7380060250 2022-07-20T02:34:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 >> v1:172.21.15.26:6789/0 conn(0x7f7380060250 legacy=0x7f73800606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003910 0x7f7364003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7364004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73640049f0 0x7f7364004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7380060df0 con 0x7f73640049f0 2022-07-20T02:34:40.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736400b780 con 0x7f7364003910 2022-07-20T02:34:40.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736400b940 con 0x7f73800aff40 2022-07-20T02:34:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73640049f0 0x7f7364004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737ed9d700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003910 0x7f7364003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737e59c700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7364004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 -- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 msgr2=0x7f7364004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73800aff40 0x7f7364004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 -- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003910 msgr2=0x7f7364003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7364003910 0x7f7364003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 -- 172.21.15.26:0/3313504705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364003260 con 0x7f73640049f0 2022-07-20T02:34:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73640049f0 0x7f7364004e70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f73780017b0 tx=0x7f7378002150).ready entity=mon.0 client_cookie=c25f055fcf3ac3f8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f737800cb80 con 0x7f73640049f0 2022-07-20T02:34:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737800cce0 con 0x7f73640049f0 2022-07-20T02:34:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737dd9b700 1 -- 172.21.15.26:0/3313504705 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7378005b80 con 0x7f73640049f0 2022-07-20T02:34:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 -- 172.21.15.26:0/3313504705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73640049f0 msgr2=0x7f7364004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 --2- 172.21.15.26:0/3313504705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73640049f0 0x7f7364004e70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f73780017b0 tx=0x7f7378002150).stop 2022-07-20T02:34:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f737f59e700 1 -- 172.21.15.26:0/3313504705 reap_dead start 2022-07-20T02:34:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 -- 172.21.15.26:0/3313504705 shutdown_connections 2022-07-20T02:34:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 -- 172.21.15.26:0/3313504705 >> 172.21.15.26:0/3313504705 conn(0x7f73800b3050 msgr2=0x7f73800b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 -- 172.21.15.26:0/3313504705 shutdown_connections 2022-07-20T02:34:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.295+0000 7f73849c9700 1 -- 172.21.15.26:0/3313504705 wait complete. 2022-07-20T02:34:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 Processor -- start 2022-07-20T02:34:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 -- start start 2022-07-20T02:34:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 0x7f7380063b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 0x7f7380143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b5260 con 0x7f7380066c00 2022-07-20T02:34:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738013f520 con 0x7f738014df80 2022-07-20T02:34:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f73849c9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738013f780 con 0x7f7380140180 2022-07-20T02:34:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f737ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f737ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42936/0 (socket says 172.21.15.26:42936) 2022-07-20T02:34:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.299+0000 7f737ed9d700 1 -- 172.21.15.26:0/4186449720 learned_addr learned my addr 172.21.15.26:0/4186449720 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737e59c700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 0x7f7380063b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737f59e700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 0x7f7380143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 -- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 msgr2=0x7f7380063b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 0x7f7380063b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 -- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 msgr2=0x7f7380143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 0x7f7380143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 -- 172.21.15.26:0/4186449720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73801408c0 con 0x7f7380066c00 2022-07-20T02:34:40.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737e59c700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 0x7f7380063b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:40.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737f59e700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 0x7f7380143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f737ed9d700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f737400b000 tx=0x7f7374006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f737400f940 con 0x7f7380066c00 2022-07-20T02:34:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737400faa0 con 0x7f7380066c00 2022-07-20T02:34:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7380140b90 con 0x7f7380066c00 2022-07-20T02:34:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f737400fe40 con 0x7f7380066c00 2022-07-20T02:34:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738014e400 con 0x7f7380066c00 2022-07-20T02:34:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738013f380 con 0x7f7380066c00 2022-07-20T02:34:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7374014440 con 0x7f7380066c00 2022-07-20T02:34:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.303+0000 7f736ffff700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 0x7f735c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.307+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f737406cb50 con 0x7f7380066c00 2022-07-20T02:34:40.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.307+0000 7f737e59c700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 0x7f735c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.307+0000 7f737e59c700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 0x7f735c055e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f7370003000 tx=0x7f7370009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.311+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f738013f380 con 0x7f7380066c00 2022-07-20T02:34:40.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.435+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738013f380 con 0x7f7380066c00 2022-07-20T02:34:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.435+0000 7f736ffff700 1 -- 172.21.15.26:0/4186449720 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16359 (secure 0 0 0) 0x7f738013f380 con 0x7f7380066c00 2022-07-20T02:34:40.468 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:40.469 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":138,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:40.117480+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 msgr2=0x7f735c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 0x7f735c055e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f7370003000 tx=0x7f7370009040).stop 2022-07-20T02:34:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 msgr2=0x7f7380066fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f737400b000 tx=0x7f7374006df0).stop 2022-07-20T02:34:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 shutdown_connections 2022-07-20T02:34:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f735c053980 0x7f735c055e40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f738014df80 0x7f7380143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7380140180 0x7f7380063b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 --2- 172.21.15.26:0/4186449720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7380066c00 0x7f7380066fe0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 >> 172.21.15.26:0/4186449720 conn(0x7f73800b3050 msgr2=0x7f738013ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:40.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 shutdown_connections 2022-07-20T02:34:40.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.439+0000 7f73849c9700 1 -- 172.21.15.26:0/4186449720 wait complete. 2022-07-20T02:34:40.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 Processor -- start 2022-07-20T02:34:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 -- start start 2022-07-20T02:34:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e740957f0 0x7f0e740b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e741433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74147080 con 0x7f0e740b2f20 2022-07-20T02:34:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e74143000 con 0x7f0e740957f0 2022-07-20T02:34:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e78875700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e741472b0 con 0x7f0e74067440 2022-07-20T02:34:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e737fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0e740b2f20 0x7f0e740950e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42224/0 (socket says 172.21.15.26:42224) 2022-07-20T02:34:40.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e737fe700 1 -- 172.21.15.26:0/3413013587 learned_addr learned my addr 172.21.15.26:0/3413013587 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:40.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e72ffd700 1 --2- 172.21.15.26:0/3413013587 >> v2:172.21.15.120:3300/0 conn(0x7f0e740957f0 0x7f0e740b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e73fff700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e741433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.615+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548221620 0 0) 0x7f0e74147080 con 0x7f0e740b2f20 2022-07-20T02:34:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54003260 con 0x7f0e740b2f20 2022-07-20T02:34:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861335579 0 0) 0x7f0e54003260 con 0x7f0e740b2f20 2022-07-20T02:34:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 msgr2=0x7f0e741433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e741433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 >> v2:172.21.15.120:3300/0 conn(0x7f0e740957f0 msgr2=0x7f0e740b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 --2- 172.21.15.26:0/3413013587 >> v2:172.21.15.120:3300/0 conn(0x7f0e740957f0 0x7f0e740b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e740a9920 con 0x7f0e740b2f20 2022-07-20T02:34:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f0e64001680 con 0x7f0e740b2f20 2022-07-20T02:34:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 >> v1:172.21.15.26:6789/0 conn(0x7f0e740b2f20 legacy=0x7f0e740950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e54003750 0x7f0e54003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e54004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e54004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e741472b0 con 0x7f0e540049f0 2022-07-20T02:34:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54004330 con 0x7f0e54003750 2022-07-20T02:34:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5400b7a0 con 0x7f0e74067440 2022-07-20T02:34:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e54004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e72ffd700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e54004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e737fe700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e54003750 0x7f0e54003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 -- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 msgr2=0x7f0e54004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74067440 0x7f0e54004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 -- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e54003750 msgr2=0x7f0e54003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e54003750 0x7f0e54003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 -- 172.21.15.26:0/3413013587 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e54003260 con 0x7f0e540049f0 2022-07-20T02:34:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e54004e70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f0e54004470 tx=0x7f0e6c006df0).ready entity=mon.0 client_cookie=115846aeafb00d62 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e6c00e830 con 0x7f0e540049f0 2022-07-20T02:34:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c00be90 con 0x7f0e540049f0 2022-07-20T02:34:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e727fc700 1 -- 172.21.15.26:0/3413013587 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e6c00eb80 con 0x7f0e540049f0 2022-07-20T02:34:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 -- 172.21.15.26:0/3413013587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 msgr2=0x7f0e54004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 --2- 172.21.15.26:0/3413013587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e54004e70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f0e54004470 tx=0x7f0e6c006df0).stop 2022-07-20T02:34:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e73fff700 1 -- 172.21.15.26:0/3413013587 reap_dead start 2022-07-20T02:34:40.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 -- 172.21.15.26:0/3413013587 shutdown_connections 2022-07-20T02:34:40.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 -- 172.21.15.26:0/3413013587 >> 172.21.15.26:0/3413013587 conn(0x7f0e740b67f0 msgr2=0x7f0e740b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:40.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 -- 172.21.15.26:0/3413013587 shutdown_connections 2022-07-20T02:34:40.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.619+0000 7f0e78875700 1 -- 172.21.15.26:0/3413013587 wait complete. 2022-07-20T02:34:40.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 Processor -- start 2022-07-20T02:34:40.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- start start 2022-07-20T02:34:40.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74152080 0x7f0e741524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e741529d0 0x7f0e74155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7409b790 con 0x7f0e540049f0 2022-07-20T02:34:40.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e740aff80 con 0x7f0e741529d0 2022-07-20T02:34:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7409b2f0 con 0x7f0e74152080 2022-07-20T02:34:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e72ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74152080 0x7f0e741524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42958/0 (socket says 172.21.15.26:42958) 2022-07-20T02:34:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 -- 172.21.15.26:0/1084710846 learned_addr learned my addr 172.21.15.26:0/1084710846 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e73fff700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e741529d0 0x7f0e74155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 -- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74152080 msgr2=0x7f0e741524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74152080 0x7f0e741524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 -- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e741529d0 msgr2=0x7f0e74155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e741529d0 0x7f0e74155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 -- 172.21.15.26:0/1084710846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7414fc30 con 0x7f0e540049f0 2022-07-20T02:34:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e737fe700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f0e6400b000 tx=0x7f0e640014e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e64015630 con 0x7f0e540049f0 2022-07-20T02:34:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64010040 con 0x7f0e540049f0 2022-07-20T02:34:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e74093170 con 0x7f0e540049f0 2022-07-20T02:34:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e64015980 con 0x7f0e540049f0 2022-07-20T02:34:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.623+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7409aba0 con 0x7f0e540049f0 2022-07-20T02:34:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e740b0140 con 0x7f0e540049f0 2022-07-20T02:34:40.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e6401e3f0 con 0x7f0e540049f0 2022-07-20T02:34:40.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e70ff9700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 0x7f0e58055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0e6406bb20 con 0x7f0e540049f0 2022-07-20T02:34:40.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e72ffd700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 0x7f0e58055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.627+0000 7f0e72ffd700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 0x7f0e58055e60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0e5c00b000 tx=0x7f0e5c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:40.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.635+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e64042880 con 0x7f0e540049f0 2022-07-20T02:34:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.759+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e740a95a0 con 0x7f0e540049f0 2022-07-20T02:34:40.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.759+0000 7f0e70ff9700 1 -- 172.21.15.26:0/1084710846 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16359 (secure 0 0 0) 0x7f0e64045ea0 con 0x7f0e540049f0 2022-07-20T02:34:40.807 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:40.813 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":138,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:40.117480+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:40.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 msgr2=0x7f0e58055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 0x7f0e58055e60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0e5c00b000 tx=0x7f0e5c008040).stop 2022-07-20T02:34:40.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 msgr2=0x7f0e741452b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f0e6400b000 tx=0x7f0e640014e0).stop 2022-07-20T02:34:40.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 shutdown_connections 2022-07-20T02:34:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0e580539a0 0x7f0e58055e60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e741529d0 0x7f0e74155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e74152080 0x7f0e741524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 --2- 172.21.15.26:0/1084710846 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e540049f0 0x7f0e741452b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:40.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 >> 172.21.15.26:0/1084710846 conn(0x7f0e740b67f0 msgr2=0x7f0e74145da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:40.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 shutdown_connections 2022-07-20T02:34:40.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.763+0000 7f0e78875700 1 -- 172.21.15.26:0/1084710846 wait complete. 2022-07-20T02:34:40.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:34:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 Processor -- start 2022-07-20T02:34:40.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 -- start start 2022-07-20T02:34:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f84ec05f840 0x7f84ec146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:40.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0b1ae0 con 0x7f84ec1465b0 2022-07-20T02:34:40.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0b1d80 con 0x7f84ec0b24c0 2022-07-20T02:34:40.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f1518700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec0b2e20 con 0x7f84ec05f840 2022-07-20T02:34:40.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84ebfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f84ec1465b0 0x7f84ec066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42244/0 (socket says 172.21.15.26:42244) 2022-07-20T02:34:40.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84ebfff700 1 -- 172.21.15.26:0/4074128815 learned_addr learned my addr 172.21.15.26:0/4074128815 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:40.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.935+0000 7f84f0d17700 1 --2- 172.21.15.26:0/4074128815 >> v2:172.21.15.26:3300/0 conn(0x7f84ec05f840 0x7f84ec146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:40.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816199719 0 0) 0x7f84ec0b1ae0 con 0x7f84ec1465b0 2022-07-20T02:34:40.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d0003260 con 0x7f84ec1465b0 2022-07-20T02:34:40.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274105232 0 0) 0x7f84ec0b1d80 con 0x7f84ec0b24c0 2022-07-20T02:34:40.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec0b1ae0 con 0x7f84ec0b24c0 2022-07-20T02:34:40.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f0d17700 1 -- 172.21.15.26:0/4074128815 >> v1:172.21.15.120:6789/0 conn(0x7f84ec0b24c0 legacy=0x7f84ec0604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f0d17700 1 -- 172.21.15.26:0/4074128815 >> v1:172.21.15.26:6789/0 conn(0x7f84ec1465b0 legacy=0x7f84ec066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:40.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f0d17700 1 -- 172.21.15.26:0/4074128815 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0b2020 con 0x7f84ec05f840 2022-07-20T02:34:41.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f0d17700 1 --2- 172.21.15.26:0/4074128815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec05f840 0x7f84ec146990 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f84e40032f0 tx=0x7f84e40091d0).ready entity=mon.0 client_cookie=7663056c6467d88b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e400b600 con 0x7f84ec05f840 2022-07-20T02:34:41.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e400b760 con 0x7f84ec05f840 2022-07-20T02:34:41.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84eb7fe700 1 -- 172.21.15.26:0/4074128815 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e400b8d0 con 0x7f84ec05f840 2022-07-20T02:34:41.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 -- 172.21.15.26:0/4074128815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec05f840 msgr2=0x7f84ec146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 --2- 172.21.15.26:0/4074128815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec05f840 0x7f84ec146990 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f84e40032f0 tx=0x7f84e40091d0).stop 2022-07-20T02:34:41.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 -- 172.21.15.26:0/4074128815 shutdown_connections 2022-07-20T02:34:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 --2- 172.21.15.26:0/4074128815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec05f840 0x7f84ec146990 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 -- 172.21.15.26:0/4074128815 >> 172.21.15.26:0/4074128815 conn(0x7f84ec0b3050 msgr2=0x7f84ec0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 -- 172.21.15.26:0/4074128815 shutdown_connections 2022-07-20T02:34:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.939+0000 7f84f1518700 1 -- 172.21.15.26:0/4074128815 wait complete. 2022-07-20T02:34:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 Processor -- start 2022-07-20T02:34:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 -- start start 2022-07-20T02:34:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84ec05f840 0x7f84ec1526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84ec0ad190 0x7f84ec141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0aa340 con 0x7f84ec1465b0 2022-07-20T02:34:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0ac2f0 con 0x7f84ec05f840 2022-07-20T02:34:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec142480 con 0x7f84ec0ad190 2022-07-20T02:34:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42970/0 (socket says 172.21.15.26:42970) 2022-07-20T02:34:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 -- 172.21.15.26:0/1262858026 learned_addr learned my addr 172.21.15.26:0/1262858026 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f0d17700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84ec0ad190 0x7f84ec141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84ebfff700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84ec05f840 0x7f84ec1526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 -- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84ec0ad190 msgr2=0x7f84ec141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84ec0ad190 0x7f84ec141f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 -- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84ec05f840 msgr2=0x7f84ec1526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84ec05f840 0x7f84ec1526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 -- 172.21.15.26:0/1262858026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4009470 con 0x7f84ec1465b0 2022-07-20T02:34:41.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e37fe700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f84d4001110 tx=0x7f84d4007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d4012400 con 0x7f84ec1465b0 2022-07-20T02:34:41.053 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4012560 con 0x7f84ec1465b0 2022-07-20T02:34:41.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec0ac6b0 con 0x7f84ec1465b0 2022-07-20T02:34:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.943+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d40128c0 con 0x7f84ec1465b0 2022-07-20T02:34:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec152e40 con 0x7f84ec1465b0 2022-07-20T02:34:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec147450 con 0x7f84ec1465b0 2022-07-20T02:34:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84d4019030 con 0x7f84ec1465b0 2022-07-20T02:34:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84e9ffb700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 0x7f84c4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f84d406b5a0 con 0x7f84ec1465b0 2022-07-20T02:34:41.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84ebfff700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 0x7f84c4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.947+0000 7f84ebfff700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 0x7f84c4055ea0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f84dc00d000 tx=0x7f84dc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:40.951+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84d4042250 con 0x7f84ec1465b0 2022-07-20T02:34:41.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.111+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f84ec0674c0 con 0x7f84ec1465b0 2022-07-20T02:34:41.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.127+0000 7f84e9ffb700 1 -- 172.21.15.26:0/1262858026 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f84d4045870 con 0x7f84ec1465b0 2022-07-20T02:34:41.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 msgr2=0x7f84c4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 0x7f84c4055ea0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f84dc00d000 tx=0x7f84dc009040).stop 2022-07-20T02:34:41.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 msgr2=0x7f84ec141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f84d4001110 tx=0x7f84d4007b60).stop 2022-07-20T02:34:41.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 shutdown_connections 2022-07-20T02:34:41.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f84ec1465b0 0x7f84ec141aa0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f84c40539e0 0x7f84c4055ea0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84ec0ad190 0x7f84ec141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 --2- 172.21.15.26:0/1262858026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84ec05f840 0x7f84ec1526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 >> 172.21.15.26:0/1262858026 conn(0x7f84ec0b3050 msgr2=0x7f84ec148770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 shutdown_connections 2022-07-20T02:34:41.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.131+0000 7f84f1518700 1 -- 172.21.15.26:0/1262858026 wait complete. 2022-07-20T02:34:41.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:41.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 Processor -- start 2022-07-20T02:34:41.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 -- start start 2022-07-20T02:34:41.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c4c0b1900 0x7f8c4c0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c140190 con 0x7f8c4c0b5d50 2022-07-20T02:34:41.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0af740 con 0x7f8c4c0b1900 2022-07-20T02:34:41.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.259+0000 7f8c52a18700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c140430 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c4c0b1900 0x7f8c4c0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8c4c0b1900 0x7f8c4c0afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42980/0 (socket says 172.21.15.26:42980) 2022-07-20T02:34:41.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 -- 172.21.15.26:0/3731333215 learned_addr learned my addr 172.21.15.26:0/3731333215 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:41.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52217700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 -- 172.21.15.26:0/3731333215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0ac8e0 msgr2=0x7f8c4c0b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 -- 172.21.15.26:0/3731333215 >> v1:172.21.15.120:6789/0 conn(0x7f8c4c0b5d50 legacy=0x7f8c4c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 -- 172.21.15.26:0/3731333215 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c1406d0 con 0x7f8c4c0b1900 2022-07-20T02:34:41.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c51215700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0b1900 0x7f8c4c0afe40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f8c3c002f70 tx=0x7f8c3c006eb0).ready entity=mon.0 client_cookie=b774d1aa4f11a428 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c50a14700 1 -- 172.21.15.26:0/3731333215 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c3c00d570 con 0x7f8c4c0b1900 2022-07-20T02:34:41.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c50a14700 1 -- 172.21.15.26:0/3731333215 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c000e90 con 0x7f8c4c0b1900 2022-07-20T02:34:41.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c50a14700 1 -- 172.21.15.26:0/3731333215 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c3c00d900 con 0x7f8c4c0b1900 2022-07-20T02:34:41.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 -- 172.21.15.26:0/3731333215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0b1900 msgr2=0x7f8c4c0afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0b1900 0x7f8c4c0afe40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f8c3c002f70 tx=0x7f8c3c006eb0).stop 2022-07-20T02:34:41.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 -- 172.21.15.26:0/3731333215 shutdown_connections 2022-07-20T02:34:41.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 --2- 172.21.15.26:0/3731333215 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0b1900 0x7f8c4c0afe40 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 -- 172.21.15.26:0/3731333215 >> 172.21.15.26:0/3731333215 conn(0x7f8c4c060310 msgr2=0x7f8c4c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 -- 172.21.15.26:0/3731333215 shutdown_connections 2022-07-20T02:34:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.263+0000 7f8c52a18700 1 -- 172.21.15.26:0/3731333215 wait complete. 2022-07-20T02:34:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 Processor -- start 2022-07-20T02:34:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 -- start start 2022-07-20T02:34:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0b1900 0x7f8c4c152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 0x7f8c4c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c144100 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c141d50 con 0x7f8c4c0b5d50 2022-07-20T02:34:41.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c141ff0 con 0x7f8c4c0b1900 2022-07-20T02:34:41.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42984/0 (socket says 172.21.15.26:42984) 2022-07-20T02:34:41.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 -- 172.21.15.26:0/2393763301 learned_addr learned my addr 172.21.15.26:0/2393763301 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:41.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51215700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0b1900 0x7f8c4c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52217700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 0x7f8c4c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 -- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0b1900 msgr2=0x7f8c4c152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0b1900 0x7f8c4c152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 -- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 msgr2=0x7f8c4c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 0x7f8c4c152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 -- 172.21.15.26:0/2393763301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c00b1e0 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52217700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 0x7f8c4c152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:41.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c51a16700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f8c4c0b53b0 tx=0x7f8c48007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c48007e70 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c48007030 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c142290 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.267+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c48007390 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c142990 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c140970 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8c48010030 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c3affd700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 0x7f8c2c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8c4806bcc0 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c51215700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 0x7f8c2c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.271+0000 7f8c51215700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 0x7f8c2c055fc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8c3c01c000 tx=0x7f8c3c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.275+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c48072080 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.439+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c4c0574e0 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.439+0000 7f8c3affd700 1 -- 172.21.15.26:0/2393763301 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f8c48042970 con 0x7f8c4c0ac8e0 2022-07-20T02:34:41.502 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:41.503 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:34:41.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 msgr2=0x7f8c2c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 0x7f8c2c055fc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8c3c01c000 tx=0x7f8c3c015000).stop 2022-07-20T02:34:41.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 msgr2=0x7f8c4c0b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f8c4c0b53b0 tx=0x7f8c48007c00).stop 2022-07-20T02:34:41.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 shutdown_connections 2022-07-20T02:34:41.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8c2c053b00 0x7f8c2c055fc0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c4c0b5d50 0x7f8c4c152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c4c0b1900 0x7f8c4c152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 --2- 172.21.15.26:0/2393763301 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c4c0ac8e0 0x7f8c4c0b4860 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 >> 172.21.15.26:0/2393763301 conn(0x7f8c4c060310 msgr2=0x7f8c4c0b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 shutdown_connections 2022-07-20T02:34:41.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.443+0000 7f8c52a18700 1 -- 172.21.15.26:0/2393763301 wait complete. 2022-07-20T02:34:41.513 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:34:41.514 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:41.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.627+0000 7f1479705700 1 Processor -- start 2022-07-20T02:34:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.627+0000 7f1479705700 1 -- start start 2022-07-20T02:34:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f14740604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1474097c50 0x7f14740609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740b3640 con 0x7f14740b1720 2022-07-20T02:34:41.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1474060ed0 con 0x7f1474097c50 2022-07-20T02:34:41.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1474095300 con 0x7f14740b3260 2022-07-20T02:34:41.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1474097c50 0x7f14740609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1474097c50 0x7f14740609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:34452/0 (socket says 172.21.15.26:34452) 2022-07-20T02:34:41.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 -- 172.21.15.26:0/335440785 learned_addr learned my addr 172.21.15.26:0/335440785 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:41.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1478f04700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f14740b1720 0x7f14740b1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56486/0 (socket says 172.21.15.26:56486) 2022-07-20T02:34:41.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1473fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f14740604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 -- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 msgr2=0x7f14740604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 --2- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f14740604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 -- 172.21.15.26:0/335440785 >> v1:172.21.15.120:6789/0 conn(0x7f14740b1720 legacy=0x7f14740b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1472ffd700 1 -- 172.21.15.26:0/335440785 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809147470 0 0) 0x7f14740b3640 con 0x7f14740b1720 2022-07-20T02:34:41.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 -- 172.21.15.26:0/335440785 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740955a0 con 0x7f1474097c50 2022-07-20T02:34:41.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f14737fe700 1 --2- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1474097c50 0x7f14740609e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f1468002f70 tx=0x7f1468006eb0).ready entity=mon.1 client_cookie=f9289693cacd147e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1472ffd700 1 -- 172.21.15.26:0/335440785 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f146800d570 con 0x7f1474097c50 2022-07-20T02:34:41.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1472ffd700 1 -- 172.21.15.26:0/335440785 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468000e90 con 0x7f1474097c50 2022-07-20T02:34:41.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1472ffd700 1 -- 172.21.15.26:0/335440785 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f146800d8c0 con 0x7f1474097c50 2022-07-20T02:34:41.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 -- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1474097c50 msgr2=0x7f14740609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.631+0000 7f1479705700 1 --2- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1474097c50 0x7f14740609e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f1468002f70 tx=0x7f1468006eb0).stop 2022-07-20T02:34:41.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- 172.21.15.26:0/335440785 shutdown_connections 2022-07-20T02:34:41.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 --2- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1474097c50 0x7f14740609e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 --2- 172.21.15.26:0/335440785 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f14740604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- 172.21.15.26:0/335440785 >> 172.21.15.26:0/335440785 conn(0x7f14740b67f0 msgr2=0x7f14740b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- 172.21.15.26:0/335440785 shutdown_connections 2022-07-20T02:34:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- 172.21.15.26:0/335440785 wait complete. 2022-07-20T02:34:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 Processor -- start 2022-07-20T02:34:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- start start 2022-07-20T02:34:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 0x7f14741529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f1474152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740653b0 con 0x7f1474097c50 2022-07-20T02:34:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741533d0 con 0x7f14740b1720 2022-07-20T02:34:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1479705700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1474094820 con 0x7f14740b3260 2022-07-20T02:34:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f14737fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 0x7f14741529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1478f04700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f1474152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43000/0 (socket says 172.21.15.26:43000) 2022-07-20T02:34:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 -- 172.21.15.26:0/1919094342 learned_addr learned my addr 172.21.15.26:0/1919094342 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 -- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 msgr2=0x7f1474152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f1474152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 -- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 msgr2=0x7f14741529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 0x7f14741529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 -- 172.21.15.26:0/1919094342 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146800b1e0 con 0x7f1474097c50 2022-07-20T02:34:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f14737fe700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 0x7f14741529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.635+0000 7f1473fff700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f146400c000 tx=0x7f1464006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1464011750 con 0x7f1474097c50 2022-07-20T02:34:41.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640118b0 con 0x7f1474097c50 2022-07-20T02:34:41.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1474094ac0 con 0x7f1474097c50 2022-07-20T02:34:41.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1464011c10 con 0x7f1474097c50 2022-07-20T02:34:41.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1474152550 con 0x7f1474097c50 2022-07-20T02:34:41.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474095840 con 0x7f1474097c50 2022-07-20T02:34:41.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1464011d70 con 0x7f1474097c50 2022-07-20T02:34:41.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 0x7f1454055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:41.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f146406c4e0 con 0x7f1474097c50 2022-07-20T02:34:41.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.639+0000 7f14737fe700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 0x7f1454055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:41.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.643+0000 7f14737fe700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 0x7f1454055fc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f146801c000 tx=0x7f1468015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:41.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.647+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14640432a0 con 0x7f1474097c50 2022-07-20T02:34:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.775+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14740574e0 con 0x7f1474097c50 2022-07-20T02:34:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f14717fa700 1 -- 172.21.15.26:0/1919094342 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f14640468c0 con 0x7f1474097c50 2022-07-20T02:34:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:41.796 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":139,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:41.123699+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 msgr2=0x7f1454055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 0x7f1454055fc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f146801c000 tx=0x7f1468015000).stop 2022-07-20T02:34:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 msgr2=0x7f14740975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f146400c000 tx=0x7f1464006df0).stop 2022-07-20T02:34:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 shutdown_connections 2022-07-20T02:34:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1454053b00 0x7f1454055fc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f14740b3260 0x7f1474152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14740b1720 0x7f14741529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 --2- 172.21.15.26:0/1919094342 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1474097c50 0x7f14740975e0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:41.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 >> 172.21.15.26:0/1919094342 conn(0x7f14740b67f0 msgr2=0x7f1474140780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:41.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 shutdown_connections 2022-07-20T02:34:41.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:41.779+0000 7f1479705700 1 -- 172.21.15.26:0/1919094342 wait complete. 2022-07-20T02:34:41.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:34:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 Processor -- start 2022-07-20T02:34:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 -- start start 2022-07-20T02:34:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f79580b9c80 0x7f79581431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79580ba060 con 0x7f79580b5b70 2022-07-20T02:34:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958061f80 con 0x7f7958062710 2022-07-20T02:34:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795e7b7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795812bee0 con 0x7f79580b9c80 2022-07-20T02:34:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795cd54700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79580b5b70 0x7f79581403e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42290/0 (socket says 172.21.15.26:42290) 2022-07-20T02:34:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f795cd54700 1 -- 172.21.15.26:0/4149508989 learned_addr learned my addr 172.21.15.26:0/4149508989 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7957fff700 1 --2- 172.21.15.26:0/4149508989 >> v2:172.21.15.120:3300/0 conn(0x7f79580b9c80 0x7f79581431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668250772 0 0) 0x7f79580ba060 con 0x7f79580b5b70 2022-07-20T02:34:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7944003260 con 0x7f79580b5b70 2022-07-20T02:34:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493022994 0 0) 0x7f7958061f80 con 0x7f7958062710 2022-07-20T02:34:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79580ba060 con 0x7f7958062710 2022-07-20T02:34:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7957fff700 1 -- 172.21.15.26:0/4149508989 >> v1:172.21.15.120:6789/0 conn(0x7f7958062710 legacy=0x7f79580b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7957fff700 1 -- 172.21.15.26:0/4149508989 >> v1:172.21.15.26:6789/0 conn(0x7f79580b5b70 legacy=0x7f79581403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.039+0000 7f7957fff700 1 -- 172.21.15.26:0/4149508989 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958062220 con 0x7f79580b9c80 2022-07-20T02:34:42.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f7957fff700 1 --2- 172.21.15.26:0/4149508989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79580b9c80 0x7f79581431a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f7948002f70 tx=0x7f7948006df0).ready entity=mon.1 client_cookie=30f3eb1377cda3a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794800d540 con 0x7f79580b9c80 2022-07-20T02:34:42.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7948000e90 con 0x7f79580b9c80 2022-07-20T02:34:42.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f7956ffd700 1 -- 172.21.15.26:0/4149508989 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794800d890 con 0x7f79580b9c80 2022-07-20T02:34:42.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- 172.21.15.26:0/4149508989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79580b9c80 msgr2=0x7f79581431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 --2- 172.21.15.26:0/4149508989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79580b9c80 0x7f79581431a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f7948002f70 tx=0x7f7948006df0).stop 2022-07-20T02:34:42.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- 172.21.15.26:0/4149508989 shutdown_connections 2022-07-20T02:34:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 --2- 172.21.15.26:0/4149508989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79580b9c80 0x7f79581431a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- 172.21.15.26:0/4149508989 >> 172.21.15.26:0/4149508989 conn(0x7f7958066f70 msgr2=0x7f795805faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- 172.21.15.26:0/4149508989 shutdown_connections 2022-07-20T02:34:42.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- 172.21.15.26:0/4149508989 wait complete. 2022-07-20T02:34:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 Processor -- start 2022-07-20T02:34:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.043+0000 7f795e7b7700 1 -- start start 2022-07-20T02:34:42.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7958062710 0x7f7958148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79580b9c80 0x7f7958147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958147510 con 0x7f79580b5b70 2022-07-20T02:34:42.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958145050 con 0x7f7958062710 2022-07-20T02:34:42.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79581457c0 con 0x7f79580b9c80 2022-07-20T02:34:42.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43012/0 (socket says 172.21.15.26:43012) 2022-07-20T02:34:42.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 -- 172.21.15.26:0/537841049 learned_addr learned my addr 172.21.15.26:0/537841049 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:42.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f7957fff700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7958062710 0x7f7958148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795cd54700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79580b9c80 0x7f7958147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 -- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79580b9c80 msgr2=0x7f7958147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79580b9c80 0x7f7958147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 -- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7958062710 msgr2=0x7f7958148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7958062710 0x7f7958148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 -- 172.21.15.26:0/537841049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794800b180 con 0x7f79580b5b70 2022-07-20T02:34:42.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79577fe700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7f795812c350 tx=0x7f794c007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794c012400 con 0x7f79580b5b70 2022-07-20T02:34:42.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c012560 con 0x7f79580b5b70 2022-07-20T02:34:42.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7958145280 con 0x7f79580b5b70 2022-07-20T02:34:42.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794c0128c0 con 0x7f79580b5b70 2022-07-20T02:34:42.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7958146690 con 0x7f79580b5b70 2022-07-20T02:34:42.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.047+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f794c019030 con 0x7f79580b5b70 2022-07-20T02:34:42.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.051+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958144df0 con 0x7f79580b5b70 2022-07-20T02:34:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.051+0000 7f79557fa700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 0x7f7938055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.051+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f794c06b5a0 con 0x7f79580b5b70 2022-07-20T02:34:42.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.051+0000 7f7957fff700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 0x7f7938055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.051+0000 7f7957fff700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 0x7f7938055ea0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f794801a000 tx=0x7f7948014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.055+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f794c042250 con 0x7f79580b5b70 2022-07-20T02:34:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.227+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7958146850 con 0x7f79580b5b70 2022-07-20T02:34:42.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f79557fa700 1 -- 172.21.15.26:0/537841049 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f794c045870 con 0x7f79580b5b70 2022-07-20T02:34:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 msgr2=0x7f7938055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 0x7f7938055ea0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f794801a000 tx=0x7f7948014000).stop 2022-07-20T02:34:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 msgr2=0x7f7958146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7f795812c350 tx=0x7f794c007b60).stop 2022-07-20T02:34:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 shutdown_connections 2022-07-20T02:34:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79580b5b70 0x7f7958146b30 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f79380539e0 0x7f7938055ea0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f79580b9c80 0x7f7958147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 --2- 172.21.15.26:0/537841049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7958062710 0x7f7958148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 >> 172.21.15.26:0/537841049 conn(0x7f7958066f70 msgr2=0x7f79580b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 shutdown_connections 2022-07-20T02:34:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.235+0000 7f795e7b7700 1 -- 172.21.15.26:0/537841049 wait complete. 2022-07-20T02:34:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:34:42.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:34:42.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.367+0000 7f6e35339700 1 Processor -- start 2022-07-20T02:34:42.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.367+0000 7f6e35339700 1 -- start start 2022-07-20T02:34:42.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 0x7f6e300b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e30064220 0x7f6e301434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300b2e40 con 0x7f6e300b7d80 2022-07-20T02:34:42.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300b4c20 con 0x7f6e30064220 2022-07-20T02:34:42.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300617a0 con 0x7f6e300b3060 2022-07-20T02:34:42.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e30064220 0x7f6e301434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6e30064220 0x7f6e301434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43024/0 (socket says 172.21.15.26:43024) 2022-07-20T02:34:42.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 -- 172.21.15.26:0/1077139244 learned_addr learned my addr 172.21.15.26:0/1077139244 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:42.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e2f7fe700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 0x7f6e300b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 -- 172.21.15.26:0/1077139244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 msgr2=0x7f6e300b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 0x7f6e300b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 -- 172.21.15.26:0/1077139244 >> v1:172.21.15.120:6789/0 conn(0x7f6e300b7d80 legacy=0x7f6e300b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 -- 172.21.15.26:0/1077139244 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e30060b20 con 0x7f6e30064220 2022-07-20T02:34:42.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e2f7fe700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 0x7f6e300b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:42.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e34b38700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e30064220 0x7f6e301434f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f6e300618e0 tx=0x7f6e28009290).ready entity=mon.0 client_cookie=dc58aa7310ddd266 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e2effd700 1 -- 172.21.15.26:0/1077139244 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e2800b860 con 0x7f6e30064220 2022-07-20T02:34:42.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e2effd700 1 -- 172.21.15.26:0/1077139244 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800b9c0 con 0x7f6e30064220 2022-07-20T02:34:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e2effd700 1 -- 172.21.15.26:0/1077139244 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e2800bb30 con 0x7f6e30064220 2022-07-20T02:34:42.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- 172.21.15.26:0/1077139244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e30064220 msgr2=0x7f6e301434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e30064220 0x7f6e301434f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f6e300618e0 tx=0x7f6e28009290).stop 2022-07-20T02:34:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- 172.21.15.26:0/1077139244 shutdown_connections 2022-07-20T02:34:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e30064220 0x7f6e301434f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 --2- 172.21.15.26:0/1077139244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e300b3060 0x7f6e300b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- 172.21.15.26:0/1077139244 >> 172.21.15.26:0/1077139244 conn(0x7f6e3005f840 msgr2=0x7f6e3005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- 172.21.15.26:0/1077139244 shutdown_connections 2022-07-20T02:34:42.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.371+0000 7f6e35339700 1 -- 172.21.15.26:0/1077139244 wait complete. 2022-07-20T02:34:42.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 Processor -- start 2022-07-20T02:34:42.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 -- start start 2022-07-20T02:34:42.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e30064220 0x7f6e300b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e300b7d80 0x7f6e3013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3013d4d0 con 0x7f6e300b3060 2022-07-20T02:34:42.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301488b0 con 0x7f6e300b7d80 2022-07-20T02:34:42.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30148b50 con 0x7f6e30064220 2022-07-20T02:34:42.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e30064220 0x7f6e300b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43028/0 (socket says 172.21.15.26:43028) 2022-07-20T02:34:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 -- 172.21.15.26:0/576766868 learned_addr learned my addr 172.21.15.26:0/576766868 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e34b38700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e300b7d80 0x7f6e3013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 -- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e30064220 msgr2=0x7f6e300b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e30064220 0x7f6e300b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 -- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e300b7d80 msgr2=0x7f6e3013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e300b7d80 0x7f6e3013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 -- 172.21.15.26:0/576766868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28009530 con 0x7f6e300b3060 2022-07-20T02:34:42.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2f7fe700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7f6e1800a000 tx=0x7f6e18008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e18011780 con 0x7f6e300b3060 2022-07-20T02:34:42.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e180118e0 con 0x7f6e300b3060 2022-07-20T02:34:42.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e30148df0 con 0x7f6e300b3060 2022-07-20T02:34:42.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.375+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e18011c40 con 0x7f6e300b3060 2022-07-20T02:34:42.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e30144a70 con 0x7f6e300b3060 2022-07-20T02:34:42.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e18012420 con 0x7f6e300b3060 2022-07-20T02:34:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e2d7fa700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 0x7f6e14055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6e1806be30 con 0x7f6e300b3060 2022-07-20T02:34:42.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30149290 con 0x7f6e300b3060 2022-07-20T02:34:42.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e2ffff700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 0x7f6e14055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.379+0000 7f6e2ffff700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 0x7f6e14055ea0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f6e2000d000 tx=0x7f6e20007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:42.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.383+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e18042b90 con 0x7f6e300b3060 2022-07-20T02:34:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.551+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6e30060dc0 con 0x7f6e300b3060 2022-07-20T02:34:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e2d7fa700 1 -- 172.21.15.26:0/576766868 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6e180461b0 con 0x7f6e300b3060 2022-07-20T02:34:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 msgr2=0x7f6e14055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 0x7f6e14055ea0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f6e2000d000 tx=0x7f6e20007b90).stop 2022-07-20T02:34:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 msgr2=0x7f6e3013caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:42.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7f6e1800a000 tx=0x7f6e18008040).stop 2022-07-20T02:34:42.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 shutdown_connections 2022-07-20T02:34:42.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6e300b3060 0x7f6e3013caf0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6e140539e0 0x7f6e14055ea0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e300b7d80 0x7f6e3013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 --2- 172.21.15.26:0/576766868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e30064220 0x7f6e300b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:42.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 >> 172.21.15.26:0/576766868 conn(0x7f6e3005f840 msgr2=0x7f6e3005cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:42.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 shutdown_connections 2022-07-20T02:34:42.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.559+0000 7f6e35339700 1 -- 172.21.15.26:0/576766868 wait complete. 2022-07-20T02:34:42.814 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:34:42.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:42.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.967+0000 7f2724739700 1 Processor -- start 2022-07-20T02:34:42.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.967+0000 7f2724739700 1 -- start start 2022-07-20T02:34:42.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b7920 0x7f271c067480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f271c0b49c0 0x7f271c144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 0x7f271c05f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:42.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c0b44a0 con 0x7f271c0b49c0 2022-07-20T02:34:42.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c144670 con 0x7f271c145230 2022-07-20T02:34:42.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0b4f40 con 0x7f271c0b7920 2022-07-20T02:34:42.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f271c0b49c0 0x7f271c144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f271c0b49c0 0x7f271c144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43036/0 (socket says 172.21.15.26:43036) 2022-07-20T02:34:42.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2722cd6700 1 --2- 172.21.15.26:0/2404893601 >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 0x7f271c05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:42.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 -- 172.21.15.26:0/2404893601 learned_addr learned my addr 172.21.15.26:0/2404893601 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:42.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f27224d5700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b7920 0x7f271c067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 -- 172.21.15.26:0/2404893601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b7920 msgr2=0x7f271c067480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:43.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b7920 0x7f271c067480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 -- 172.21.15.26:0/2404893601 >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 msgr2=0x7f271c05f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:43.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2404893601 >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 0x7f271c05f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 -- 172.21.15.26:0/2404893601 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c1448a0 con 0x7f271c0b49c0 2022-07-20T02:34:43.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2722cd6700 1 --2- 172.21.15.26:0/2404893601 >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 0x7f271c05f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:43.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b49c0 0x7f271c144d40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f2710002f70 tx=0x7f2710006eb0).ready entity=mon.0 client_cookie=7cf4645fae03ec2b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:43.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f27214d3700 1 -- 172.21.15.26:0/2404893601 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f271000d570 con 0x7f271c0b49c0 2022-07-20T02:34:43.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f27214d3700 1 -- 172.21.15.26:0/2404893601 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710000e90 con 0x7f271c0b49c0 2022-07-20T02:34:43.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f27214d3700 1 -- 172.21.15.26:0/2404893601 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f271000d900 con 0x7f271c0b49c0 2022-07-20T02:34:43.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- 172.21.15.26:0/2404893601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b49c0 msgr2=0x7f271c144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b49c0 0x7f271c144d40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f2710002f70 tx=0x7f2710006eb0).stop 2022-07-20T02:34:43.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- 172.21.15.26:0/2404893601 shutdown_connections 2022-07-20T02:34:43.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- 172.21.15.26:0/2404893601 >> v2:172.21.15.120:3300/0 conn(0x7f271c145230 0x7f271c05f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b49c0 0x7f271c144d40 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 --2- 172.21.15.26:0/2404893601 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b7920 0x7f271c067480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- 172.21.15.26:0/2404893601 >> 172.21.15.26:0/2404893601 conn(0x7f271c064b10 msgr2=0x7f271c064ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:43.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- 172.21.15.26:0/2404893601 shutdown_connections 2022-07-20T02:34:43.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.971+0000 7f2724739700 1 -- 172.21.15.26:0/2404893601 wait complete. 2022-07-20T02:34:43.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 Processor -- start 2022-07-20T02:34:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 -- start start 2022-07-20T02:34:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b49c0 0x7f271c0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c145230 0x7f271c0adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0ac390 con 0x7f271c0b7920 2022-07-20T02:34:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0ac5e0 con 0x7f271c145230 2022-07-20T02:34:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2724739700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c154220 con 0x7f271c0b49c0 2022-07-20T02:34:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43042/0 (socket says 172.21.15.26:43042) 2022-07-20T02:34:43.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 -- 172.21.15.26:0/2167753935 learned_addr learned my addr 172.21.15.26:0/2167753935 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:43.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f27224d5700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b49c0 0x7f271c0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2722cd6700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c145230 0x7f271c0adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 -- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b49c0 msgr2=0x7f271c0af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b49c0 0x7f271c0af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 -- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c145230 msgr2=0x7f271c0adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c145230 0x7f271c0adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.975+0000 7f2721cd4700 1 -- 172.21.15.26:0/2167753935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c00b180 con 0x7f271c0b7920 2022-07-20T02:34:43.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f2721cd4700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7f2710002f70 tx=0x7f2710015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:43.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2710011760 con 0x7f271c0b7920 2022-07-20T02:34:43.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100118c0 con 0x7f271c0b7920 2022-07-20T02:34:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271000b1e0 con 0x7f271c0b7920 2022-07-20T02:34:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2710011c20 con 0x7f271c0b7920 2022-07-20T02:34:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c0ae150 con 0x7f271c0b7920 2022-07-20T02:34:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c0574e0 con 0x7f271c0b7920 2022-07-20T02:34:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2710025030 con 0x7f271c0b7920 2022-07-20T02:34:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 0x7f26fc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f271006d050 con 0x7f271c0b7920 2022-07-20T02:34:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.979+0000 7f27224d5700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 0x7f26fc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.983+0000 7f27224d5700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 0x7f26fc055e10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f271400b000 tx=0x7f27140091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:42.987+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2710043e10 con 0x7f271c0b7920 2022-07-20T02:34:43.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.151+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f271c1543e0 con 0x7f271c0b7920 2022-07-20T02:34:43.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.151+0000 7f271b7fe700 1 -- 172.21.15.26:0/2167753935 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f2710023020 con 0x7f271c0b7920 2022-07-20T02:34:43.159 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:43.159 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:34:43.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 msgr2=0x7f26fc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 0x7f26fc055e10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f271400b000 tx=0x7f27140091d0).stop 2022-07-20T02:34:43.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 msgr2=0x7f271c153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7f2710002f70 tx=0x7f2710015000).stop 2022-07-20T02:34:43.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 shutdown_connections 2022-07-20T02:34:43.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f271c0b7920 0x7f271c153a20 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26fc053950 0x7f26fc055e10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f271c145230 0x7f271c0adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 --2- 172.21.15.26:0/2167753935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271c0b49c0 0x7f271c0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 >> 172.21.15.26:0/2167753935 conn(0x7f271c064b10 msgr2=0x7f271c145f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:43.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 shutdown_connections 2022-07-20T02:34:43.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.155+0000 7f2724739700 1 -- 172.21.15.26:0/2167753935 wait complete. 2022-07-20T02:34:43.173 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:34:43.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:34:40.117843+0000', 'modified': '2022-07-20T02:34:41.123008+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8139': {'gid': 8139, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1171289829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 1171289829}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1171289829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:34:43.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:34:43.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:34:43.669 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8139,"want_state":"up:active","state":"up:active","rank_uptime":3.5313515290000002,"mdsmap_epoch":72,"osdmap_epoch":140,"osdmap_epoch_barrier":138,"uptime":8.5917255420000007} 2022-07-20T02:34:43.683 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8139, "mdsmap_epoch": 72, "osdmap_epoch": 140, "osdmap_epoch_barrier": 138, "rank_uptime": 3.531351529, "state": "up:active", "uptime": 8.591725542, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:34:43.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 Processor -- start 2022-07-20T02:34:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 -- start start 2022-07-20T02:34:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 0x7f86bc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f86bc147000 0x7f86bc0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:43.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86bc1474e0 con 0x7f86bc060f00 2022-07-20T02:34:43.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86bc05d3b0 con 0x7f86bc147000 2022-07-20T02:34:43.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c297a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc1431e0 con 0x7f86bc05cfd0 2022-07-20T02:34:43.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f86bc147000 0x7f86bc0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f86bc147000 0x7f86bc0b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43048/0 (socket says 172.21.15.26:43048) 2022-07-20T02:34:43.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 -- 172.21.15.26:0/3131982073 learned_addr learned my addr 172.21.15.26:0/3131982073 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:43.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1978700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 0x7f86bc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:43.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 -- 172.21.15.26:0/3131982073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 msgr2=0x7f86bc143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:43.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 0x7f86bc143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:43.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 -- 172.21.15.26:0/3131982073 >> v1:172.21.15.120:6789/0 conn(0x7f86bc060f00 legacy=0x7f86bc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:43.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 -- 172.21.15.26:0/3131982073 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc143480 con 0x7f86bc147000 2022-07-20T02:34:43.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1978700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 0x7f86bc143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:43.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c1177700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc0b4730 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f86bc068660 tx=0x7f86b8006eb0).ready entity=mon.0 client_cookie=74d9d3413e13558a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:43.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.803+0000 7f86c0976700 1 -- 172.21.15.26:0/3131982073 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86b800d570 con 0x7f86bc147000 2022-07-20T02:34:43.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c0976700 1 -- 172.21.15.26:0/3131982073 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8000e90 con 0x7f86bc147000 2022-07-20T02:34:43.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c0976700 1 -- 172.21.15.26:0/3131982073 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86b800d8c0 con 0x7f86bc147000 2022-07-20T02:34:43.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- 172.21.15.26:0/3131982073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 msgr2=0x7f86bc0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:43.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc0b4730 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f86bc068660 tx=0x7f86b8006eb0).stop 2022-07-20T02:34:44.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- 172.21.15.26:0/3131982073 shutdown_connections 2022-07-20T02:34:44.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc0b4730 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- 172.21.15.26:0/3131982073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc05cfd0 0x7f86bc143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- 172.21.15.26:0/3131982073 >> 172.21.15.26:0/3131982073 conn(0x7f86bc0620c0 msgr2=0x7f86bc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- 172.21.15.26:0/3131982073 shutdown_connections 2022-07-20T02:34:44.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- 172.21.15.26:0/3131982073 wait complete. 2022-07-20T02:34:44.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 Processor -- start 2022-07-20T02:34:44.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- start start 2022-07-20T02:34:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86bc060f00 0x7f86bc1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 0x7f86bc152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc0b0580 con 0x7f86bc147000 2022-07-20T02:34:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc0b0040 con 0x7f86bc060f00 2022-07-20T02:34:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c297a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc0b02e0 con 0x7f86bc145b70 2022-07-20T02:34:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c1177700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c1177700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43056/0 (socket says 172.21.15.26:43056) 2022-07-20T02:34:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.807+0000 7f86c1177700 1 -- 172.21.15.26:0/94818385 learned_addr learned my addr 172.21.15.26:0/94818385 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c2179700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 0x7f86bc152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1978700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86bc060f00 0x7f86bc1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 -- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 msgr2=0x7f86bc152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 0x7f86bc152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 -- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86bc060f00 msgr2=0x7f86bc1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86bc060f00 0x7f86bc1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 -- 172.21.15.26:0/94818385 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b800b1e0 con 0x7f86bc147000 2022-07-20T02:34:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c2179700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 0x7f86bc152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1177700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f86bc068660 tx=0x7f86b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86b80118f0 con 0x7f86bc147000 2022-07-20T02:34:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8011a50 con 0x7f86bc147000 2022-07-20T02:34:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86b8011db0 con 0x7f86bc147000 2022-07-20T02:34:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86bc145850 con 0x7f86bc147000 2022-07-20T02:34:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86bc1443d0 con 0x7f86bc147000 2022-07-20T02:34:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86bc0b77c0 con 0x7f86bc147000 2022-07-20T02:34:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f86b8025030 con 0x7f86bc147000 2022-07-20T02:34:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 0x7f869c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f86b806d480 con 0x7f86bc147000 2022-07-20T02:34:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.811+0000 7f86c1978700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 0x7f869c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.815+0000 7f86c1978700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 0x7f869c055e10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f86ac00b000 tx=0x7f86ac008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.819+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86bc0b77c0 con 0x7f86bc147000 2022-07-20T02:34:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.943+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86bc0b77c0 con 0x7f86bc147000 2022-07-20T02:34:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.943+0000 7f86aaffd700 1 -- 172.21.15.26:0/94818385 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f86bc0b77c0 con 0x7f86bc147000 2022-07-20T02:34:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:44.041 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":140,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:42.132316+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:44.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 msgr2=0x7f869c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 0x7f869c055e10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f86ac00b000 tx=0x7f86ac008040).stop 2022-07-20T02:34:44.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 msgr2=0x7f86bc152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f86bc068660 tx=0x7f86b8015000).stop 2022-07-20T02:34:44.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 shutdown_connections 2022-07-20T02:34:44.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86bc147000 0x7f86bc152770 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f869c053950 0x7f869c055e10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86bc145b70 0x7f86bc152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 --2- 172.21.15.26:0/94818385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86bc060f00 0x7f86bc1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 >> 172.21.15.26:0/94818385 conn(0x7f86bc05cfd0 msgr2=0x7f86bc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 shutdown_connections 2022-07-20T02:34:44.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:43.947+0000 7f86c297a700 1 -- 172.21.15.26:0/94818385 wait complete. 2022-07-20T02:34:44.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:34:38.295386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:34:39.171843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:34:44.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:34:44.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 Processor -- start 2022-07-20T02:34:44.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 -- start start 2022-07-20T02:34:44.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9a4c065980 0x7f9a4c065d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c05d220 con 0x7f9a4c1449d0 2022-07-20T02:34:44.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c0b35d0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a53d66700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c0b3870 con 0x7f9a4c065980 2022-07-20T02:34:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a51301700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9a4c1449d0 0x7f9a4c144e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42344/0 (socket says 172.21.15.26:42344) 2022-07-20T02:34:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a51301700 1 -- 172.21.15.26:0/11365980 learned_addr learned my addr 172.21.15.26:0/11365980 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a51b02700 1 --2- 172.21.15.26:0/11365980 >> v2:172.21.15.120:3300/0 conn(0x7f9a4c065980 0x7f9a4c065d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462636987 0 0) 0x7f9a4c05d220 con 0x7f9a4c1449d0 2022-07-20T02:34:44.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a34003260 con 0x7f9a4c1449d0 2022-07-20T02:34:44.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721943034 0 0) 0x7f9a4c0b35d0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c05d220 con 0x7f9a4c13dd40 2022-07-20T02:34:44.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810818927 0 0) 0x7f9a34003260 con 0x7f9a4c1449d0 2022-07-20T02:34:44.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a51b02700 1 -- 172.21.15.26:0/11365980 >> v1:172.21.15.120:6789/0 conn(0x7f9a4c13dd40 legacy=0x7f9a4c13e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.187+0000 7f9a51b02700 1 -- 172.21.15.26:0/11365980 >> v1:172.21.15.26:6789/0 conn(0x7f9a4c1449d0 legacy=0x7f9a4c144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a51b02700 1 -- 172.21.15.26:0/11365980 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c144530 con 0x7f9a4c065980 2022-07-20T02:34:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a51b02700 1 --2- 172.21.15.26:0/11365980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c065980 0x7f9a4c065d60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f9a4c065590 tx=0x7f9a3c006df0).ready entity=mon.1 client_cookie=f4deb49f3ed1a1eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a48001660 con 0x7f9a4c065980 2022-07-20T02:34:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c000e90 con 0x7f9a4c065980 2022-07-20T02:34:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a50b00700 1 -- 172.21.15.26:0/11365980 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a3c012460 con 0x7f9a4c065980 2022-07-20T02:34:44.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- 172.21.15.26:0/11365980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c065980 msgr2=0x7f9a4c065d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 --2- 172.21.15.26:0/11365980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c065980 0x7f9a4c065d60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f9a4c065590 tx=0x7f9a3c006df0).stop 2022-07-20T02:34:44.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- 172.21.15.26:0/11365980 shutdown_connections 2022-07-20T02:34:44.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 --2- 172.21.15.26:0/11365980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c065980 0x7f9a4c065d60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- 172.21.15.26:0/11365980 >> 172.21.15.26:0/11365980 conn(0x7f9a4c05c580 msgr2=0x7f9a4c05c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- 172.21.15.26:0/11365980 shutdown_connections 2022-07-20T02:34:44.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- 172.21.15.26:0/11365980 wait complete. 2022-07-20T02:34:44.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 Processor -- start 2022-07-20T02:34:44.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.191+0000 7f9a53d66700 1 -- start start 2022-07-20T02:34:44.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 0x7f9a4c0b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c1449d0 0x7f9a4c143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c142be0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0aea50 con 0x7f9a4c1449d0 2022-07-20T02:34:44.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0aecf0 con 0x7f9a4c065980 2022-07-20T02:34:44.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43068/0 (socket says 172.21.15.26:43068) 2022-07-20T02:34:44.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 -- 172.21.15.26:0/2629286600 learned_addr learned my addr 172.21.15.26:0/2629286600 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:44.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a52303700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c1449d0 0x7f9a4c143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51b02700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 0x7f9a4c0b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 -- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 msgr2=0x7f9a4c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 0x7f9a4c0b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 -- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c1449d0 msgr2=0x7f9a4c143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c1449d0 0x7f9a4c143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 -- 172.21.15.26:0/2629286600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c00b180 con 0x7f9a4c13dd40 2022-07-20T02:34:44.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51b02700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 0x7f9a4c0b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:44.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a51301700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f9a48001840 tx=0x7f9a480016c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a48007170 con 0x7f9a4c13dd40 2022-07-20T02:34:44.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a480072d0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c142e10 con 0x7f9a4c13dd40 2022-07-20T02:34:44.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a48004710 con 0x7f9a4c13dd40 2022-07-20T02:34:44.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.195+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c1424a0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c142fa0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a4801b070 con 0x7f9a4c13dd40 2022-07-20T02:34:44.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a3affd700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 0x7f9a2c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9a4806b560 con 0x7f9a4c13dd40 2022-07-20T02:34:44.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a51b02700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 0x7f9a2c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.199+0000 7f9a51b02700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 0x7f9a2c055f60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f9a3c018000 tx=0x7f9a3c014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.207+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a48042320 con 0x7f9a4c13dd40 2022-07-20T02:34:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.335+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9a4c0574e0 con 0x7f9a4c13dd40 2022-07-20T02:34:44.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.339+0000 7f9a3affd700 1 -- 172.21.15.26:0/2629286600 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9a48045940 con 0x7f9a4c13dd40 2022-07-20T02:34:44.384 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:34:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 msgr2=0x7f9a2c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 0x7f9a2c055f60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f9a3c018000 tx=0x7f9a3c014000).stop 2022-07-20T02:34:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 msgr2=0x7f9a4c1435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f9a48001840 tx=0x7f9a480016c0).stop 2022-07-20T02:34:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 shutdown_connections 2022-07-20T02:34:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a4c13dd40 0x7f9a4c1435d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9a2c053aa0 0x7f9a2c055f60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a4c1449d0 0x7f9a4c143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 --2- 172.21.15.26:0/2629286600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a4c065980 0x7f9a4c0b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 >> 172.21.15.26:0/2629286600 conn(0x7f9a4c05c580 msgr2=0x7f9a4c13fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 shutdown_connections 2022-07-20T02:34:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.343+0000 7f9a53d66700 1 -- 172.21.15.26:0/2629286600 wait complete. 2022-07-20T02:34:44.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:34:44.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.527+0000 7f30dedd3700 1 Processor -- start 2022-07-20T02:34:44.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.527+0000 7f30dedd3700 1 -- start start 2022-07-20T02:34:44.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f30d80618a0 0x7f30d8147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d8061c80 con 0x7f30d80b22d0 2022-07-20T02:34:44.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d80b2810 con 0x7f30d80618a0 2022-07-20T02:34:44.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80b2ab0 con 0x7f30d8143f80 2022-07-20T02:34:44.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f30d80618a0 0x7f30d8147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30de5d2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dd5d0700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f30d80b22d0 0x7f30d8143870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56570/0 (socket says 172.21.15.26:56570) 2022-07-20T02:34:44.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dd5d0700 1 -- 172.21.15.26:0/3699979396 learned_addr learned my addr 172.21.15.26:0/3699979396 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:44.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 --2- 172.21.15.26:0/3699979396 >> v2:172.21.15.26:3300/0 conn(0x7f30d80618a0 0x7f30d8147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43076/0 (socket says 172.21.15.26:43076) 2022-07-20T02:34:44.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dcdcf700 1 -- 172.21.15.26:0/3699979396 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624975331 0 0) 0x7f30d8061c80 con 0x7f30d80b22d0 2022-07-20T02:34:44.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 -- 172.21.15.26:0/3699979396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 msgr2=0x7f30d80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:44.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 -- 172.21.15.26:0/3699979396 >> v1:172.21.15.120:6789/0 conn(0x7f30d80b22d0 legacy=0x7f30d8143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.587 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 -- 172.21.15.26:0/3699979396 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80b2d50 con 0x7f30d80618a0 2022-07-20T02:34:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30de5d2700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dddd1700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d8147250 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f30d4002f70 tx=0x7f30d4006df0).ready entity=mon.0 client_cookie=12225b9952ce5b20 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dcdcf700 1 -- 172.21.15.26:0/3699979396 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d400d570 con 0x7f30d80618a0 2022-07-20T02:34:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dcdcf700 1 -- 172.21.15.26:0/3699979396 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d4000e90 con 0x7f30d80618a0 2022-07-20T02:34:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dcdcf700 1 -- 172.21.15.26:0/3699979396 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d400d8c0 con 0x7f30d80618a0 2022-07-20T02:34:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- 172.21.15.26:0/3699979396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 msgr2=0x7f30d8147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d8147250 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f30d4002f70 tx=0x7f30d4006df0).stop 2022-07-20T02:34:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- 172.21.15.26:0/3699979396 shutdown_connections 2022-07-20T02:34:44.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 --2- 172.21.15.26:0/3699979396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d8147250 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- 172.21.15.26:0/3699979396 >> 172.21.15.26:0/3699979396 conn(0x7f30d8066f70 msgr2=0x7f30d8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- 172.21.15.26:0/3699979396 shutdown_connections 2022-07-20T02:34:44.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.531+0000 7f30dedd3700 1 -- 172.21.15.26:0/3699979396 wait complete. 2022-07-20T02:34:44.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 Processor -- start 2022-07-20T02:34:44.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 -- start start 2022-07-20T02:34:44.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 0x7f30d80af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80b0080 con 0x7f30d80618a0 2022-07-20T02:34:44.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80b02e0 con 0x7f30d80b22d0 2022-07-20T02:34:44.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dedd3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d813eea0 con 0x7f30d8143f80 2022-07-20T02:34:44.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43082/0 (socket says 172.21.15.26:43082) 2022-07-20T02:34:44.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 -- 172.21.15.26:0/2878330700 learned_addr learned my addr 172.21.15.26:0/2878330700 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:44.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30de5d2700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dd5d0700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 0x7f30d80af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 -- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 msgr2=0x7f30d80ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 -- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 msgr2=0x7f30d80af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 0x7f30d80af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dddd1700 1 -- 172.21.15.26:0/2878330700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d0009470 con 0x7f30d80618a0 2022-07-20T02:34:44.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.535+0000 7f30dd5d0700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 0x7f30d80af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:44.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dddd1700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f30d4002f70 tx=0x7f30d4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d400dea0 con 0x7f30d80618a0 2022-07-20T02:34:44.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d4013e90 con 0x7f30d80618a0 2022-07-20T02:34:44.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d401e5f0 con 0x7f30d80618a0 2022-07-20T02:34:44.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d400b180 con 0x7f30d80618a0 2022-07-20T02:34:44.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d813f3e0 con 0x7f30d80618a0 2022-07-20T02:34:44.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d80574e0 con 0x7f30d80618a0 2022-07-20T02:34:44.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f30d401c070 con 0x7f30d80618a0 2022-07-20T02:34:44.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f30d406cb10 con 0x7f30d80618a0 2022-07-20T02:34:44.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dd5d0700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.539+0000 7f30dd5d0700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 0x7f30b8055f60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f30cc00d000 tx=0x7f30cc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.547+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f30d40437c0 con 0x7f30d80618a0 2022-07-20T02:34:44.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.671+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f30d80b66d0 con 0x7f30d80618a0 2022-07-20T02:34:44.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30caffd700 1 -- 172.21.15.26:0/2878330700 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7f30d4023070 con 0x7f30d80618a0 2022-07-20T02:34:44.684 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:44.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 msgr2=0x7f30b8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 0x7f30b8055f60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f30cc00d000 tx=0x7f30cc009040).stop 2022-07-20T02:34:44.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 msgr2=0x7f30d81420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f30d4002f70 tx=0x7f30d4014000).stop 2022-07-20T02:34:44.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 shutdown_connections 2022-07-20T02:34:44.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d8143f80 0x7f30d80ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d80b22d0 0x7f30d80af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 --2- 172.21.15.26:0/2878330700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d80618a0 0x7f30d81420c0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 >> 172.21.15.26:0/2878330700 conn(0x7f30d8066f70 msgr2=0x7f30d80b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 shutdown_connections 2022-07-20T02:34:44.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.675+0000 7f30dedd3700 1 -- 172.21.15.26:0/2878330700 wait complete. 2022-07-20T02:34:44.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:34:44.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.835+0000 7fc4f9c76700 1 Processor -- start 2022-07-20T02:34:44.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.835+0000 7fc4f9c76700 1 -- start start 2022-07-20T02:34:44.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 0x7fc4f405c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc4f40ac8e0 0x7fc4f4143340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f40b1090 con 0x7fc4f40b06d0 2022-07-20T02:34:44.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f41406f0 con 0x7fc4f40a9520 2022-07-20T02:34:44.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40b0bd0 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f8c74700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 0x7fc4f405c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc4f40ac8e0 0x7fc4f4143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc4f40ac8e0 0x7fc4f4143340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:34552/0 (socket says 172.21.15.26:34552) 2022-07-20T02:34:44.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 -- 172.21.15.26:0/2791655651 learned_addr learned my addr 172.21.15.26:0/2791655651 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:44.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f8c74700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 0x7fc4f405c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43090/0 (socket says 172.21.15.26:43090) 2022-07-20T02:34:44.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4ebfff700 1 --1- 172.21.15.26:0/2791655651 >> v1:172.21.15.120:6789/0 conn(0x7fc4f40b06d0 0x7fc4f4142c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56586/0 (socket says 172.21.15.26:56586) 2022-07-20T02:34:44.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 -- 172.21.15.26:0/2791655651 >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 msgr2=0x7fc4f405c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:44.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 --2- 172.21.15.26:0/2791655651 >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 0x7fc4f405c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 -- 172.21.15.26:0/2791655651 >> v1:172.21.15.120:6789/0 conn(0x7fc4f40b06d0 legacy=0x7fc4f4142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 -- 172.21.15.26:0/2791655651 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40b12c0 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9475700 1 --2- 172.21.15.26:0/2791655651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f4143340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fc4ec0032f0 tx=0x7fc4ec0090f0).ready entity=mon.1 client_cookie=e3e1c45b66c10f30 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4eb7fe700 1 -- 172.21.15.26:0/2791655651 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4ec00cc60 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4eb7fe700 1 -- 172.21.15.26:0/2791655651 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec00cdc0 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4eb7fe700 1 -- 172.21.15.26:0/2791655651 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4ec00f650 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2791655651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40ac8e0 msgr2=0x7fc4f4143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2791655651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f4143340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fc4ec0032f0 tx=0x7fc4ec0090f0).stop 2022-07-20T02:34:44.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2791655651 shutdown_connections 2022-07-20T02:34:44.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2791655651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f4143340 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2791655651 >> v2:172.21.15.26:3300/0 conn(0x7fc4f40a9520 0x7fc4f405c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2791655651 >> 172.21.15.26:0/2791655651 conn(0x7fc4f4060310 msgr2=0x7fc4f40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:44.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2791655651 shutdown_connections 2022-07-20T02:34:44.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.839+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2791655651 wait complete. 2022-07-20T02:34:44.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 Processor -- start 2022-07-20T02:34:44.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 -- start start 2022-07-20T02:34:44.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 0x7fc4f413f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4f40b06d0 0x7fc4f413d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f413da70 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40657f0 con 0x7fc4f40a9520 2022-07-20T02:34:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9c76700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f413f5f0 con 0x7fc4f40b06d0 2022-07-20T02:34:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43098/0 (socket says 172.21.15.26:43098) 2022-07-20T02:34:44.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 -- 172.21.15.26:0/2930508306 learned_addr learned my addr 172.21.15.26:0/2930508306 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:44.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f8c74700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 0x7fc4f413f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f9475700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4f40b06d0 0x7fc4f413d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 -- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4f40b06d0 msgr2=0x7fc4f413d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4f40b06d0 0x7fc4f413d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 -- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 msgr2=0x7fc4f413f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 0x7fc4f413f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:44.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 -- 172.21.15.26:0/2930508306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec00b5e0 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4f8c74700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 0x7fc4f413f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:44.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.843+0000 7fc4ebfff700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc4f40b0af0 tx=0x7fc4e4007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4e40070d0 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4007230 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f414e050 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4e4005550 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f414dd60 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f413fe60 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc4e4010070 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 0x7fc4d4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:44.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4f8c74700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 0x7fc4d4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:44.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc4e406b660 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.847+0000 7fc4f8c74700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 0x7fc4d4055e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fc4f000f000 tx=0x7fc4f000f040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:44.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.855+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4e4042310 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.983+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4f413f730 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.983+0000 7fc4e9ffb700 1 -- 172.21.15.26:0/2930508306 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc4e4045930 con 0x7fc4f40ac8e0 2022-07-20T02:34:44.992 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:44.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 msgr2=0x7fc4d4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 0x7fc4d4055e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fc4f000f000 tx=0x7fc4f000f040).stop 2022-07-20T02:34:44.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 msgr2=0x7fc4f40b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:44.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc4f40b0af0 tx=0x7fc4e4007c40).stop 2022-07-20T02:34:44.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 shutdown_connections 2022-07-20T02:34:44.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc4f40ac8e0 0x7fc4f40b4370 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc4d4053950 0x7fc4d4055e10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4f40b06d0 0x7fc4f413d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 --2- 172.21.15.26:0/2930508306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f40a9520 0x7fc4f413f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 >> 172.21.15.26:0/2930508306 conn(0x7fc4f4060310 msgr2=0x7fc4f40ab440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:45.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 shutdown_connections 2022-07-20T02:34:45.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:44.987+0000 7fc4f9c76700 1 -- 172.21.15.26:0/2930508306 wait complete. 2022-07-20T02:34:45.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:34:45.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 Processor -- start 2022-07-20T02:34:45.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 -- start start 2022-07-20T02:34:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02d4067240 0x7f02d40b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40657b0 0x7f02d4144f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40b14b0 con 0x7f02d40b2ca0 2022-07-20T02:34:45.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d4067620 con 0x7f02d4067240 2022-07-20T02:34:45.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02db31c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40aecf0 con 0x7f02d40657b0 2022-07-20T02:34:45.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02da31a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02d4067240 0x7f02d40b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02da31a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02d4067240 0x7f02d40b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43104/0 (socket says 172.21.15.26:43104) 2022-07-20T02:34:45.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02da31a700 1 -- 172.21.15.26:0/3520563436 learned_addr learned my addr 172.21.15.26:0/3520563436 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:45.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.123+0000 7f02dab1b700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40657b0 0x7f02d4144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02da31a700 1 -- 172.21.15.26:0/3520563436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40657b0 msgr2=0x7f02d4144f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02da31a700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40657b0 0x7f02d4144f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02da31a700 1 -- 172.21.15.26:0/3520563436 >> v1:172.21.15.120:6789/0 conn(0x7f02d40b2ca0 legacy=0x7f02d4144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02da31a700 1 -- 172.21.15.26:0/3520563436 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40b1980 con 0x7f02d4067240 2022-07-20T02:34:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02da31a700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d4067240 0x7f02d40b3700 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f02d0002f70 tx=0x7f02d0006eb0).ready entity=mon.0 client_cookie=27a0b7efb29565ad server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02d9318700 1 -- 172.21.15.26:0/3520563436 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d000d5c0 con 0x7f02d4067240 2022-07-20T02:34:45.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02d9318700 1 -- 172.21.15.26:0/3520563436 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d0000e90 con 0x7f02d4067240 2022-07-20T02:34:45.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02d9318700 1 -- 172.21.15.26:0/3520563436 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d000d950 con 0x7f02d4067240 2022-07-20T02:34:45.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 -- 172.21.15.26:0/3520563436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d4067240 msgr2=0x7f02d40b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d4067240 0x7f02d40b3700 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f02d0002f70 tx=0x7f02d0006eb0).stop 2022-07-20T02:34:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 -- 172.21.15.26:0/3520563436 shutdown_connections 2022-07-20T02:34:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40657b0 0x7f02d4144f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 --2- 172.21.15.26:0/3520563436 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d4067240 0x7f02d40b3700 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 -- 172.21.15.26:0/3520563436 >> 172.21.15.26:0/3520563436 conn(0x7f02d405f840 msgr2=0x7f02d405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 -- 172.21.15.26:0/3520563436 shutdown_connections 2022-07-20T02:34:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.127+0000 7f02db31c700 1 -- 172.21.15.26:0/3520563436 wait complete. 2022-07-20T02:34:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 Processor -- start 2022-07-20T02:34:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- start start 2022-07-20T02:34:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d4067240 0x7f02d4154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40b2ca0 0x7f02d4150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40ab980 con 0x7f02d40657b0 2022-07-20T02:34:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4154940 con 0x7f02d4067240 2022-07-20T02:34:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40abbe0 con 0x7f02d40b2ca0 2022-07-20T02:34:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43110/0 (socket says 172.21.15.26:43110) 2022-07-20T02:34:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 -- 172.21.15.26:0/214785466 learned_addr learned my addr 172.21.15.26:0/214785466 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02d9b19700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d4067240 0x7f02d4154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02dab1b700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40b2ca0 0x7f02d4150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 -- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40b2ca0 msgr2=0x7f02d4150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40b2ca0 0x7f02d4150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 -- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d4067240 msgr2=0x7f02d4154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d4067240 0x7f02d4154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 -- 172.21.15.26:0/214785466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d000b1e0 con 0x7f02d40657b0 2022-07-20T02:34:45.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02da31a700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f02d0002f70 tx=0x7f02d0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d0013de0 con 0x7f02d40657b0 2022-07-20T02:34:45.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d001e400 con 0x7f02d40657b0 2022-07-20T02:34:45.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d001e760 con 0x7f02d40657b0 2022-07-20T02:34:45.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d40abe80 con 0x7f02d40657b0 2022-07-20T02:34:45.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.131+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d40ad310 con 0x7f02d40657b0 2022-07-20T02:34:45.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d413d410 con 0x7f02d40657b0 2022-07-20T02:34:45.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f02d0025030 con 0x7f02d40657b0 2022-07-20T02:34:45.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02c77fe700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 0x7f02b4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f02d006cf90 con 0x7f02d40657b0 2022-07-20T02:34:45.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02d9b19700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 0x7f02b4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.135+0000 7f02d9b19700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 0x7f02b4055ea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f02c800d000 tx=0x7f02c8007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.139+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02d413d410 con 0x7f02d40657b0 2022-07-20T02:34:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.271+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02d413d410 con 0x7f02d40657b0 2022-07-20T02:34:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.271+0000 7f02c77fe700 1 -- 172.21.15.26:0/214785466 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f02d413d410 con 0x7f02d40657b0 2022-07-20T02:34:45.280 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:45.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 msgr2=0x7f02b4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 0x7f02b4055ea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f02c800d000 tx=0x7f02c8007b90).stop 2022-07-20T02:34:45.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 msgr2=0x7f02d40b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f02d0002f70 tx=0x7f02d0014000).stop 2022-07-20T02:34:45.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 shutdown_connections 2022-07-20T02:34:45.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f02b40539e0 0x7f02b4055ea0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d40b2ca0 0x7f02d4150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d4067240 0x7f02d4154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 --2- 172.21.15.26:0/214785466 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02d40657b0 0x7f02d40b6b40 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 >> 172.21.15.26:0/214785466 conn(0x7f02d405f840 msgr2=0x7f02d40aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:45.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 shutdown_connections 2022-07-20T02:34:45.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.275+0000 7f02db31c700 1 -- 172.21.15.26:0/214785466 wait complete. 2022-07-20T02:34:45.303 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:34:45.303 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:45.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 Processor -- start 2022-07-20T02:34:45.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- start start 2022-07-20T02:34:45.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa40b3260 0x7fdaa4067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdaa4061860 0x7fdaa40979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa405ff70 con 0x7fdaa4097eb0 2022-07-20T02:34:45.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa40b3640 con 0x7fdaa4061860 2022-07-20T02:34:45.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa40601a0 con 0x7fdaa40b3260 2022-07-20T02:34:45.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdaa4061860 0x7fdaa40979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdaa4061860 0x7fdaa40979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43118/0 (socket says 172.21.15.26:43118) 2022-07-20T02:34:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 -- 172.21.15.26:0/7004097 learned_addr learned my addr 172.21.15.26:0/7004097 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:45.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa37fe700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa40b3260 0x7fdaa4067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 -- 172.21.15.26:0/7004097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa40b3260 msgr2=0x7fdaa4067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:45.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa40b3260 0x7fdaa4067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 -- 172.21.15.26:0/7004097 >> v1:172.21.15.120:6789/0 conn(0x7fdaa4097eb0 legacy=0x7fdaa40b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:45.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 -- 172.21.15.26:0/7004097 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40606e0 con 0x7fdaa4061860 2022-07-20T02:34:45.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa2ffd700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40979c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7fdaa405fb20 tx=0x7fda94009290).ready entity=mon.0 client_cookie=e04f31b45d573240 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa27fc700 1 -- 172.21.15.26:0/7004097 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9400b860 con 0x7fdaa4061860 2022-07-20T02:34:45.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa27fc700 1 -- 172.21.15.26:0/7004097 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9400b9c0 con 0x7fdaa4061860 2022-07-20T02:34:45.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa27fc700 1 -- 172.21.15.26:0/7004097 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9400bb70 con 0x7fdaa4061860 2022-07-20T02:34:45.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/7004097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 msgr2=0x7fdaa40979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40979c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7fdaa405fb20 tx=0x7fda94009290).stop 2022-07-20T02:34:45.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/7004097 shutdown_connections 2022-07-20T02:34:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40979c0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/7004097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa40b3260 0x7fdaa4067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/7004097 >> 172.21.15.26:0/7004097 conn(0x7fdaa40b67f0 msgr2=0x7fdaa40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/7004097 shutdown_connections 2022-07-20T02:34:45.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.419+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/7004097 wait complete. 2022-07-20T02:34:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 Processor -- start 2022-07-20T02:34:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- start start 2022-07-20T02:34:45.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa4097eb0 0x7fdaa409b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 0x7fdaa409b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa409bb50 con 0x7fdaa4061860 2022-07-20T02:34:45.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa40b4360 con 0x7fdaa40b3260 2022-07-20T02:34:45.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa4155f10 con 0x7fdaa4097eb0 2022-07-20T02:34:45.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43124/0 (socket says 172.21.15.26:43124) 2022-07-20T02:34:45.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 -- 172.21.15.26:0/889739087 learned_addr learned my addr 172.21.15.26:0/889739087 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:45.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa3fff700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 0x7fdaa409b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa2ffd700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa4097eb0 0x7fdaa409b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 -- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa4097eb0 msgr2=0x7fdaa409b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa4097eb0 0x7fdaa409b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 -- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 msgr2=0x7fdaa409b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 0x7fdaa409b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 -- 172.21.15.26:0/889739087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94009530 con 0x7fdaa4061860 2022-07-20T02:34:45.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa3fff700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 0x7fdaa409b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:45.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa37fe700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7fda8c0030b0 tx=0x7fda8c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda8c011780 con 0x7fdaa4061860 2022-07-20T02:34:45.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa40b0cd0 con 0x7fdaa4061860 2022-07-20T02:34:45.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c0118e0 con 0x7fdaa4061860 2022-07-20T02:34:45.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa40b1210 con 0x7fdaa4061860 2022-07-20T02:34:45.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.423+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda8c011c40 con 0x7fdaa4061860 2022-07-20T02:34:45.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa40b13d0 con 0x7fdaa4061860 2022-07-20T02:34:45.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fda8c012420 con 0x7fdaa4061860 2022-07-20T02:34:45.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa0ff9700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 0x7fda88055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda8c06c2c0 con 0x7fdaa4061860 2022-07-20T02:34:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa2ffd700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 0x7fda88055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.427+0000 7fdaa2ffd700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 0x7fda88055fc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fda94019000 tx=0x7fda94016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.431+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fda8c042f80 con 0x7fdaa4061860 2022-07-20T02:34:45.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.591+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa40574e0 con 0x7fdaa4061860 2022-07-20T02:34:45.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.591+0000 7fdaa0ff9700 1 -- 172.21.15.26:0/889739087 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7fda8c0465a0 con 0x7fdaa4061860 2022-07-20T02:34:45.610 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:45.611 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:34:45.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 msgr2=0x7fda88055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 0x7fda88055fc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fda94019000 tx=0x7fda94016000).stop 2022-07-20T02:34:45.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 msgr2=0x7fdaa40b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:45.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7fda8c0030b0 tx=0x7fda8c008040).stop 2022-07-20T02:34:45.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 shutdown_connections 2022-07-20T02:34:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fda88053b00 0x7fda88055fc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa40b3260 0x7fdaa409b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaa4097eb0 0x7fdaa409b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 --2- 172.21.15.26:0/889739087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdaa4061860 0x7fdaa40b3cb0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 >> 172.21.15.26:0/889739087 conn(0x7fdaa40b67f0 msgr2=0x7fdaa4095b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 shutdown_connections 2022-07-20T02:34:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:45.595+0000 7fdaa8b7e700 1 -- 172.21.15.26:0/889739087 wait complete. 2022-07-20T02:34:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:34:45.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:34:40.117843+0000', 'modified': '2022-07-20T02:34:41.123008+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8139': {'gid': 8139, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1171289829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 1171289829}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1171289829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:34:45.939 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:34:45.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:34:46.071 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8139,"want_state":"up:active","state":"up:active","rank_uptime":5.9301571370000001,"mdsmap_epoch":72,"osdmap_epoch":140,"osdmap_epoch_barrier":138,"uptime":10.990531234000001} 2022-07-20T02:34:46.072 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8139, "mdsmap_epoch": 72, "osdmap_epoch": 140, "osdmap_epoch_barrier": 138, "rank_uptime": 5.930157137, "state": "up:active", "uptime": 10.990531234, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:34:46.074 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:34:46.074 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:34:46.093 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:34:46.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.105 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:46.137 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:34:46.227 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:46.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:46.284 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:46.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.286 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:34:46.339 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:34:46.340 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:46.398 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:34:46.418 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:46.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:46.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:34:46.465+0000 7fa8c6dba080 -1 init, newargv = 0x561cc3f90900 newargc=9ceph-fuse[23351]: starting ceph client 2022-07-20T02:34:46.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:34:46.477 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:46.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.482 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:34:46.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23351]: starting fuse 2022-07-20T02:34:46.572 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:34:46.573 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:34:46.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:34:46.595 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:34:46.596 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:34:46.596 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:34:46.597 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:34:46.598 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:34:46.604 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:34:46.605 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:34:46.606 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:34:46.606 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:34:46.624 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:34:46.637 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:34:46.638 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:34:46.639 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:34:46.640 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:34:46.665 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:34:46.666 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:34:46.666 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:34:46.669 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:34:46.670 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:34:46.670 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:34:46.671 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:34:46.678 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:34:46.741 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23351.asok 2022-07-20T02:34:46.742 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23351.asok 2022-07-20T02:34:46.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23351.asok status 2022-07-20T02:34:46.884 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:34:46.890 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:34:46.892 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:34:46.893 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:34:46.893 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:34:46.894 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:34:46.894 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:34:46.895 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23351", 2022-07-20T02:34:46.895 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:34:46.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:34:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:34:46.902 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:34:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 7028, 2022-07-20T02:34:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:34:46.903 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:34:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:34:46.904 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 7028 2022-07-20T02:34:46.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:34:46.911 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:34:46.911 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:34:46.924 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:34:46.925 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 148260855 2022-07-20T02:34:46.925 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:34:46.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:34:46.926 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:34:46.927 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:34:46.927 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:34:46.928 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 148260855 2022-07-20T02:34:46.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:34:46.928 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.7028 172.21.15.159:0/148260855", 2022-07-20T02:34:46.929 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/148260855", 2022-07-20T02:34:46.930 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:34:46.930 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 72, 2022-07-20T02:34:46.931 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 140, 2022-07-20T02:34:46.932 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:34:46.932 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:34:46.932 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:34:46.933 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:34:46.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:46.939 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:46.986 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:47.012 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:34:47.012 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:47.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:47.026 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:47.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:47.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 Processor -- start 2022-07-20T02:34:47.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 -- start start 2022-07-20T02:34:47.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 0x7f3c001474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c000a7fc0 0x7f3c001479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c000b62d0 con 0x7f3c000a8c40 2022-07-20T02:34:47.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c00147eb0 con 0x7f3c000a7fc0 2022-07-20T02:34:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.231+0000 7f3c07341700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0014aac0 con 0x7f3c000b5ef0 2022-07-20T02:34:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c000a7fc0 0x7f3c001479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f3c000a7fc0 0x7f3c001479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43132/0 (socket says 172.21.15.26:43132) 2022-07-20T02:34:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 -- 172.21.15.26:0/101704791 learned_addr learned my addr 172.21.15.26:0/101704791 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:47.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c0633f700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 0x7f3c001474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 -- 172.21.15.26:0/101704791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 msgr2=0x7f3c001474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 0x7f3c001474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 -- 172.21.15.26:0/101704791 >> v1:172.21.15.120:6789/0 conn(0x7f3c000a8c40 legacy=0x7f3c000aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 -- 172.21.15.26:0/101704791 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0014ad60 con 0x7f3c000a7fc0 2022-07-20T02:34:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c0633f700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 0x7f3c001474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:34:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c05b3e700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a7fc0 0x7f3c001479c0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f3bfc0032f0 tx=0x7f3bfc009290).ready entity=mon.0 client_cookie=99681b441e0e5834 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c0533d700 1 -- 172.21.15.26:0/101704791 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bfc00b8a0 con 0x7f3c000a7fc0 2022-07-20T02:34:47.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c0533d700 1 -- 172.21.15.26:0/101704791 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc00ba00 con 0x7f3c000a7fc0 2022-07-20T02:34:47.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c0533d700 1 -- 172.21.15.26:0/101704791 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bfc00bb70 con 0x7f3c000a7fc0 2022-07-20T02:34:47.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- 172.21.15.26:0/101704791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a7fc0 msgr2=0x7f3c001479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a7fc0 0x7f3c001479c0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f3bfc0032f0 tx=0x7f3bfc009290).stop 2022-07-20T02:34:47.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- 172.21.15.26:0/101704791 shutdown_connections 2022-07-20T02:34:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a7fc0 0x7f3c001479c0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 --2- 172.21.15.26:0/101704791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000b5ef0 0x7f3c001474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- 172.21.15.26:0/101704791 >> 172.21.15.26:0/101704791 conn(0x7f3c00060310 msgr2=0x7f3c000b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:47.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- 172.21.15.26:0/101704791 shutdown_connections 2022-07-20T02:34:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- 172.21.15.26:0/101704791 wait complete. 2022-07-20T02:34:47.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 Processor -- start 2022-07-20T02:34:47.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.235+0000 7f3c07341700 1 -- start start 2022-07-20T02:34:47.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000a7fc0 0x7f3c0015a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c000b5ef0 0x7f3c001565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00146910 con 0x7f3c000a8c40 2022-07-20T02:34:47.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00159ba0 con 0x7f3c000b5ef0 2022-07-20T02:34:47.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00062db0 con 0x7f3c000a7fc0 2022-07-20T02:34:47.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43140/0 (socket says 172.21.15.26:43140) 2022-07-20T02:34:47.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 -- 172.21.15.26:0/1865767867 learned_addr learned my addr 172.21.15.26:0/1865767867 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:47.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c06b40700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c000b5ef0 0x7f3c001565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c0633f700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000a7fc0 0x7f3c0015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 -- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000a7fc0 msgr2=0x7f3c0015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000a7fc0 0x7f3c0015a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 -- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c000b5ef0 msgr2=0x7f3c001565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c000b5ef0 0x7f3c001565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 -- 172.21.15.26:0/1865767867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009530 con 0x7f3c000a8c40 2022-07-20T02:34:47.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c05b3e700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3bfc0032f0 tx=0x7f3bfc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bfc021440 con 0x7f3c000a8c40 2022-07-20T02:34:47.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0215a0 con 0x7f3c000a8c40 2022-07-20T02:34:47.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c000638c0 con 0x7f3c000a8c40 2022-07-20T02:34:47.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bfc021900 con 0x7f3c000a8c40 2022-07-20T02:34:47.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.239+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0015a070 con 0x7f3c000a8c40 2022-07-20T02:34:47.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0015a230 con 0x7f3c000a8c40 2022-07-20T02:34:47.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3bfc029070 con 0x7f3c000a8c40 2022-07-20T02:34:47.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3bef7fe700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 0x7f3be0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3bfc06d660 con 0x7f3c000a8c40 2022-07-20T02:34:47.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3c0633f700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 0x7f3be0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.243+0000 7f3c0633f700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 0x7f3be0055ea0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f3bf000d000 tx=0x7f3bf0008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.251+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3bfc044380 con 0x7f3c000a8c40 2022-07-20T02:34:47.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c0014b000 con 0x7f3c000a8c40 2022-07-20T02:34:47.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3bef7fe700 1 -- 172.21.15.26:0/1865767867 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f3bfc015070 con 0x7f3c000a8c40 2022-07-20T02:34:47.398 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:47.399 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":140,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:42.132316+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 msgr2=0x7f3be0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 0x7f3be0055ea0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f3bf000d000 tx=0x7f3bf0008040).stop 2022-07-20T02:34:47.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 msgr2=0x7f3c0005c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3bfc0032f0 tx=0x7f3bfc016000).stop 2022-07-20T02:34:47.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 shutdown_connections 2022-07-20T02:34:47.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c000a8c40 0x7f3c0005c8b0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3be00539e0 0x7f3be0055ea0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c000b5ef0 0x7f3c001565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 --2- 172.21.15.26:0/1865767867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c000a7fc0 0x7f3c0015a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 >> 172.21.15.26:0/1865767867 conn(0x7f3c00060310 msgr2=0x7f3c00148ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:47.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 shutdown_connections 2022-07-20T02:34:47.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.375+0000 7f3c07341700 1 -- 172.21.15.26:0/1865767867 wait complete. 2022-07-20T02:34:47.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:34:38.295386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:34:39.171843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:34:47.486 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:34:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 Processor -- start 2022-07-20T02:34:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 -- start start 2022-07-20T02:34:47.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc450147000 0x7fc450143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc450060f00 0x7fc45013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4501474e0 con 0x7fc45005cfd0 2022-07-20T02:34:47.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4500b4e40 con 0x7fc450060f00 2022-07-20T02:34:47.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45716f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45005d3b0 con 0x7fc450147000 2022-07-20T02:34:47.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc450060f00 0x7fc45013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45616d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc45005cfd0 0x7fc4500b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42426/0 (socket says 172.21.15.26:42426) 2022-07-20T02:34:47.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45616d700 1 -- 172.21.15.26:0/1646929646 learned_addr learned my addr 172.21.15.26:0/1646929646 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:47.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45596c700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc450147000 0x7fc450143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 -- 172.21.15.26:0/1646929646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc450147000 msgr2=0x7fc450143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:47.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc450147000 0x7fc450143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 -- 172.21.15.26:0/1646929646 >> v1:172.21.15.26:6789/0 conn(0x7fc45005cfd0 legacy=0x7fc4500b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 -- 172.21.15.26:0/1646929646 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4501431e0 con 0x7fc450060f00 2022-07-20T02:34:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.607+0000 7fc45696e700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc45013c9b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fc4480032f0 tx=0x7fc4480091d0).ready entity=mon.0 client_cookie=17e60e39a1c824ef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45516b700 1 -- 172.21.15.26:0/1646929646 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44800b6c0 con 0x7fc450060f00 2022-07-20T02:34:47.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45516b700 1 -- 172.21.15.26:0/1646929646 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800b820 con 0x7fc450060f00 2022-07-20T02:34:47.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45516b700 1 -- 172.21.15.26:0/1646929646 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44800b990 con 0x7fc450060f00 2022-07-20T02:34:47.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- 172.21.15.26:0/1646929646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 msgr2=0x7fc45013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc45013c9b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fc4480032f0 tx=0x7fc4480091d0).stop 2022-07-20T02:34:47.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- 172.21.15.26:0/1646929646 shutdown_connections 2022-07-20T02:34:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc45013c9b0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 --2- 172.21.15.26:0/1646929646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc450147000 0x7fc450143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- 172.21.15.26:0/1646929646 >> 172.21.15.26:0/1646929646 conn(0x7fc4500620c0 msgr2=0x7fc45005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:47.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- 172.21.15.26:0/1646929646 shutdown_connections 2022-07-20T02:34:47.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- 172.21.15.26:0/1646929646 wait complete. 2022-07-20T02:34:47.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 Processor -- start 2022-07-20T02:34:47.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.611+0000 7fc45716f700 1 -- start start 2022-07-20T02:34:47.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450147000 0x7fc450152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4500af150 0x7fc450152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45013d270 con 0x7fc450060f00 2022-07-20T02:34:47.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc450145ff0 con 0x7fc450147000 2022-07-20T02:34:47.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc450146290 con 0x7fc4500af150 2022-07-20T02:34:47.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43152/0 (socket says 172.21.15.26:43152) 2022-07-20T02:34:47.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 -- 172.21.15.26:0/3192714351 learned_addr learned my addr 172.21.15.26:0/3192714351 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:47.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45696e700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4500af150 0x7fc450152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45596c700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450147000 0x7fc450152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 -- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4500af150 msgr2=0x7fc450152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4500af150 0x7fc450152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 -- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450147000 msgr2=0x7fc450152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450147000 0x7fc450152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 -- 172.21.15.26:0/3192714351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448009470 con 0x7fc450060f00 2022-07-20T02:34:47.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45616d700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fc450068660 tx=0x7fc440007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4400070d0 con 0x7fc450060f00 2022-07-20T02:34:47.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc440007230 con 0x7fc450060f00 2022-07-20T02:34:47.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc450146530 con 0x7fc450060f00 2022-07-20T02:34:47.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc440005610 con 0x7fc450060f00 2022-07-20T02:34:47.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.615+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4500ad8a0 con 0x7fc450060f00 2022-07-20T02:34:47.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4501527d0 con 0x7fc450060f00 2022-07-20T02:34:47.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc440010070 con 0x7fc450060f00 2022-07-20T02:34:47.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc43f7fe700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 0x7fc430055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:47.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc44006b750 con 0x7fc450060f00 2022-07-20T02:34:47.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc45596c700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 0x7fc430055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:47.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.619+0000 7fc45596c700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 0x7fc430055f30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fc44c00b000 tx=0x7fc44c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:47.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.627+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4501527d0 con 0x7fc450060f00 2022-07-20T02:34:47.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.759+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc4501527d0 con 0x7fc450060f00 2022-07-20T02:34:47.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.759+0000 7fc43f7fe700 1 -- 172.21.15.26:0/3192714351 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc4501527d0 con 0x7fc450060f00 2022-07-20T02:34:47.764 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:34:47.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 msgr2=0x7fc430055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 0x7fc430055f30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fc44c00b000 tx=0x7fc44c006df0).stop 2022-07-20T02:34:47.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 msgr2=0x7fc4500b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:47.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fc450068660 tx=0x7fc440007ca0).stop 2022-07-20T02:34:47.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 shutdown_connections 2022-07-20T02:34:47.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc430053a70 0x7fc430055f30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc4500af150 0x7fc450152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450147000 0x7fc450152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 --2- 172.21.15.26:0/3192714351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc450060f00 0x7fc4500b8b50 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:47.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 >> 172.21.15.26:0/3192714351 conn(0x7fc45005cfd0 msgr2=0x7fc4500b79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:47.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 shutdown_connections 2022-07-20T02:34:47.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.763+0000 7fc45716f700 1 -- 172.21.15.26:0/3192714351 wait complete. 2022-07-20T02:34:47.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:34:47.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:48.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.911+0000 7fd79d8c1700 1 Processor -- start 2022-07-20T02:34:48.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.911+0000 7fd79d8c1700 1 -- start start 2022-07-20T02:34:48.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7980b9c80 0x7fd7981431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd7980b5b70 0x7fd7981403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980ba060 con 0x7fd798062710 2022-07-20T02:34:48.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79812bee0 con 0x7fd7980b5b70 2022-07-20T02:34:48.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd798061f80 con 0x7fd7980b9c80 2022-07-20T02:34:48.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7967fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd798062710 0x7fd7980b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42442/0 (socket says 172.21.15.26:42442) 2022-07-20T02:34:48.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd7980b5b70 0x7fd7981403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7967fc700 1 -- 172.21.15.26:0/3500571344 learned_addr learned my addr 172.21.15.26:0/3500571344 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:48.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd796ffd700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7980b9c80 0x7fd7981431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199636201 0 0) 0x7fd7980ba060 con 0x7fd798062710 2022-07-20T02:34:48.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c003260 con 0x7fd798062710 2022-07-20T02:34:48.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 -- 172.21.15.26:0/3500571344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7980b9c80 msgr2=0x7fd7981431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:48.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7980b9c80 0x7fd7981431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015600546 0 0) 0x7fd77c003260 con 0x7fd798062710 2022-07-20T02:34:48.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 -- 172.21.15.26:0/3500571344 >> v1:172.21.15.26:6789/0 conn(0x7fd798062710 legacy=0x7fd7980b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 -- 172.21.15.26:0/3500571344 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798062220 con 0x7fd7980b5b70 2022-07-20T02:34:48.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd7977fe700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981403e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7fd78c001800 tx=0x7fd7900091d0).ready entity=mon.0 client_cookie=85b2b9a59aabaf06 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd78c001660 con 0x7fd7980b5b70 2022-07-20T02:34:48.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000f510 con 0x7fd7980b5b70 2022-07-20T02:34:48.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd795ffb700 1 -- 172.21.15.26:0/3500571344 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79000f680 con 0x7fd7980b5b70 2022-07-20T02:34:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- 172.21.15.26:0/3500571344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 msgr2=0x7fd7981403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981403e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7fd78c001800 tx=0x7fd7900091d0).stop 2022-07-20T02:34:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- 172.21.15.26:0/3500571344 shutdown_connections 2022-07-20T02:34:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981403e0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/3500571344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7980b9c80 0x7fd7981431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- 172.21.15.26:0/3500571344 >> 172.21.15.26:0/3500571344 conn(0x7fd798066f70 msgr2=0x7fd79805faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- 172.21.15.26:0/3500571344 shutdown_connections 2022-07-20T02:34:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.915+0000 7fd79d8c1700 1 -- 172.21.15.26:0/3500571344 wait complete. 2022-07-20T02:34:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 Processor -- start 2022-07-20T02:34:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 -- start start 2022-07-20T02:34:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 0x7fd7980b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7980b9c80 0x7fd798155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79812c3d0 con 0x7fd7980b5b70 2022-07-20T02:34:48.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798156090 con 0x7fd7980b9c80 2022-07-20T02:34:48.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd79d8c1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd798152cd0 con 0x7fd798062710 2022-07-20T02:34:48.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd7967fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd796ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 0x7fd7980b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd7967fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43168/0 (socket says 172.21.15.26:43168) 2022-07-20T02:34:48.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd7967fc700 1 -- 172.21.15.26:0/2587670375 learned_addr learned my addr 172.21.15.26:0/2587670375 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:48.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd7977fe700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7980b9c80 0x7fd798155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.919+0000 7fd7967fc700 1 -- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 msgr2=0x7fd7980b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd7967fc700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 0x7fd7980b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd7967fc700 1 -- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7980b9c80 msgr2=0x7fd798155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd7967fc700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7980b9c80 0x7fd798155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd7967fc700 1 -- 172.21.15.26:0/2587670375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790009470 con 0x7fd7980b5b70 2022-07-20T02:34:48.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd796ffd700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 0x7fd7980b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:48.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd7967fc700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fd78c0018c0 tx=0x7fd78c001b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd78c007170 con 0x7fd7980b5b70 2022-07-20T02:34:48.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0072d0 con 0x7fd7980b5b70 2022-07-20T02:34:48.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd798152f70 con 0x7fd7980b5b70 2022-07-20T02:34:48.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd78c004920 con 0x7fd7980b5b70 2022-07-20T02:34:48.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7981474a0 con 0x7fd7980b5b70 2022-07-20T02:34:48.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7980b6650 con 0x7fd7980b5b70 2022-07-20T02:34:48.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd78c01b070 con 0x7fd7980b5b70 2022-07-20T02:34:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 0x7fd774055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd78c06b880 con 0x7fd7980b5b70 2022-07-20T02:34:48.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd796ffd700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 0x7fd774055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.923+0000 7fd796ffd700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 0x7fd774055ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fd78800b000 tx=0x7fd788006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:47.927+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd78c042640 con 0x7fd7980b5b70 2022-07-20T02:34:48.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.051+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd7980574e0 con 0x7fd7980b5b70 2022-07-20T02:34:48.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.051+0000 7fd787fff700 1 -- 172.21.15.26:0/2587670375 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd78c045c60 con 0x7fd7980b5b70 2022-07-20T02:34:48.247 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:48.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 msgr2=0x7fd774055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 0x7fd774055ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fd78800b000 tx=0x7fd788006df0).stop 2022-07-20T02:34:48.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 msgr2=0x7fd7981556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fd78c0018c0 tx=0x7fd78c001b60).stop 2022-07-20T02:34:48.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 shutdown_connections 2022-07-20T02:34:48.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd7980b5b70 0x7fd7981556b0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd774053b30 0x7fd774055ff0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7980b9c80 0x7fd798155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 --2- 172.21.15.26:0/2587670375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd798062710 0x7fd7980b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 >> 172.21.15.26:0/2587670375 conn(0x7fd798066f70 msgr2=0x7fd798142f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 shutdown_connections 2022-07-20T02:34:48.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.055+0000 7fd79d8c1700 1 -- 172.21.15.26:0/2587670375 wait complete. 2022-07-20T02:34:48.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:34:48.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:48.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:48.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:34:48.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.483+0000 7fc839767700 1 Processor -- start 2022-07-20T02:34:48.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.483+0000 7fc839767700 1 -- start start 2022-07-20T02:34:48.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340957f0 0x7fc8340b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc834067440 0x7fc8341433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc834147080 con 0x7fc8340b2f20 2022-07-20T02:34:48.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc834143000 con 0x7fc834067440 2022-07-20T02:34:48.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341472b0 con 0x7fc8340957f0 2022-07-20T02:34:48.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc834067440 0x7fc8341433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc834067440 0x7fc8341433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43178/0 (socket says 172.21.15.26:43178) 2022-07-20T02:34:48.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 -- 172.21.15.26:0/1993234661 learned_addr learned my addr 172.21.15.26:0/1993234661 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:48.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc8337fe700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340957f0 0x7fc8340b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 -- 172.21.15.26:0/1993234661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340957f0 msgr2=0x7fc8340b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:48.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340957f0 0x7fc8340b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 -- 172.21.15.26:0/1993234661 >> v1:172.21.15.120:6789/0 conn(0x7fc8340b2f20 legacy=0x7fc8340950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 -- 172.21.15.26:0/1993234661 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340a9920 con 0x7fc834067440 2022-07-20T02:34:48.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc838f66700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8341433f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fc828002f70 tx=0x7fc828006eb0).ready entity=mon.0 client_cookie=7427d932f7aba122 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc832ffd700 1 -- 172.21.15.26:0/1993234661 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d570 con 0x7fc834067440 2022-07-20T02:34:48.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc832ffd700 1 -- 172.21.15.26:0/1993234661 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc828000e90 con 0x7fc834067440 2022-07-20T02:34:48.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc832ffd700 1 -- 172.21.15.26:0/1993234661 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d900 con 0x7fc834067440 2022-07-20T02:34:48.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 -- 172.21.15.26:0/1993234661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 msgr2=0x7fc8341433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.487+0000 7fc839767700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8341433f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fc828002f70 tx=0x7fc828006eb0).stop 2022-07-20T02:34:48.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- 172.21.15.26:0/1993234661 shutdown_connections 2022-07-20T02:34:48.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8341433f0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 --2- 172.21.15.26:0/1993234661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340957f0 0x7fc8340b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- 172.21.15.26:0/1993234661 >> 172.21.15.26:0/1993234661 conn(0x7fc8340b67f0 msgr2=0x7fc8340b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- 172.21.15.26:0/1993234661 shutdown_connections 2022-07-20T02:34:48.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- 172.21.15.26:0/1993234661 wait complete. 2022-07-20T02:34:48.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 Processor -- start 2022-07-20T02:34:48.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- start start 2022-07-20T02:34:48.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8340957f0 0x7fc834155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340b2f20 0x7fc834155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340934e0 con 0x7fc834067440 2022-07-20T02:34:48.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834156440 con 0x7fc8340957f0 2022-07-20T02:34:48.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83409aee0 con 0x7fc8340b2f20 2022-07-20T02:34:48.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43182/0 (socket says 172.21.15.26:43182) 2022-07-20T02:34:48.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 -- 172.21.15.26:0/4283611451 learned_addr learned my addr 172.21.15.26:0/4283611451 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:48.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc838f66700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340b2f20 0x7fc834155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc8337fe700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8340957f0 0x7fc834155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 -- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340b2f20 msgr2=0x7fc834155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340b2f20 0x7fc834155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 -- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8340957f0 msgr2=0x7fc834155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8340957f0 0x7fc834155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 -- 172.21.15.26:0/4283611451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82800b1e0 con 0x7fc834067440 2022-07-20T02:34:48.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc833fff700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fc8341432d0 tx=0x7fc82c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82c00e4a0 con 0x7fc834067440 2022-07-20T02:34:48.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c00e600 con 0x7fc834067440 2022-07-20T02:34:48.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83409b180 con 0x7fc834067440 2022-07-20T02:34:48.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.491+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82c00e960 con 0x7fc834067440 2022-07-20T02:34:48.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83409b790 con 0x7fc834067440 2022-07-20T02:34:48.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83409b950 con 0x7fc834067440 2022-07-20T02:34:48.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc82c019030 con 0x7fc834067440 2022-07-20T02:34:48.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc8317fa700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 0x7fc814055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc82c06b800 con 0x7fc834067440 2022-07-20T02:34:48.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc8337fe700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 0x7fc814055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.495+0000 7fc8337fe700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 0x7fc814055e40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc82400b000 tx=0x7fc824006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.503+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc82c042520 con 0x7fc834067440 2022-07-20T02:34:48.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.631+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc8340a95a0 con 0x7fc834067440 2022-07-20T02:34:48.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.631+0000 7fc8317fa700 1 -- 172.21.15.26:0/4283611451 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc82c045b40 con 0x7fc834067440 2022-07-20T02:34:48.656 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:48.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 msgr2=0x7fc814055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 0x7fc814055e40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc82400b000 tx=0x7fc824006df0).stop 2022-07-20T02:34:48.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 msgr2=0x7fc8340b1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fc8341432d0 tx=0x7fc82c001400).stop 2022-07-20T02:34:48.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 shutdown_connections 2022-07-20T02:34:48.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc814053980 0x7fc814055e40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8340b2f20 0x7fc834155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8340957f0 0x7fc834155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 --2- 172.21.15.26:0/4283611451 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc834067440 0x7fc8340b1a30 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 >> 172.21.15.26:0/4283611451 conn(0x7fc8340b67f0 msgr2=0x7fc8340b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 shutdown_connections 2022-07-20T02:34:48.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.635+0000 7fc839767700 1 -- 172.21.15.26:0/4283611451 wait complete. 2022-07-20T02:34:48.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:34:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 Processor -- start 2022-07-20T02:34:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 -- start start 2022-07-20T02:34:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 0x7fcb38147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb380b22d0 0x7fcb38143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb38061c80 con 0x7fcb38143d60 2022-07-20T02:34:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb380b2810 con 0x7fcb380b22d0 2022-07-20T02:34:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3d3f2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb380b2ab0 con 0x7fcb380618a0 2022-07-20T02:34:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb380b22d0 0x7fcb38143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3cbf1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcb38143d60 0x7fcb380b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42470/0 (socket says 172.21.15.26:42470) 2022-07-20T02:34:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb3cbf1700 1 -- 172.21.15.26:0/3782173223 learned_addr learned my addr 172.21.15.26:0/3782173223 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb37fff700 1 --2- 172.21.15.26:0/3782173223 >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 0x7fcb38147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 -- 172.21.15.26:0/3782173223 >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 msgr2=0x7fcb38147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 --2- 172.21.15.26:0/3782173223 >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 0x7fcb38147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 -- 172.21.15.26:0/3782173223 >> v1:172.21.15.26:6789/0 conn(0x7fcb38143d60 legacy=0x7fcb380b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 -- 172.21.15.26:0/3782173223 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb380b2d50 con 0x7fcb380b22d0 2022-07-20T02:34:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb37fff700 1 --2- 172.21.15.26:0/3782173223 >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 0x7fcb38147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb377fe700 1 --2- 172.21.15.26:0/3782173223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38143870 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fcb28002f70 tx=0x7fcb28006df0).ready entity=mon.0 client_cookie=b6d2a62ca3ef9715 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb36ffd700 1 -- 172.21.15.26:0/3782173223 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb2800d5c0 con 0x7fcb380b22d0 2022-07-20T02:34:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb36ffd700 1 -- 172.21.15.26:0/3782173223 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28000e90 con 0x7fcb380b22d0 2022-07-20T02:34:48.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.819+0000 7fcb36ffd700 1 -- 172.21.15.26:0/3782173223 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb2800d910 con 0x7fcb380b22d0 2022-07-20T02:34:48.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/3782173223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 msgr2=0x7fcb38143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/3782173223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38143870 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fcb28002f70 tx=0x7fcb28006df0).stop 2022-07-20T02:34:48.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/3782173223 shutdown_connections 2022-07-20T02:34:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/3782173223 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38143870 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/3782173223 >> v2:172.21.15.120:3300/0 conn(0x7fcb380618a0 0x7fcb38147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/3782173223 >> 172.21.15.26:0/3782173223 conn(0x7fcb38066f70 msgr2=0x7fcb38067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/3782173223 shutdown_connections 2022-07-20T02:34:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/3782173223 wait complete. 2022-07-20T02:34:48.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 Processor -- start 2022-07-20T02:34:48.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- start start 2022-07-20T02:34:48.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb380618a0 0x7fcb38146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb38143d60 0x7fcb380afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb38153610 con 0x7fcb380b22d0 2022-07-20T02:34:48.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380af240 con 0x7fcb380618a0 2022-07-20T02:34:48.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3d3f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380af4e0 con 0x7fcb38143d60 2022-07-20T02:34:48.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43198/0 (socket says 172.21.15.26:43198) 2022-07-20T02:34:48.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb377fe700 1 -- 172.21.15.26:0/1492669487 learned_addr learned my addr 172.21.15.26:0/1492669487 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:48.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb37fff700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb380618a0 0x7fcb38146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.823+0000 7fcb3cbf1700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb38143d60 0x7fcb380afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 -- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb38143d60 msgr2=0x7fcb380afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb38143d60 0x7fcb380afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 -- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb380618a0 msgr2=0x7fcb38146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb380618a0 0x7fcb38146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 -- 172.21.15.26:0/1492669487 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2800b180 con 0x7fcb380b22d0 2022-07-20T02:34:48.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb377fe700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7fcb28002f70 tx=0x7fcb28014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb28016070 con 0x7fcb380b22d0 2022-07-20T02:34:48.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2800d940 con 0x7fcb380b22d0 2022-07-20T02:34:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb380af780 con 0x7fcb380b22d0 2022-07-20T02:34:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb28013de0 con 0x7fcb380b22d0 2022-07-20T02:34:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb380aebc0 con 0x7fcb380b22d0 2022-07-20T02:34:48.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb380ae830 con 0x7fcb380b22d0 2022-07-20T02:34:48.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb28025030 con 0x7fcb380b22d0 2022-07-20T02:34:48.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.827+0000 7fcb357fa700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 0x7fcb18055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:48.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.831+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcb2806ce70 con 0x7fcb380b22d0 2022-07-20T02:34:48.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.831+0000 7fcb37fff700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 0x7fcb18055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:48.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.831+0000 7fcb37fff700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 0x7fcb18055ea0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fcb2c00c000 tx=0x7fcb2c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:48.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.839+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb380ae830 con 0x7fcb380b22d0 2022-07-20T02:34:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb380ae830 con 0x7fcb380b22d0 2022-07-20T02:34:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb357fa700 1 -- 172.21.15.26:0/1492669487 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcb380ae830 con 0x7fcb380b22d0 2022-07-20T02:34:48.972 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 msgr2=0x7fcb18055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 0x7fcb18055ea0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fcb2c00c000 tx=0x7fcb2c008040).stop 2022-07-20T02:34:48.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 msgr2=0x7fcb38144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:48.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.967+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7fcb28002f70 tx=0x7fcb28014000).stop 2022-07-20T02:34:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 shutdown_connections 2022-07-20T02:34:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb380b22d0 0x7fcb38144a00 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcb180539e0 0x7fcb18055ea0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb38143d60 0x7fcb380afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 --2- 172.21.15.26:0/1492669487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb380618a0 0x7fcb38146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 >> 172.21.15.26:0/1492669487 conn(0x7fcb38066f70 msgr2=0x7fcb38067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 shutdown_connections 2022-07-20T02:34:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:48.971+0000 7fcb3d3f2700 1 -- 172.21.15.26:0/1492669487 wait complete. 2022-07-20T02:34:48.997 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:34:48.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:49.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 Processor -- start 2022-07-20T02:34:49.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- start start 2022-07-20T02:34:49.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 0x7f3288067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f32880980d0 0x7f3288142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.178 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328805ff70 con 0x7f32880b3260 2022-07-20T02:34:49.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880b3640 con 0x7f32880980d0 2022-07-20T02:34:49.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880601a0 con 0x7f3288061860 2022-07-20T02:34:49.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f32880980d0 0x7f3288142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328659c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f32880b3260 0x7f32880979c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42482/0 (socket says 172.21.15.26:42482) 2022-07-20T02:34:49.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f32880980d0 0x7f3288142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43204/0 (socket says 172.21.15.26:43204) 2022-07-20T02:34:49.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328659c700 1 -- 172.21.15.26:0/422178858 learned_addr learned my addr 172.21.15.26:0/422178858 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:49.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f327ffff700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 0x7f3288067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 -- 172.21.15.26:0/422178858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 msgr2=0x7f3288067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:49.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 0x7f3288067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 -- 172.21.15.26:0/422178858 >> v1:172.21.15.26:6789/0 conn(0x7f32880b3260 legacy=0x7f32880979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 -- 172.21.15.26:0/422178858 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880606e0 con 0x7f32880980d0 2022-07-20T02:34:49.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3285d9b700 1 -- 172.21.15.26:0/422178858 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201110631 0 0) 0x7f328805ff70 con 0x7f32880b3260 2022-07-20T02:34:49.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f327ffff700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 0x7f3288067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:34:49.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3286d9d700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32880980d0 0x7f3288142d30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f32800032f0 tx=0x7f32800091d0).ready entity=mon.0 client_cookie=97d57e17a33f223a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:49.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3285d9b700 1 -- 172.21.15.26:0/422178858 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3280009ea0 con 0x7f32880980d0 2022-07-20T02:34:49.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3285d9b700 1 -- 172.21.15.26:0/422178858 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328000b8a0 con 0x7f32880980d0 2022-07-20T02:34:49.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f3285d9b700 1 -- 172.21.15.26:0/422178858 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328000ba10 con 0x7f32880980d0 2022-07-20T02:34:49.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- 172.21.15.26:0/422178858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32880980d0 msgr2=0x7f3288142d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32880980d0 0x7f3288142d30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f32800032f0 tx=0x7f32800091d0).stop 2022-07-20T02:34:49.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- 172.21.15.26:0/422178858 shutdown_connections 2022-07-20T02:34:49.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32880980d0 0x7f3288142d30 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 --2- 172.21.15.26:0/422178858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3288061860 0x7f3288067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- 172.21.15.26:0/422178858 >> 172.21.15.26:0/422178858 conn(0x7f32880b67f0 msgr2=0x7f32880b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:49.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- 172.21.15.26:0/422178858 shutdown_connections 2022-07-20T02:34:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.111+0000 7f328759e700 1 -- 172.21.15.26:0/422178858 wait complete. 2022-07-20T02:34:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 Processor -- start 2022-07-20T02:34:49.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- start start 2022-07-20T02:34:49.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32880980d0 0x7f3288062f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32880b3260 0x7f3288063440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328814ddf0 con 0x7f3288061860 2022-07-20T02:34:49.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3288064ab0 con 0x7f32880b3260 2022-07-20T02:34:49.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880b0120 con 0x7f32880980d0 2022-07-20T02:34:49.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43208/0 (socket says 172.21.15.26:43208) 2022-07-20T02:34:49.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 -- 172.21.15.26:0/1002806069 learned_addr learned my addr 172.21.15.26:0/1002806069 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:49.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f327ffff700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32880980d0 0x7f3288062f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f3286d9d700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32880b3260 0x7f3288063440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 -- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32880980d0 msgr2=0x7f3288062f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32880980d0 0x7f3288062f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 -- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32880b3260 msgr2=0x7f3288063440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32880b3260 0x7f3288063440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 -- 172.21.15.26:0/1002806069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280009470 con 0x7f3288061860 2022-07-20T02:34:49.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328659c700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f3278000fe0 tx=0x7f3278007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:49.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32780070d0 con 0x7f3288061860 2022-07-20T02:34:49.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278007230 con 0x7f3288061860 2022-07-20T02:34:49.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32880644b0 con 0x7f3288061860 2022-07-20T02:34:49.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3278005610 con 0x7f3288061860 2022-07-20T02:34:49.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.115+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328809bbd0 con 0x7f3288061860 2022-07-20T02:34:49.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880574e0 con 0x7f3288061860 2022-07-20T02:34:49.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3278010070 con 0x7f3288061860 2022-07-20T02:34:49.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f327f7fe700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 0x7f326c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:49.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f327806b740 con 0x7f3288061860 2022-07-20T02:34:49.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f327ffff700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 0x7f326c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:49.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.119+0000 7f327ffff700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 0x7f326c055fc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f327000c000 tx=0x7f3270008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:49.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.127+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3278042460 con 0x7f3288061860 2022-07-20T02:34:49.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.287+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f328806afc0 con 0x7f3288061860 2022-07-20T02:34:49.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f327f7fe700 1 -- 172.21.15.26:0/1002806069 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f3278045a80 con 0x7f3288061860 2022-07-20T02:34:49.325 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:49.325 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:34:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 msgr2=0x7f326c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 0x7f326c055fc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f327000c000 tx=0x7f3270008040).stop 2022-07-20T02:34:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 msgr2=0x7f328809aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f3278000fe0 tx=0x7f3278007c40).stop 2022-07-20T02:34:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 shutdown_connections 2022-07-20T02:34:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f326c053b00 0x7f326c055fc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32880b3260 0x7f3288063440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32880980d0 0x7f3288062f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 --2- 172.21.15.26:0/1002806069 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3288061860 0x7f328809aef0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 >> 172.21.15.26:0/1002806069 conn(0x7f32880b67f0 msgr2=0x7f32880b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 shutdown_connections 2022-07-20T02:34:49.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:49.291+0000 7f328759e700 1 -- 172.21.15.26:0/1002806069 wait complete. 2022-07-20T02:34:49.338 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:34:49.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:34:40.117843+0000', 'modified': '2022-07-20T02:34:41.123008+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8139': {'gid': 8139, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1171289829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 1171289829}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1171289829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:34:49.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:34:49.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:34:49.779 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8139,"want_state":"up:active","state":"up:active","rank_uptime":9.6275758919999994,"mdsmap_epoch":72,"osdmap_epoch":140,"osdmap_epoch_barrier":138,"uptime":14.687950689999999} 2022-07-20T02:34:49.781 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8139, "mdsmap_epoch": 72, "osdmap_epoch": 140, "osdmap_epoch_barrier": 138, "rank_uptime": 9.627575892, "state": "up:active", "uptime": 14.68795069, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:34:49.782 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:34:49.783 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:34:49.783 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:34:49.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:49.785 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:49.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:49.880 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:49.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:49.955 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:49.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:49.957 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:49.976 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:34:49.992 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:50.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:50.113 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:50.151 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:50.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:34:50.095+0000 7fa704240080 -1 init, newargv = 0x55aaf2f40810 newargc=9ceph-fuse[ 2022-07-20T02:34:50.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:22946]: starting ceph client 2022-07-20T02:34:50.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22946]: starting fuse 2022-07-20T02:34:50.197 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:50.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.205 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:50.317 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:34:50.318 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:34:50.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:34:50.321 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:34:50.322 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:34:50.323 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:34:50.323 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:34:50.324 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:50.363 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:34:50.375 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:34:50.376 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:34:50.376 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:50.377 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:34:50.378 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:34:50.379 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:34:50.380 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:34:50.380 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:50.381 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:34:50.427 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:34:50.428 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:34:50.428 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:34:50.429 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:34:50.430 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:50.431 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:34:50.432 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:34:50.513 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.22946.asok 2022-07-20T02:34:50.514 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22946.asok 2022-07-20T02:34:50.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22946.asok status 2022-07-20T02:34:50.658 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:34:50.659 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:34:50.660 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:34:50.661 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:34:50.662 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:34:50.662 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:34:50.663 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:34:50.663 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "22946", 2022-07-20T02:34:50.664 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:34:50.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:50.665 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:34:50.667 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:34:50.667 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 8355, 2022-07-20T02:34:50.668 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:34:50.677 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:34:50.711 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:34:50.711 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 8355 2022-07-20T02:34:50.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:50.712 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:50.713 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:50.713 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:50.723 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 7361246 2022-07-20T02:34:50.723 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:34:50.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:50.724 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:50.725 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:50.725 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:50.726 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 7361246 2022-07-20T02:34:50.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:50.732 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.8355 172.21.15.176:0/7361246", 2022-07-20T02:34:50.732 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/7361246", 2022-07-20T02:34:50.733 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:34:50.734 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 72, 2022-07-20T02:34:50.734 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 140, 2022-07-20T02:34:50.735 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:34:50.735 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:34:50.736 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:34:50.736 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:34:50.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.741 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:50.741 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:50.751 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:50.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:50.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.777 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:50.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.842 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:50.843 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:50.893 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:50.894 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:50.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:50.895 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:50.895 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:50.954 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:50.955 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:50.975 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:34:50.975 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:34:50.976 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:51.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[22946]: fuse finished with error 0 and tester_r 0 2022-07-20T02:34:51.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:51.057 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:51.057 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:51.073 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:34:51.073 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:34:51.076 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:34:51.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:51.096 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:51.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:34:51.161 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:34:51.167 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:34:51.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:34:51.197 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2022-07-20T02:34:51.197 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2022-07-20T02:34:51.197 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:34:51.198 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2022-07-20T02:34:51.198 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.198 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2022-07-20T02:34:51.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2022-07-20T02:34:51.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.229 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2022-07-20T02:34:51.229 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2022-07-20T02:34:51.229 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.244 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2022-07-20T02:34:51.245 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2022-07-20T02:34:51.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.246 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:34:51.246 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2022-07-20T02:34:51.247 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.247 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:34:51.248 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:34:51.248 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.248 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2022-07-20T02:34:51.249 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.249 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2022-07-20T02:34:51.249 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2022-07-20T02:34:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:34:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:34:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2022-07-20T02:34:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:34:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2022-07-20T02:34:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:34:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:34:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.279 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:34:51.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.279 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2022-07-20T02:34:51.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.279 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2022-07-20T02:34:51.284 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.285 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2022-07-20T02:34:51.285 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2022-07-20T02:34:51.285 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2022-07-20T02:34:51.285 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:34:51.286 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:34:51.286 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.286 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2022-07-20T02:34:51.286 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2022-07-20T02:34:51.287 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.287 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2022-07-20T02:34:51.287 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2022-07-20T02:34:51.287 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.288 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2022-07-20T02:34:51.288 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2022-07-20T02:34:51.288 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.288 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2022-07-20T02:34:51.288 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2022-07-20T02:34:51.289 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2022-07-20T02:34:51.331 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.339 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2022-07-20T02:34:51.339 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2022-07-20T02:34:51.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.341 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:34:51.342 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.348 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2022-07-20T02:34:51.348 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.348 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2022-07-20T02:34:51.348 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.349 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:34:51.349 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2022-07-20T02:34:51.354 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.355 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2022-07-20T02:34:51.355 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2022-07-20T02:34:51.356 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2022-07-20T02:34:51.356 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2022-07-20T02:34:51.356 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2022-07-20T02:34:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2022-07-20T02:34:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2022-07-20T02:34:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:34:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2022-07-20T02:34:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2022-07-20T02:34:51.451 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:34:51.451 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2022-07-20T02:34:51.451 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:34:51.452 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:34:51.452 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2022-07-20T02:34:51.459 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:34:51.460 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.460 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2022-07-20T02:34:51.460 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:34:51.461 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2022-07-20T02:34:51.461 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:34:51.461 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:34:51.462 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.462 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.463 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2022-07-20T02:34:51.463 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.464 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2022-07-20T02:34:51.464 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:34:51.465 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:34:51.465 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2022-07-20T02:34:51.465 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:34:51.471 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2022-07-20T02:34:51.471 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2022-07-20T02:34:51.471 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.472 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2022-07-20T02:34:51.472 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2022-07-20T02:34:51.472 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.472 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.473 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:34:51.473 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:34:51.473 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:34:51.473 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.476 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:34:51.476 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:34:51.477 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:34:51.477 INFO:teuthology.orchestra.run.smithi026.stdout: mon op complaint time = 120 2022-07-20T02:34:51.477 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:51.477 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2022-07-20T02:34:51.477 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2022-07-20T02:34:51.478 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2022-07-20T02:34:51.478 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2022-07-20T02:34:51.478 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:34:51.478 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:34:51.479 INFO:teuthology.orchestra.run.smithi026.stdout: client mount timeout = 600 2022-07-20T02:34:51.479 INFO:teuthology.orchestra.run.smithi026.stdout: debug client = 20 2022-07-20T02:34:51.479 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:34:51.479 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:34:51.479 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:34:51.480 INFO:teuthology.orchestra.run.smithi026.stdout: client debug getattr caps = true 2022-07-20T02:34:51.480 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2022-07-20T02:34:51.480 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2022-07-20T02:34:51.480 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2022-07-20T02:34:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.a] 2022-07-20T02:34:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.c] 2022-07-20T02:34:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.b] 2022-07-20T02:34:51.481 INFO:teuthology.orchestra.run.smithi026.stdout:[mds] 2022-07-20T02:34:51.481 INFO:teuthology.orchestra.run.smithi026.stdout: debug mds = 20 2022-07-20T02:34:51.482 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:34:51.482 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal frag = True 2022-07-20T02:34:51.482 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal fragment size max = 10000 2022-07-20T02:34:51.482 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal merge size = 5 2022-07-20T02:34:51.483 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split bits = 3 2022-07-20T02:34:51.483 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split size = 100 2022-07-20T02:34:51.483 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug frag = True 2022-07-20T02:34:51.484 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug scatterstat = True 2022-07-20T02:34:51.484 INFO:teuthology.orchestra.run.smithi026.stdout: mds op complaint time = 180 2022-07-20T02:34:51.501 INFO:teuthology.orchestra.run.smithi026.stdout: mds verify scatter = True 2022-07-20T02:34:51.502 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:34:51.502 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:34:51.503 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:34:51.517 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2022-07-20T02:34:51.517 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2022-07-20T02:34:51.517 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:34:51.604 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2022-07-20T02:34:51.605 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.605 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2022-07-20T02:34:51.605 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.606 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2022-07-20T02:34:51.606 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.606 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2022-07-20T02:34:51.606 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2022-07-20T02:34:51.607 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.641 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2022-07-20T02:34:51.641 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2022-07-20T02:34:51.642 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.643 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:34:51.643 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2022-07-20T02:34:51.643 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.644 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:34:51.644 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:34:51.645 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.645 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2022-07-20T02:34:51.659 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2022-07-20T02:34:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2022-07-20T02:34:51.669 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:34:51.669 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:34:51.669 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2022-07-20T02:34:51.669 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:34:51.676 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2022-07-20T02:34:51.677 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:34:51.677 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:34:51.677 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.678 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:34:51.678 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.679 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2022-07-20T02:34:51.679 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.680 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2022-07-20T02:34:51.681 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.682 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2022-07-20T02:34:51.682 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2022-07-20T02:34:51.682 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2022-07-20T02:34:51.683 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:34:51.683 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:34:51.684 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.684 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2022-07-20T02:34:51.685 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2022-07-20T02:34:51.685 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.685 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2022-07-20T02:34:51.686 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2022-07-20T02:34:51.686 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.687 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2022-07-20T02:34:51.687 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2022-07-20T02:34:51.688 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.688 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2022-07-20T02:34:51.688 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2022-07-20T02:34:51.689 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2022-07-20T02:34:51.689 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.701 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2022-07-20T02:34:51.703 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2022-07-20T02:34:51.715 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.715 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:34:51.716 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.716 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2022-07-20T02:34:51.717 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.717 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2022-07-20T02:34:51.736 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.737 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:34:51.737 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2022-07-20T02:34:51.737 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.738 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2022-07-20T02:34:51.738 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2022-07-20T02:34:51.738 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2022-07-20T02:34:51.738 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2022-07-20T02:34:51.739 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2022-07-20T02:34:51.739 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2022-07-20T02:34:51.739 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2022-07-20T02:34:51.739 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:34:51.739 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2022-07-20T02:34:51.740 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2022-07-20T02:34:51.743 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:34:51.743 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2022-07-20T02:34:51.750 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:34:51.752 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:34:51.758 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2022-07-20T02:34:51.758 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:34:51.759 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.759 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2022-07-20T02:34:51.760 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:34:51.769 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2022-07-20T02:34:51.769 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:34:51.771 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:34:51.771 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.800 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.801 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2022-07-20T02:34:51.801 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.802 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2022-07-20T02:34:51.802 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:34:51.803 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:34:51.803 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2022-07-20T02:34:51.804 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:34:51.805 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2022-07-20T02:34:51.805 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2022-07-20T02:34:51.806 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.806 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2022-07-20T02:34:51.807 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2022-07-20T02:34:51.807 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.808 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.808 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:34:51.808 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:34:51.809 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:34:51.809 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.810 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:34:51.810 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:34:51.811 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:34:51.838 INFO:teuthology.orchestra.run.smithi176.stdout: mon op complaint time = 120 2022-07-20T02:34:51.845 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:34:51.846 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2022-07-20T02:34:51.846 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2022-07-20T02:34:51.847 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2022-07-20T02:34:51.852 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2022-07-20T02:34:51.852 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:34:51.853 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:34:51.853 INFO:teuthology.orchestra.run.smithi176.stdout: client mount timeout = 600 2022-07-20T02:34:51.853 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2022-07-20T02:34:51.854 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:34:51.854 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:34:51.855 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:34:51.855 INFO:teuthology.orchestra.run.smithi176.stdout: client debug getattr caps = true 2022-07-20T02:34:51.855 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2022-07-20T02:34:51.857 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2022-07-20T02:34:51.857 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2022-07-20T02:34:51.858 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2022-07-20T02:34:51.858 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2022-07-20T02:34:51.858 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2022-07-20T02:34:51.859 INFO:teuthology.orchestra.run.smithi176.stdout:[mds] 2022-07-20T02:34:51.859 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds = 20 2022-07-20T02:34:51.860 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:34:51.860 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal frag = True 2022-07-20T02:34:51.861 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal fragment size max = 10000 2022-07-20T02:34:51.861 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal merge size = 5 2022-07-20T02:34:51.862 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split bits = 3 2022-07-20T02:34:51.863 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split size = 100 2022-07-20T02:34:51.863 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug frag = True 2022-07-20T02:34:51.863 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug scatterstat = True 2022-07-20T02:34:51.864 INFO:teuthology.orchestra.run.smithi176.stdout: mds op complaint time = 180 2022-07-20T02:34:51.864 INFO:teuthology.orchestra.run.smithi176.stdout: mds verify scatter = True 2022-07-20T02:34:51.865 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:34:51.865 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:34:51.879 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:34:51.880 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2022-07-20T02:34:51.880 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2022-07-20T02:34:51.881 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:34:51.881 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2022-07-20T02:34:51.882 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.882 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2022-07-20T02:34:51.883 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.883 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2022-07-20T02:34:51.896 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.897 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2022-07-20T02:34:51.897 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2022-07-20T02:34:51.906 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.907 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2022-07-20T02:34:51.907 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2022-07-20T02:34:51.908 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.908 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:34:51.908 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2022-07-20T02:34:51.909 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.909 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:34:51.910 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:34:51.915 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.915 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2022-07-20T02:34:51.916 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.916 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2022-07-20T02:34:51.917 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2022-07-20T02:34:51.917 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:34:51.918 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:34:51.918 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2022-07-20T02:34:51.918 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:34:51.921 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2022-07-20T02:34:51.921 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:34:51.921 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:34:51.922 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.922 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:34:51.922 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.922 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2022-07-20T02:34:51.923 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.923 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2022-07-20T02:34:51.923 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.923 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2022-07-20T02:34:51.924 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2022-07-20T02:34:51.924 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2022-07-20T02:34:51.924 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:34:51.924 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:34:51.924 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.925 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2022-07-20T02:34:51.925 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2022-07-20T02:34:51.925 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.925 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2022-07-20T02:34:51.926 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2022-07-20T02:34:51.926 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.927 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2022-07-20T02:34:51.927 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2022-07-20T02:34:51.927 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.927 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2022-07-20T02:34:51.928 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2022-07-20T02:34:51.928 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2022-07-20T02:34:51.928 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.928 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2022-07-20T02:34:51.929 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2022-07-20T02:34:51.929 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.929 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:34:51.929 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.930 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2022-07-20T02:34:51.930 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.930 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2022-07-20T02:34:51.931 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.931 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:34:51.931 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2022-07-20T02:34:51.932 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.932 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2022-07-20T02:34:51.932 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2022-07-20T02:34:51.932 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2022-07-20T02:34:51.933 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression mode = aggressive 2022-07-20T02:34:51.933 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2022-07-20T02:34:51.933 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2022-07-20T02:34:51.933 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2022-07-20T02:34:51.941 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:34:51.942 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2022-07-20T02:34:51.943 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2022-07-20T02:34:51.943 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:34:51.944 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2022-07-20T02:34:51.944 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:34:51.944 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:34:51.945 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2022-07-20T02:34:51.945 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:34:51.946 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.946 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2022-07-20T02:34:51.963 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:34:51.964 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2022-07-20T02:34:51.967 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:34:51.968 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:34:51.968 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.969 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2022-07-20T02:34:51.970 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.970 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2022-07-20T02:34:51.970 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:34:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:34:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2022-07-20T02:34:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:34:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2022-07-20T02:34:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2022-07-20T02:34:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2022-07-20T02:34:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2022-07-20T02:34:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:34:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:34:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:34:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:34:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:34:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:34:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2022-07-20T02:34:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:34:51.976 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2022-07-20T02:34:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2022-07-20T02:34:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2022-07-20T02:34:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2022-07-20T02:34:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:34:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:34:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2022-07-20T02:34:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2022-07-20T02:34:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:34:51.987 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:34:51.987 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:34:51.987 INFO:teuthology.orchestra.run.smithi159.stdout: client debug getattr caps = true 2022-07-20T02:34:51.988 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2022-07-20T02:34:51.988 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2022-07-20T02:34:51.988 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2022-07-20T02:34:51.988 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2022-07-20T02:34:51.989 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2022-07-20T02:34:51.995 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2022-07-20T02:34:51.995 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2022-07-20T02:34:51.996 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2022-07-20T02:34:51.996 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:34:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal frag = True 2022-07-20T02:34:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal fragment size max = 10000 2022-07-20T02:34:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal merge size = 5 2022-07-20T02:34:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split bits = 3 2022-07-20T02:34:51.997 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split size = 100 2022-07-20T02:34:51.998 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2022-07-20T02:34:51.998 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2022-07-20T02:34:51.998 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2022-07-20T02:34:51.998 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2022-07-20T02:34:51.999 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:34:52.010 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:34:52.010 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:34:52.010 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2022-07-20T02:34:52.011 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2022-07-20T02:34:52.011 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:34:52.011 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2022-07-20T02:34:52.011 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2022-07-20T02:34:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2022-07-20T02:34:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2022-07-20T02:34:52.013 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2022-07-20T02:34:52.013 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.013 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2022-07-20T02:34:52.013 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2022-07-20T02:34:52.014 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.014 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:34:52.015 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2022-07-20T02:34:52.015 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.015 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:34:52.016 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:34:52.016 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.016 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2022-07-20T02:34:52.017 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.017 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2022-07-20T02:34:52.018 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2022-07-20T02:34:52.018 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:34:52.018 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:34:52.018 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2022-07-20T02:34:52.019 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:34:52.019 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2022-07-20T02:34:52.019 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:34:52.019 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:34:52.019 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.020 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:34:52.020 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.020 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2022-07-20T02:34:52.020 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.021 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2022-07-20T02:34:52.021 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.021 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2022-07-20T02:34:52.021 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2022-07-20T02:34:52.021 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2022-07-20T02:34:52.022 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:34:52.022 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:34:52.022 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.022 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2022-07-20T02:34:52.022 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2022-07-20T02:34:52.023 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.023 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2022-07-20T02:34:52.023 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2022-07-20T02:34:52.023 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.024 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2022-07-20T02:34:52.024 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2022-07-20T02:34:52.024 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.024 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2022-07-20T02:34:52.024 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2022-07-20T02:34:52.025 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2022-07-20T02:34:52.025 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.025 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2022-07-20T02:34:52.025 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2022-07-20T02:34:52.026 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.026 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:34:52.027 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.027 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2022-07-20T02:34:52.028 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.028 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2022-07-20T02:34:52.077 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.078 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:34:52.078 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2022-07-20T02:34:52.078 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.078 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2022-07-20T02:34:52.079 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2022-07-20T02:34:52.079 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2022-07-20T02:34:52.079 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore compression mode = aggressive 2022-07-20T02:34:52.079 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2022-07-20T02:34:52.079 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 20 2022-07-20T02:34:52.080 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 20 2022-07-20T02:34:52.080 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:34:52.080 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2022-07-20T02:34:52.080 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2022-07-20T02:34:52.081 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:34:52.081 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2022-07-20T02:34:52.081 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:34:52.081 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:34:52.081 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2022-07-20T02:34:52.082 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:34:52.082 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.082 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2022-07-20T02:34:52.082 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:34:52.083 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2022-07-20T02:34:52.083 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:34:52.083 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:34:52.083 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:52.084 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:52.084 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2022-07-20T02:34:52.084 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.084 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2022-07-20T02:34:52.084 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:34:52.085 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:34:52.085 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2022-07-20T02:34:52.085 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:34:52.085 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2022-07-20T02:34:52.087 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2022-07-20T02:34:52.087 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:34:52.088 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2022-07-20T02:34:52.088 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2022-07-20T02:34:52.088 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:34:52.088 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.088 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:34:52.089 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:34:52.089 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:34:52.089 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.089 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:34:52.090 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:34:52.090 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:34:52.090 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2022-07-20T02:34:52.090 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:34:52.090 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2022-07-20T02:34:52.091 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2022-07-20T02:34:52.091 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2022-07-20T02:34:52.091 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2022-07-20T02:34:52.091 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:34:52.092 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:34:52.092 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2022-07-20T02:34:52.092 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2022-07-20T02:34:52.092 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:34:52.093 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:34:52.093 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:34:52.093 INFO:teuthology.orchestra.run.smithi120.stdout: client debug getattr caps = true 2022-07-20T02:34:52.093 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2022-07-20T02:34:52.093 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2022-07-20T02:34:52.099 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2022-07-20T02:34:52.100 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2022-07-20T02:34:52.115 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2022-07-20T02:34:52.116 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2022-07-20T02:34:52.116 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2022-07-20T02:34:52.117 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2022-07-20T02:34:52.135 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:34:52.138 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal frag = True 2022-07-20T02:34:52.139 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal fragment size max = 10000 2022-07-20T02:34:52.141 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal merge size = 5 2022-07-20T02:34:52.142 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split bits = 3 2022-07-20T02:34:52.174 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split size = 100 2022-07-20T02:34:52.175 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2022-07-20T02:34:52.176 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2022-07-20T02:34:52.177 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2022-07-20T02:34:52.178 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2022-07-20T02:34:52.178 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:34:52.178 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:34:52.181 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:34:52.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:34:52.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.295+0000 7faa8da1f700 1 Processor -- start 2022-07-20T02:34:52.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- start start 2022-07-20T02:34:52.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880b02a0 0x7faa8805be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa880ae110 0x7faa8805c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880ae890 con 0x7faa880a9020 2022-07-20T02:34:52.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa880aeb30 con 0x7faa880ae110 2022-07-20T02:34:52.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa881478a0 con 0x7faa880b02a0 2022-07-20T02:34:52.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa880ae110 0x7faa8805c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa880ae110 0x7faa8805c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43220/0 (socket says 172.21.15.26:43220) 2022-07-20T02:34:52.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 -- 172.21.15.26:0/451374361 learned_addr learned my addr 172.21.15.26:0/451374361 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:52.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa86ffd700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880b02a0 0x7faa8805be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 -- 172.21.15.26:0/451374361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880b02a0 msgr2=0x7faa8805be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:52.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880b02a0 0x7faa8805be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 -- 172.21.15.26:0/451374361 >> v1:172.21.15.120:6789/0 conn(0x7faa880a9020 legacy=0x7faa88147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 -- 172.21.15.26:0/451374361 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88147b40 con 0x7faa880ae110 2022-07-20T02:34:52.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa877fe700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880ae110 0x7faa8805c440 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7faa800032f0 tx=0x7faa80009290).ready entity=mon.0 client_cookie=37ad03d1a0f80b77 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa85ffb700 1 -- 172.21.15.26:0/451374361 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa8000b8a0 con 0x7faa880ae110 2022-07-20T02:34:52.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa85ffb700 1 -- 172.21.15.26:0/451374361 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8000ba00 con 0x7faa880ae110 2022-07-20T02:34:52.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa85ffb700 1 -- 172.21.15.26:0/451374361 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa8000bb70 con 0x7faa880ae110 2022-07-20T02:34:52.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- 172.21.15.26:0/451374361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880ae110 msgr2=0x7faa8805c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880ae110 0x7faa8805c440 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7faa800032f0 tx=0x7faa80009290).stop 2022-07-20T02:34:52.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- 172.21.15.26:0/451374361 shutdown_connections 2022-07-20T02:34:52.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880ae110 0x7faa8805c440 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 --2- 172.21.15.26:0/451374361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880b02a0 0x7faa8805be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- 172.21.15.26:0/451374361 >> 172.21.15.26:0/451374361 conn(0x7faa880606e0 msgr2=0x7faa88060af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- 172.21.15.26:0/451374361 shutdown_connections 2022-07-20T02:34:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.299+0000 7faa8da1f700 1 -- 172.21.15.26:0/451374361 wait complete. 2022-07-20T02:34:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 Processor -- start 2022-07-20T02:34:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 -- start start 2022-07-20T02:34:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880ae110 0x7faa880ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa880b02a0 0x7faa880ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880ab3c0 con 0x7faa880a9020 2022-07-20T02:34:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880aa0a0 con 0x7faa880b02a0 2022-07-20T02:34:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa8da1f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8814a9c0 con 0x7faa880ae110 2022-07-20T02:34:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43222/0 (socket says 172.21.15.26:43222) 2022-07-20T02:34:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 -- 172.21.15.26:0/4070571867 learned_addr learned my addr 172.21.15.26:0/4070571867 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa877fe700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa880b02a0 0x7faa880ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa867fc700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880ae110 0x7faa880ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 -- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880ae110 msgr2=0x7faa880ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880ae110 0x7faa880ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 -- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa880b02a0 msgr2=0x7faa880ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa880b02a0 0x7faa880ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 -- 172.21.15.26:0/4070571867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa80009530 con 0x7faa880a9020 2022-07-20T02:34:52.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.303+0000 7faa86ffd700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7faa7c00c000 tx=0x7faa7c006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7c011750 con 0x7faa880a9020 2022-07-20T02:34:52.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c0118b0 con 0x7faa880a9020 2022-07-20T02:34:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8814ac60 con 0x7faa880a9020 2022-07-20T02:34:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7c011c10 con 0x7faa880a9020 2022-07-20T02:34:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8814b8f0 con 0x7faa880a9020 2022-07-20T02:34:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa881486f0 con 0x7faa880a9020 2022-07-20T02:34:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa7c011d70 con 0x7faa880a9020 2022-07-20T02:34:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 0x7faa64055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faa7c06bfc0 con 0x7faa880a9020 2022-07-20T02:34:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa867fc700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 0x7faa64055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.307+0000 7faa867fc700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 0x7faa64055ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7faa7800d000 tx=0x7faa78000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.315+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa7c042d20 con 0x7faa880a9020 2022-07-20T02:34:52.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.439+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa880574e0 con 0x7faa880a9020 2022-07-20T02:34:52.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.439+0000 7faa77fff700 1 -- 172.21.15.26:0/4070571867 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16333 (secure 0 0 0) 0x7faa7c046340 con 0x7faa880a9020 2022-07-20T02:34:52.449 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:52.450 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":140,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:34:42.132316+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:34:38.295386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T02:34:39.171843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:34:52.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 msgr2=0x7faa64055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 0x7faa64055ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7faa7800d000 tx=0x7faa78000bc0).stop 2022-07-20T02:34:52.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 msgr2=0x7faa88146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7faa7c00c000 tx=0x7faa7c006d90).stop 2022-07-20T02:34:52.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 shutdown_connections 2022-07-20T02:34:52.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa640539e0 0x7faa64055ea0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa880b02a0 0x7faa880ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa880ae110 0x7faa880ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 --2- 172.21.15.26:0/4070571867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa880a9020 0x7faa88146900 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 >> 172.21.15.26:0/4070571867 conn(0x7faa880606e0 msgr2=0x7faa881443a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:52.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 shutdown_connections 2022-07-20T02:34:52.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.443+0000 7faa8da1f700 1 -- 172.21.15.26:0/4070571867 wait complete. 2022-07-20T02:34:52.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:34:38.295386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:34:39.171843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:34:52.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:34:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.635+0000 7f04ab4f9700 1 Processor -- start 2022-07-20T02:34:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.635+0000 7f04ab4f9700 1 -- start start 2022-07-20T02:34:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a413d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a405bed0 con 0x7f04a40617a0 2022-07-20T02:34:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a4061b80 con 0x7f04a40aa020 2022-07-20T02:34:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a405c100 con 0x7f04a413ff60 2022-07-20T02:34:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aa4f7700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f04a40617a0 0x7f04a405cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42510/0 (socket says 172.21.15.26:42510) 2022-07-20T02:34:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aa4f7700 1 -- 172.21.15.26:0/3260293957 learned_addr learned my addr 172.21.15.26:0/3260293957 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aacf8700 1 --2- 172.21.15.26:0/3260293957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854896290 0 0) 0x7f04a405bed0 con 0x7f04a40617a0 2022-07-20T02:34:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003260 con 0x7f04a40617a0 2022-07-20T02:34:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363874888 0 0) 0x7f04a4061b80 con 0x7f04a40aa020 2022-07-20T02:34:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aacf8700 1 -- 172.21.15.26:0/3260293957 >> v1:172.21.15.120:6789/0 conn(0x7f04a40aa020 legacy=0x7f04a413d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aacf8700 1 -- 172.21.15.26:0/3260293957 >> v1:172.21.15.26:6789/0 conn(0x7f04a40617a0 legacy=0x7f04a405cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aacf8700 1 -- 172.21.15.26:0/3260293957 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a405c3a0 con 0x7f04a413ff60 2022-07-20T02:34:52.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04aacf8700 1 --2- 172.21.15.26:0/3260293957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a413d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f049c0032f0 tx=0x7f049c0091d0).ready entity=mon.2 client_cookie=96208100076aaf89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f049c00b6c0 con 0x7f04a413ff60 2022-07-20T02:34:52.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c00b820 con 0x7f04a413ff60 2022-07-20T02:34:52.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04a94f5700 1 -- 172.21.15.26:0/3260293957 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f049c00b990 con 0x7f04a413ff60 2022-07-20T02:34:52.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- 172.21.15.26:0/3260293957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 msgr2=0x7f04a413d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/3260293957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a413d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f049c0032f0 tx=0x7f049c0091d0).stop 2022-07-20T02:34:52.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- 172.21.15.26:0/3260293957 shutdown_connections 2022-07-20T02:34:52.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/3260293957 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a413d840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- 172.21.15.26:0/3260293957 >> 172.21.15.26:0/3260293957 conn(0x7f04a40b66e0 msgr2=0x7f04a40b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:52.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- 172.21.15.26:0/3260293957 shutdown_connections 2022-07-20T02:34:52.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.639+0000 7f04ab4f9700 1 -- 172.21.15.26:0/3260293957 wait complete. 2022-07-20T02:34:52.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 Processor -- start 2022-07-20T02:34:52.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 -- start start 2022-07-20T02:34:52.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40617a0 0x7f04a413ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a4142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a413c060 con 0x7f04a40aa020 2022-07-20T02:34:52.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a413abe0 con 0x7f04a40617a0 2022-07-20T02:34:52.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a413c290 con 0x7f04a413ff60 2022-07-20T02:34:52.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43236/0 (socket says 172.21.15.26:43236) 2022-07-20T02:34:52.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 -- 172.21.15.26:0/1208365088 learned_addr learned my addr 172.21.15.26:0/1208365088 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:52.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04aa4f7700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40617a0 0x7f04a413ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04aacf8700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a4142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 -- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 msgr2=0x7f04a4142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a4142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 -- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40617a0 msgr2=0x7f04a413ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40617a0 0x7f04a413ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 -- 172.21.15.26:0/1208365088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c009470 con 0x7f04a40aa020 2022-07-20T02:34:52.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04a9cf6700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f0498000bc0 tx=0x7f0498007a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f049800ce00 con 0x7f04a40aa020 2022-07-20T02:34:52.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049800f470 con 0x7f04a40aa020 2022-07-20T02:34:52.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a413ae40 con 0x7f04a40aa020 2022-07-20T02:34:52.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.643+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f049800f7d0 con 0x7f04a40aa020 2022-07-20T02:34:52.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a41432f0 con 0x7f04a40aa020 2022-07-20T02:34:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a413c7b0 con 0x7f04a40aa020 2022-07-20T02:34:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0498019030 con 0x7f04a40aa020 2022-07-20T02:34:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04977fe700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 0x7f0484055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f049806b3c0 con 0x7f04a40aa020 2022-07-20T02:34:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04aa4f7700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 0x7f0484055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.647+0000 7f04aa4f7700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 0x7f0484055fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f04a000b000 tx=0x7f04a0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.655+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0498042230 con 0x7f04a40aa020 2022-07-20T02:34:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f04a413afd0 con 0x7f04a40aa020 2022-07-20T02:34:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04977fe700 1 -- 172.21.15.26:0/1208365088 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0498045850 con 0x7f04a40aa020 2022-07-20T02:34:52.816 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:34:52.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 msgr2=0x7f0484055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 0x7f0484055fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f04a000b000 tx=0x7f04a0009040).stop 2022-07-20T02:34:52.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 msgr2=0x7f04a413b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f0498000bc0 tx=0x7f0498007a70).stop 2022-07-20T02:34:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 shutdown_connections 2022-07-20T02:34:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f04a40aa020 0x7f04a413b2e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0484053b00 0x7f0484055fc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04a413ff60 0x7f04a4142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 --2- 172.21.15.26:0/1208365088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04a40617a0 0x7f04a413ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 >> 172.21.15.26:0/1208365088 conn(0x7f04a40b66e0 msgr2=0x7f04a4140640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 shutdown_connections 2022-07-20T02:34:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.783+0000 7f04ab4f9700 1 -- 172.21.15.26:0/1208365088 wait complete. 2022-07-20T02:34:52.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:34:52.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.931+0000 7fcbbbfff700 1 Processor -- start 2022-07-20T02:34:52.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.931+0000 7fcbbbfff700 1 -- start start 2022-07-20T02:34:52.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbbc0b2170 0x7fcbbc0a9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcbbc1405a0 0x7fcbbc1409e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc05d150 con 0x7fcbbc0abad0 2022-07-20T02:34:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc05d380 con 0x7fcbbc0b2170 2022-07-20T02:34:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc140100 con 0x7fcbbc1405a0 2022-07-20T02:34:52.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbbc0b2170 0x7fcbbc0a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcbbc0b2170 0x7fcbbc0a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43244/0 (socket says 172.21.15.26:43244) 2022-07-20T02:34:52.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 -- 172.21.15.26:0/268850087 learned_addr learned my addr 172.21.15.26:0/268850087 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:52.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbb3fff700 1 --2- 172.21.15.26:0/268850087 >> v2:172.21.15.120:3300/0 conn(0x7fcbbc1405a0 0x7fcbbc1409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 -- 172.21.15.26:0/268850087 >> v2:172.21.15.120:3300/0 conn(0x7fcbbc1405a0 msgr2=0x7fcbbc1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:52.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 --2- 172.21.15.26:0/268850087 >> v2:172.21.15.120:3300/0 conn(0x7fcbbc1405a0 0x7fcbbc1409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 -- 172.21.15.26:0/268850087 >> v1:172.21.15.120:6789/0 conn(0x7fcbbc0abad0 legacy=0x7fcbbc143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 -- 172.21.15.26:0/268850087 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc13fe30 con 0x7fcbbc0b2170 2022-07-20T02:34:52.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbaffd700 1 --2- 172.21.15.26:0/268850087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0a9540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7fcbac002f70 tx=0x7fcbac006eb0).ready entity=mon.0 client_cookie=9348c57550169716 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbba7fc700 1 -- 172.21.15.26:0/268850087 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbac00d5c0 con 0x7fcbbc0b2170 2022-07-20T02:34:52.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbba7fc700 1 -- 172.21.15.26:0/268850087 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac000e90 con 0x7fcbbc0b2170 2022-07-20T02:34:52.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbba7fc700 1 -- 172.21.15.26:0/268850087 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbac00d950 con 0x7fcbbc0b2170 2022-07-20T02:34:52.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- 172.21.15.26:0/268850087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 msgr2=0x7fcbbc0a9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/268850087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0a9540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7fcbac002f70 tx=0x7fcbac006eb0).stop 2022-07-20T02:34:52.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- 172.21.15.26:0/268850087 shutdown_connections 2022-07-20T02:34:52.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/268850087 >> v2:172.21.15.120:3300/0 conn(0x7fcbbc1405a0 0x7fcbbc1409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/268850087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0a9540 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- 172.21.15.26:0/268850087 >> 172.21.15.26:0/268850087 conn(0x7fcbbc062700 msgr2=0x7fcbbc062ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:52.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- 172.21.15.26:0/268850087 shutdown_connections 2022-07-20T02:34:52.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.935+0000 7fcbbbfff700 1 -- 172.21.15.26:0/268850087 wait complete. 2022-07-20T02:34:52.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 Processor -- start 2022-07-20T02:34:52.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 -- start start 2022-07-20T02:34:52.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbbc0abad0 0x7fcbbc143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbbc1405a0 0x7fcbbc0657e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc065cd0 con 0x7fcbbc0b2170 2022-07-20T02:34:52.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0633e0 con 0x7fcbbc0abad0 2022-07-20T02:34:52.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbbfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc063680 con 0x7fcbbc1405a0 2022-07-20T02:34:52.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43252/0 (socket says 172.21.15.26:43252) 2022-07-20T02:34:52.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 -- 172.21.15.26:0/163564779 learned_addr learned my addr 172.21.15.26:0/163564779 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:52.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbaffd700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbbc0abad0 0x7fcbbc143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbbb7fe700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbbc1405a0 0x7fcbbc0657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 -- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbbc1405a0 msgr2=0x7fcbbc0657e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbbc1405a0 0x7fcbbc0657e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 -- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbbc0abad0 msgr2=0x7fcbbc143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:52.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbbc0abad0 0x7fcbbc143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:52.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 -- 172.21.15.26:0/163564779 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac00b1e0 con 0x7fcbbc0b2170 2022-07-20T02:34:52.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb3fff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fcba400c000 tx=0x7fcba4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba4011920 con 0x7fcbbc0b2170 2022-07-20T02:34:52.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.939+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4011a80 con 0x7fcbbc0b2170 2022-07-20T02:34:52.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc063920 con 0x7fcbbc0b2170 2022-07-20T02:34:52.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba4011de0 con 0x7fcbbc0b2170 2022-07-20T02:34:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc152c80 con 0x7fcbbc0b2170 2022-07-20T02:34:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc0574e0 con 0x7fcbbc0b2170 2022-07-20T02:34:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcba4021070 con 0x7fcbbc0b2170 2022-07-20T02:34:52.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbb8ff9700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 0x7fcba0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbbaffd700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 0x7fcba0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.943+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcba406e310 con 0x7fcbbc0b2170 2022-07-20T02:34:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.947+0000 7fcbbaffd700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 0x7fcba0055e10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fcbac01c000 tx=0x7fcbac015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:52.951+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcba4044f60 con 0x7fcbbc0b2170 2022-07-20T02:34:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.079+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcbbc06ad30 con 0x7fcbbc0b2170 2022-07-20T02:34:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.079+0000 7fcbb8ff9700 1 -- 172.21.15.26:0/163564779 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcba4048580 con 0x7fcbbc0b2170 2022-07-20T02:34:53.085 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:53.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 msgr2=0x7fcba0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 0x7fcba0055e10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fcbac01c000 tx=0x7fcbac015000).stop 2022-07-20T02:34:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 msgr2=0x7fcbbc0652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fcba400c000 tx=0x7fcba4006df0).stop 2022-07-20T02:34:53.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 shutdown_connections 2022-07-20T02:34:53.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcbbc0b2170 0x7fcbbc0652f0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcba0053950 0x7fcba0055e10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbbc1405a0 0x7fcbbc0657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 --2- 172.21.15.26:0/163564779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbbc0abad0 0x7fcbbc143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 >> 172.21.15.26:0/163564779 conn(0x7fcbbc062700 msgr2=0x7fcbbc0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 shutdown_connections 2022-07-20T02:34:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.083+0000 7fcbbbfff700 1 -- 172.21.15.26:0/163564779 wait complete. 2022-07-20T02:34:53.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:34:53.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.231+0000 7fcd2e741700 1 Processor -- start 2022-07-20T02:34:53.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.231+0000 7fcd2e741700 1 -- start start 2022-07-20T02:34:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd280b46b0 0x7fcd280ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd281471f0 con 0x7fcd280b5d90 2022-07-20T02:34:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd28147490 con 0x7fcd28146a20 2022-07-20T02:34:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd280b16f0 con 0x7fcd280b46b0 2022-07-20T02:34:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2df40700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcd280b5d90 0x7fcd2812c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42540/0 (socket says 172.21.15.26:42540) 2022-07-20T02:34:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2df40700 1 -- 172.21.15.26:0/861607106 learned_addr learned my addr 172.21.15.26:0/861607106 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2cf3e700 1 --2- 172.21.15.26:0/861607106 >> v2:172.21.15.120:3300/0 conn(0x7fcd280b46b0 0x7fcd280ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455349179 0 0) 0x7fcd281471f0 con 0x7fcd280b5d90 2022-07-20T02:34:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd10003260 con 0x7fcd280b5d90 2022-07-20T02:34:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323145020 0 0) 0x7fcd28147490 con 0x7fcd28146a20 2022-07-20T02:34:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd281471f0 con 0x7fcd28146a20 2022-07-20T02:34:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661825540 0 0) 0x7fcd10003260 con 0x7fcd280b5d90 2022-07-20T02:34:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2cf3e700 1 -- 172.21.15.26:0/861607106 >> v1:172.21.15.120:6789/0 conn(0x7fcd28146a20 legacy=0x7fcd280b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2cf3e700 1 -- 172.21.15.26:0/861607106 >> v1:172.21.15.26:6789/0 conn(0x7fcd280b5d90 legacy=0x7fcd2812c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2cf3e700 1 -- 172.21.15.26:0/861607106 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280b6590 con 0x7fcd280b46b0 2022-07-20T02:34:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2cf3e700 1 --2- 172.21.15.26:0/861607106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd280ba5b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fcd24001780 tx=0x7fcd18006df0).ready entity=mon.1 client_cookie=23a75ab1f1c9f553 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd200019e0 con 0x7fcd280b46b0 2022-07-20T02:34:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd18000e90 con 0x7fcd280b46b0 2022-07-20T02:34:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd17fff700 1 -- 172.21.15.26:0/861607106 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd240015e0 con 0x7fcd280b46b0 2022-07-20T02:34:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- 172.21.15.26:0/861607106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 msgr2=0x7fcd280ba5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 --2- 172.21.15.26:0/861607106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd280ba5b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fcd24001780 tx=0x7fcd18006df0).stop 2022-07-20T02:34:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- 172.21.15.26:0/861607106 shutdown_connections 2022-07-20T02:34:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 --2- 172.21.15.26:0/861607106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd280ba5b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- 172.21.15.26:0/861607106 >> 172.21.15.26:0/861607106 conn(0x7fcd2805fa50 msgr2=0x7fcd28061640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- 172.21.15.26:0/861607106 shutdown_connections 2022-07-20T02:34:53.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.235+0000 7fcd2e741700 1 -- 172.21.15.26:0/861607106 wait complete. 2022-07-20T02:34:53.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 Processor -- start 2022-07-20T02:34:53.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- start start 2022-07-20T02:34:53.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd28143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd280b5d90 0x7fcd280b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28142360 con 0x7fcd28146a20 2022-07-20T02:34:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd281414c0 con 0x7fcd280b46b0 2022-07-20T02:34:53.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28141d70 con 0x7fcd280b5d90 2022-07-20T02:34:53.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43268/0 (socket says 172.21.15.26:43268) 2022-07-20T02:34:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 -- 172.21.15.26:0/3291619087 learned_addr learned my addr 172.21.15.26:0/3291619087 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:53.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2d73f700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd28143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2cf3e700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd280b5d90 0x7fcd280b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 -- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd280b5d90 msgr2=0x7fcd280b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd280b5d90 0x7fcd280b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 -- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 msgr2=0x7fcd28143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd28143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 -- 172.21.15.26:0/3291619087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1800b180 con 0x7fcd28146a20 2022-07-20T02:34:53.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2d73f700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd28143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:53.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2df40700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fcd20001360 tx=0x7fcd20001aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd20004ab0 con 0x7fcd28146a20 2022-07-20T02:34:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd20004c10 con 0x7fcd28146a20 2022-07-20T02:34:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd281416f0 con 0x7fcd28146a20 2022-07-20T02:34:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd20007170 con 0x7fcd28146a20 2022-07-20T02:34:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.239+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd28142010 con 0x7fcd28146a20 2022-07-20T02:34:53.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd280b6290 con 0x7fcd28146a20 2022-07-20T02:34:53.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd2001b070 con 0x7fcd28146a20 2022-07-20T02:34:53.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd167fc700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 0x7fcd08055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcd2006baa0 con 0x7fcd28146a20 2022-07-20T02:34:53.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd2d73f700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 0x7fcd08055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.243+0000 7fcd2d73f700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 0x7fcd08055f30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fcd2400c000 tx=0x7fcd24008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.251+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd200427c0 con 0x7fcd28146a20 2022-07-20T02:34:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd280574e0 con 0x7fcd28146a20 2022-07-20T02:34:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd167fc700 1 -- 172.21.15.26:0/3291619087 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcd20045de0 con 0x7fcd28146a20 2022-07-20T02:34:53.387 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 msgr2=0x7fcd08055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 0x7fcd08055f30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fcd2400c000 tx=0x7fcd24008040).stop 2022-07-20T02:34:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 msgr2=0x7fcd281427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.375+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fcd20001360 tx=0x7fcd20001aa0).stop 2022-07-20T02:34:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 shutdown_connections 2022-07-20T02:34:53.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd28146a20 0x7fcd281427f0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcd08053a70 0x7fcd08055f30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd280b5d90 0x7fcd280b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 --2- 172.21.15.26:0/3291619087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd280b46b0 0x7fcd28143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 >> 172.21.15.26:0/3291619087 conn(0x7fcd2805fa50 msgr2=0x7fcd280b9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 shutdown_connections 2022-07-20T02:34:53.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.379+0000 7fcd2e741700 1 -- 172.21.15.26:0/3291619087 wait complete. 2022-07-20T02:34:53.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:34:53.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 Processor -- start 2022-07-20T02:34:53.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 -- start start 2022-07-20T02:34:53.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f03c005cfe0 0x7f03c014b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c014bbb0 con 0x7f03c0060c90 2022-07-20T02:34:53.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c01477c0 con 0x7f03c005cfe0 2022-07-20T02:34:53.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00611d0 con 0x7f03c0099d20 2022-07-20T02:34:53.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c6226700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f03c005cfe0 0x7f03c014b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c6226700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:36560/0 (socket says 172.21.15.26:36560) 2022-07-20T02:34:53.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c6226700 1 -- 172.21.15.26:0/2094268136 learned_addr learned my addr 172.21.15.26:0/2094268136 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:53.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5224700 1 --1- 172.21.15.26:0/2094268136 >> v1:172.21.15.120:6789/0 conn(0x7f03c0060c90 0x7f03c01480f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56766/0 (socket says 172.21.15.26:56766) 2022-07-20T02:34:53.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 -- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 msgr2=0x7f03c00b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:53.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 -- 172.21.15.26:0/2094268136 >> v1:172.21.15.120:6789/0 conn(0x7f03c0060c90 legacy=0x7f03c01480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 -- 172.21.15.26:0/2094268136 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c01479b0 con 0x7f03c005cfe0 2022-07-20T02:34:53.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c6226700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:53.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c5a25700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c005cfe0 0x7f03c014b6c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f03bc0032f0 tx=0x7f03bc0091d0).ready entity=mon.1 client_cookie=78779b9b87355143 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c4a23700 1 -- 172.21.15.26:0/2094268136 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03bc00b6c0 con 0x7f03c005cfe0 2022-07-20T02:34:53.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c4a23700 1 -- 172.21.15.26:0/2094268136 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc00b820 con 0x7f03c005cfe0 2022-07-20T02:34:53.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c4a23700 1 -- 172.21.15.26:0/2094268136 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03bc00b990 con 0x7f03c005cfe0 2022-07-20T02:34:53.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 -- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c005cfe0 msgr2=0x7f03c014b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.531+0000 7f03c7c89700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c005cfe0 0x7f03c014b6c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f03bc0032f0 tx=0x7f03bc0091d0).stop 2022-07-20T02:34:53.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- 172.21.15.26:0/2094268136 shutdown_connections 2022-07-20T02:34:53.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0099d20 0x7f03c00b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 --2- 172.21.15.26:0/2094268136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c005cfe0 0x7f03c014b6c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- 172.21.15.26:0/2094268136 >> 172.21.15.26:0/2094268136 conn(0x7f03c00b21f0 msgr2=0x7f03c00b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- 172.21.15.26:0/2094268136 shutdown_connections 2022-07-20T02:34:53.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- 172.21.15.26:0/2094268136 wait complete. 2022-07-20T02:34:53.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 Processor -- start 2022-07-20T02:34:53.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- start start 2022-07-20T02:34:53.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0060c90 0x7f03c015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c0099d20 0x7f03c015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00ae2b0 con 0x7f03c005cfe0 2022-07-20T02:34:53.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00ae8e0 con 0x7f03c0099d20 2022-07-20T02:34:53.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c7c89700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0063450 con 0x7f03c0060c90 2022-07-20T02:34:53.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c5a25700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c5a25700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43278/0 (socket says 172.21.15.26:43278) 2022-07-20T02:34:53.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c5a25700 1 -- 172.21.15.26:0/758948904 learned_addr learned my addr 172.21.15.26:0/758948904 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:53.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c5224700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0060c90 0x7f03c015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.535+0000 7f03c6226700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c0099d20 0x7f03c015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 -- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0060c90 msgr2=0x7f03c015a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0060c90 0x7f03c015a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 -- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c0099d20 msgr2=0x7f03c015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c0099d20 0x7f03c015aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 -- 172.21.15.26:0/758948904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b800b1e0 con 0x7f03c005cfe0 2022-07-20T02:34:53.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c5a25700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f03bc0032f0 tx=0x7f03bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03bc011d70 con 0x7f03c005cfe0 2022-07-20T02:34:53.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc021440 con 0x7f03c005cfe0 2022-07-20T02:34:53.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03bc0217a0 con 0x7f03c005cfe0 2022-07-20T02:34:53.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc009470 con 0x7f03c005cfe0 2022-07-20T02:34:53.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c00aeb10 con 0x7f03c005cfe0 2022-07-20T02:34:53.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c015a390 con 0x7f03c005cfe0 2022-07-20T02:34:53.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.539+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f03bc027020 con 0x7f03c005cfe0 2022-07-20T02:34:53.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.543+0000 7f03aeffd700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 0x7f03a0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.543+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f03bc06d3a0 con 0x7f03c005cfe0 2022-07-20T02:34:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.543+0000 7f03c5224700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 0x7f03a0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.543+0000 7f03c5224700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 0x7f03a0055e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f03b000d000 tx=0x7f03b0007b80).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.547+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03bc044160 con 0x7f03c005cfe0 2022-07-20T02:34:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.675+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03c00aee90 con 0x7f03c005cfe0 2022-07-20T02:34:53.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.675+0000 7f03aeffd700 1 -- 172.21.15.26:0/758948904 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f03bc0150a0 con 0x7f03c005cfe0 2022-07-20T02:34:53.684 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:34:53.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 msgr2=0x7f03a0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 0x7f03a0055e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f03b000d000 tx=0x7f03b0007b80).stop 2022-07-20T02:34:53.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 msgr2=0x7f03c014a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f03bc0032f0 tx=0x7f03bc016000).stop 2022-07-20T02:34:53.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 shutdown_connections 2022-07-20T02:34:53.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f03a0053950 0x7f03a0055e10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03c0099d20 0x7f03c015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c0060c90 0x7f03c015a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 --2- 172.21.15.26:0/758948904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03c005cfe0 0x7f03c014a880 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 >> 172.21.15.26:0/758948904 conn(0x7f03c00b21f0 msgr2=0x7f03c0145660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 shutdown_connections 2022-07-20T02:34:53.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.679+0000 7f03c7c89700 1 -- 172.21.15.26:0/758948904 wait complete. 2022-07-20T02:34:53.754 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:34:53.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:34:53.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.887+0000 7fe98801f700 1 Processor -- start 2022-07-20T02:34:53.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.887+0000 7fe98801f700 1 -- start start 2022-07-20T02:34:53.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe98005cb50 0x7fe98005be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9800ae890 con 0x7fe9800a9020 2022-07-20T02:34:53.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9800aeb30 con 0x7fe9800b02a0 2022-07-20T02:34:53.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9801478a0 con 0x7fe98005cb50 2022-07-20T02:34:53.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9855ba700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe9800a9020 0x7fe98005c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42568/0 (socket says 172.21.15.26:42568) 2022-07-20T02:34:53.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9855ba700 1 -- 172.21.15.26:0/2763501142 learned_addr learned my addr 172.21.15.26:0/2763501142 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:34:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9865bc700 1 --2- 172.21.15.26:0/2763501142 >> v2:172.21.15.120:3300/0 conn(0x7fe98005cb50 0x7fe98005be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760020630 0 0) 0x7fe9800ae890 con 0x7fe9800a9020 2022-07-20T02:34:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968003260 con 0x7fe9800a9020 2022-07-20T02:34:53.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590243204 0 0) 0x7fe9800aeb30 con 0x7fe9800b02a0 2022-07-20T02:34:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9800ae890 con 0x7fe9800b02a0 2022-07-20T02:34:53.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9865bc700 1 -- 172.21.15.26:0/2763501142 >> v1:172.21.15.120:6789/0 conn(0x7fe9800b02a0 legacy=0x7fe980147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9865bc700 1 -- 172.21.15.26:0/2763501142 >> v1:172.21.15.26:6789/0 conn(0x7fe9800a9020 legacy=0x7fe98005c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433247445 0 0) 0x7fe968003260 con 0x7fe9800a9020 2022-07-20T02:34:53.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9865bc700 1 -- 172.21.15.26:0/2763501142 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980147b40 con 0x7fe98005cb50 2022-07-20T02:34:53.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe9865bc700 1 --2- 172.21.15.26:0/2763501142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe98005cb50 0x7fe98005be30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fe970001110 tx=0x7fe97c006df0).ready entity=mon.1 client_cookie=a1fa55f06c595ab6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe970001660 con 0x7fe98005cb50 2022-07-20T02:34:53.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c000e90 con 0x7fe98005cb50 2022-07-20T02:34:53.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe984db9700 1 -- 172.21.15.26:0/2763501142 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe97c012460 con 0x7fe98005cb50 2022-07-20T02:34:53.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- 172.21.15.26:0/2763501142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe98005cb50 msgr2=0x7fe98005be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 --2- 172.21.15.26:0/2763501142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe98005cb50 0x7fe98005be30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fe970001110 tx=0x7fe97c006df0).stop 2022-07-20T02:34:53.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- 172.21.15.26:0/2763501142 shutdown_connections 2022-07-20T02:34:53.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 --2- 172.21.15.26:0/2763501142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe98005cb50 0x7fe98005be30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- 172.21.15.26:0/2763501142 >> 172.21.15.26:0/2763501142 conn(0x7fe9800606e0 msgr2=0x7fe980060af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:53.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- 172.21.15.26:0/2763501142 shutdown_connections 2022-07-20T02:34:53.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.891+0000 7fe98801f700 1 -- 172.21.15.26:0/2763501142 wait complete. 2022-07-20T02:34:53.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 Processor -- start 2022-07-20T02:34:53.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- start start 2022-07-20T02:34:53.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe98005cb50 0x7fe9800ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9800b02a0 0x7fe980149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:53.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800ab4e0 con 0x7fe9800a9020 2022-07-20T02:34:53.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800ac560 con 0x7fe9800b02a0 2022-07-20T02:34:53.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800ab740 con 0x7fe98005cb50 2022-07-20T02:34:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43294/0 (socket says 172.21.15.26:43294) 2022-07-20T02:34:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 -- 172.21.15.26:0/592794887 learned_addr learned my addr 172.21.15.26:0/592794887 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe985dbb700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe98005cb50 0x7fe9800ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9865bc700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9800b02a0 0x7fe980149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 -- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe98005cb50 msgr2=0x7fe9800ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe98005cb50 0x7fe9800ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 -- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9800b02a0 msgr2=0x7fe980149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9800b02a0 0x7fe980149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 -- 172.21.15.26:0/592794887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c00b180 con 0x7fe9800a9020 2022-07-20T02:34:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe9855ba700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7fe970000fe0 tx=0x7fe970001970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe970007170 con 0x7fe9800a9020 2022-07-20T02:34:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9700072d0 con 0x7fe9800a9020 2022-07-20T02:34:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9800ab9e0 con 0x7fe9800a9020 2022-07-20T02:34:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe970004710 con 0x7fe9800a9020 2022-07-20T02:34:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.895+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98014d4a0 con 0x7fe9800a9020 2022-07-20T02:34:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9801486f0 con 0x7fe9800a9020 2022-07-20T02:34:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe97001b070 con 0x7fe9800a9020 2022-07-20T02:34:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe96effd700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 0x7fe960055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe97006b670 con 0x7fe9800a9020 2022-07-20T02:34:54.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe985dbb700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 0x7fe960055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:54.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.899+0000 7fe985dbb700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 0x7fe960055ed0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fe97800d000 tx=0x7fe978007d90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:54.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:53.907+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe970042430 con 0x7fe9800a9020 2022-07-20T02:34:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.067+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9800574e0 con 0x7fe9800a9020 2022-07-20T02:34:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.071+0000 7fe96effd700 1 -- 172.21.15.26:0/592794887 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7fe970045a50 con 0x7fe9800a9020 2022-07-20T02:34:54.077 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:54.077 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:34:54.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.071+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 msgr2=0x7fe960055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.071+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 0x7fe960055ed0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fe97800d000 tx=0x7fe978007d90).stop 2022-07-20T02:34:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.071+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 msgr2=0x7fe98014cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7fe970000fe0 tx=0x7fe970001970).stop 2022-07-20T02:34:54.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 shutdown_connections 2022-07-20T02:34:54.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9800a9020 0x7fe98014cfb0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:54.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe960053a10 0x7fe960055ed0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:54.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9800b02a0 0x7fe980149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:54.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 --2- 172.21.15.26:0/592794887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe98005cb50 0x7fe9800ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:54.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 >> 172.21.15.26:0/592794887 conn(0x7fe9800606e0 msgr2=0x7fe9800b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:54.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 shutdown_connections 2022-07-20T02:34:54.126 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:54.075+0000 7fe98801f700 1 -- 172.21.15.26:0/592794887 wait complete. 2022-07-20T02:34:54.127 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:34:54.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:34:40.117843+0000', 'modified': '2022-07-20T02:34:41.123008+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8139': {'gid': 8139, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1171289829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 1171289829}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1171289829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:34:54.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:34:54.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T02:34:54.639 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8139,"want_state":"up:active","state":"up:active","rank_uptime":14.47753625,"mdsmap_epoch":72,"osdmap_epoch":140,"osdmap_epoch_barrier":138,"uptime":19.537902324000001} 2022-07-20T02:34:54.643 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8139, "mdsmap_epoch": 72, "osdmap_epoch": 140, "osdmap_epoch_barrier": 138, "rank_uptime": 14.47753625, "state": "up:active", "uptime": 19.537902324, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:34:54.644 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:34:54.645 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:34:54.652 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:34:54.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:54.655 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:54.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:54.785 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:54.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:54.855 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:54.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:54.856 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:54.891 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:34:54.913 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:54.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:34:55.015 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:34:55.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:34:55.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:34:55.007+0000 7f003de99080 -1 init, newargv = 0x55da253daba0 newargc=9ceph-fuse[23023]: starting ceph client 2022-07-20T02:34:55.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr: 2022-07-20T02:34:55.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23023]: starting fuse 2022-07-20T02:34:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:34:55.072 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:34:55.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.073 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:34:55.123 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:34:55.123 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:34:55.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:34:55.125 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:34:55.127 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:34:55.127 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:34:55.128 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:34:55.129 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:55.129 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:34:55.130 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:34:55.130 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:34:55.130 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:55.131 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:34:55.131 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:34:55.132 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:34:55.132 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:34:55.132 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:55.133 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:34:55.133 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:34:55.135 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:34:55.135 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:34:55.137 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:34:55.137 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:34:55.145 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:34:55.148 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:34:55.206 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.23023.asok 2022-07-20T02:34:55.227 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23023.asok 2022-07-20T02:34:55.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23023.asok status 2022-07-20T02:34:55.377 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:34:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:34:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:34:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:34:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:34:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:34:55.409 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:34:55.410 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "23023", 2022-07-20T02:34:55.410 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:34:55.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:55.412 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:34:55.413 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:34:55.413 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 7097, 2022-07-20T02:34:55.414 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:34:55.420 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:34:55.423 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:34:55.423 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 7097 2022-07-20T02:34:55.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:55.424 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:55.424 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:55.425 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:55.425 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 712409151 2022-07-20T02:34:55.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:34:55.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:55.431 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:34:55.432 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:34:55.433 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:34:55.434 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 712409151 2022-07-20T02:34:55.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:34:55.435 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.7097 172.21.15.176:0/712409151", 2022-07-20T02:34:55.435 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/712409151", 2022-07-20T02:34:55.449 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:34:55.450 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 72, 2022-07-20T02:34:55.450 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 140, 2022-07-20T02:34:55.450 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:34:55.451 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:34:55.451 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:34:55.452 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:34:55.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.463 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:55.464 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:55.485 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:55.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:55.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.486 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:55.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:55.562 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:55.565 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:55.585 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:34:55.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:55.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T02:34:55.597 DEBUG:teuthology.orchestra.run.smithi159:> import time 2022-07-20T02:34:55.598 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:34:55.599 DEBUG:teuthology.orchestra.run.smithi159:> with open("/home/ubuntu/cephtest/mnt.0/testfile", '"'"'w'"'"') as f: 2022-07-20T02:34:55.630 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content'"'"') 2022-07-20T02:34:55.631 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-20T02:34:55.636 DEBUG:teuthology.orchestra.run.smithi159:> f.write('"'"'content2'"'"') 2022-07-20T02:34:55.636 DEBUG:teuthology.orchestra.run.smithi159:> while True: 2022-07-20T02:34:55.637 DEBUG:teuthology.orchestra.run.smithi159:> time.sleep(1) 2022-07-20T02:34:55.648 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:34:55.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T02:34:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-20T02:34:55.725 INFO:teuthology.orchestra.run.smithi159.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/testfile': No such file or directory 2022-07-20T02:34:56.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T02:34:56.792 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T02:34:56.793 DEBUG:tasks.cephfs.mount:File testfile became visible from 0 after 1s 2022-07-20T02:34:56.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo ls /home/ubuntu/cephtest/mnt.1/testfile 2022-07-20T02:34:56.845 INFO:teuthology.orchestra.run.smithi176.stdout:/home/ubuntu/cephtest/mnt.1/testfile 2022-07-20T02:34:56.845 DEBUG:tasks.cephfs.mount:File testfile became visible from 1 after 0s 2022-07-20T02:34:56.867 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T02:34:56.868 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest/mnt.1 && sudo cat testfile 2022-07-20T02:34:56.938 INFO:teuthology.orchestra.run.smithi176.stdout:content 2022-07-20T02:34:56.954 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:34:56.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:56.996 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:57.026 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:57.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:34:57.142 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:34:57.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:57.156 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:57.157 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:57.201 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:34:57.202 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:34:57.202 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:34:57.203 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:34:57.204 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:57.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23351]: fuse finished with error 0 and tester_r 0 2022-07-20T02:34:57.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:57.265 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:57.280 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:57.346 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:34:57.350 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:34:57.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:57.365 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:34:57.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:57.402 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:57.403 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:57.533 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:34:57.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:34:57.535 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:34:57.536 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:34:57.547 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:57.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23023]: fuse finished with error 0 and tester_r 0 2022-07-20T02:34:57.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:34:57.573 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:34:57.574 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:57.661 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:34:57.667 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:34:57.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:34:57.669 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:34:57.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:34:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 Processor -- start 2022-07-20T02:34:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 -- start start 2022-07-20T02:34:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404005d180 0x7f40400afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4040060f00 0x7f40400b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404005d560 con 0x7f40400609f0 2022-07-20T02:34:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404014bd60 con 0x7f4040060f00 2022-07-20T02:34:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f404750b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4040098b80 con 0x7f404005d180 2022-07-20T02:34:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4040060f00 0x7f40400b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4040060f00 0x7f40400b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43302/0 (socket says 172.21.15.26:43302) 2022-07-20T02:34:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 -- 172.21.15.26:0/803655552 learned_addr learned my addr 172.21.15.26:0/803655552 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4046509700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404005d180 0x7f40400afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045507700 1 -- 172.21.15.26:0/803655552 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383889820 0 0) 0x7f404005d560 con 0x7f40400609f0 2022-07-20T02:34:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045507700 1 -- 172.21.15.26:0/803655552 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4028003260 con 0x7f40400609f0 2022-07-20T02:34:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 -- 172.21.15.26:0/803655552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404005d180 msgr2=0x7f40400afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:34:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404005d180 0x7f40400afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 -- 172.21.15.26:0/803655552 >> v1:172.21.15.120:6789/0 conn(0x7f40400609f0 legacy=0x7f404005f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:57.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 -- 172.21.15.26:0/803655552 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400a8ff0 con 0x7f4040060f00 2022-07-20T02:34:57.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.827+0000 7f4045d08700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040060f00 0x7f40400b03a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f403c0032f0 tx=0x7f403c009290).ready entity=mon.0 client_cookie=f46ac78ecaffa92f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:57.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f4045507700 1 -- 172.21.15.26:0/803655552 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403c00b6c0 con 0x7f4040060f00 2022-07-20T02:34:57.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f4045507700 1 -- 172.21.15.26:0/803655552 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c00b820 con 0x7f4040060f00 2022-07-20T02:34:57.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f4045507700 1 -- 172.21.15.26:0/803655552 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403c00b990 con 0x7f4040060f00 2022-07-20T02:34:57.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- 172.21.15.26:0/803655552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040060f00 msgr2=0x7f40400b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:57.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040060f00 0x7f40400b03a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f403c0032f0 tx=0x7f403c009290).stop 2022-07-20T02:34:57.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- 172.21.15.26:0/803655552 shutdown_connections 2022-07-20T02:34:57.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040060f00 0x7f40400b03a0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 --2- 172.21.15.26:0/803655552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f404005d180 0x7f40400afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- 172.21.15.26:0/803655552 >> 172.21.15.26:0/803655552 conn(0x7f40400b2080 msgr2=0x7f40400b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:34:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- 172.21.15.26:0/803655552 shutdown_connections 2022-07-20T02:34:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- 172.21.15.26:0/803655552 wait complete. 2022-07-20T02:34:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 Processor -- start 2022-07-20T02:34:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.831+0000 7f404750b700 1 -- start start 2022-07-20T02:34:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40400609f0 0x7f404015a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 0x7f4040146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4040159f50 con 0x7f404005d180 2022-07-20T02:34:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40400ae630 con 0x7f40400609f0 2022-07-20T02:34:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40400a9d40 con 0x7f4040143b20 2022-07-20T02:34:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43306/0 (socket says 172.21.15.26:43306) 2022-07-20T02:34:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 -- 172.21.15.26:0/2128432759 learned_addr learned my addr 172.21.15.26:0/2128432759 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:34:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4045d08700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40400609f0 0x7f404015a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046d0a700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 0x7f4040146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 -- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 msgr2=0x7f4040146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 0x7f4040146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 -- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40400609f0 msgr2=0x7f404015a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:34:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40400609f0 0x7f404015a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:34:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 -- 172.21.15.26:0/2128432759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c009530 con 0x7f404005d180 2022-07-20T02:34:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046d0a700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 0x7f4040146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:34:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f4046509700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f403000c000 tx=0x7f4030006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4030010780 con 0x7f404005d180 2022-07-20T02:34:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40300108e0 con 0x7f404005d180 2022-07-20T02:34:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404015a180 con 0x7f404005d180 2022-07-20T02:34:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4030010c40 con 0x7f404005d180 2022-07-20T02:34:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.835+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40400a99c0 con 0x7f404005d180 2022-07-20T02:34:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404015ae50 con 0x7f404005d180 2022-07-20T02:34:57.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f403001c460 con 0x7f404005d180 2022-07-20T02:34:57.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f402f7fe700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 0x7f4020055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:34:57.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f403006c410 con 0x7f404005d180 2022-07-20T02:34:57.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f4045d08700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 0x7f4020055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:34:57.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.839+0000 7f4045d08700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 0x7f4020055e10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f403c01b000 tx=0x7f403c016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:34:57.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.843+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f404015ae50 con 0x7f404005d180 2022-07-20T02:34:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.971+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f404015ae50 con 0x7f404005d180 2022-07-20T02:34:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.971+0000 7f402f7fe700 1 -- 172.21.15.26:0/2128432759 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v140) v1 ==== 81+0+34099 (secure 0 0 0) 0x7f404015ae50 con 0x7f404005d180 2022-07-20T02:34:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:34:58.044 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:34:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 140, 2022-07-20T02:34:58.083 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:34:58.083 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:34:58.084 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:34:42.132316+0000", 2022-07-20T02:34:58.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:58.085 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:34:58.085 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:34:58.086 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:34:58.086 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:34:58.087 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:34:58.087 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:34:58.088 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:34:58.088 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:34:58.088 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:58.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:34:58.089 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:34:58.089 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:34:58.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:34:58.090 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:34:58.091 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 26, 2022-07-20T02:34:58.091 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:34:58.092 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:34:58.092 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:34:58.093 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:34:58.093 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:34:58.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.094 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:34:58.094 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:34:58.095 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:34:58.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:58.096 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:58.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:58.097 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:58.097 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:58.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:58.182 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:58.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:58.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:34:58.184 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:34:58.184 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:34:58.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:34:58.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:34:58.185 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:58.193 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:58.193 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:58.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:58.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:58.195 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:58.195 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:58.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.203 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:34:58.203 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:58.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:58.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:58.204 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:58.205 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:58.205 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:58.206 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:58.206 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:58.207 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:58.207 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:58.207 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:58.208 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:58.208 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:58.209 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:58.209 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:58.209 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:58.227 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:58.227 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:58.228 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:58.228 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:58.229 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:58.229 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:58.229 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:58.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.244 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:58.245 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:58.246 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:58.247 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:58.248 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:58.249 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:58.255 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:34:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:34:58.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:58.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:34:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.271 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:34:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:34:58.272 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:34:58.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:58.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:58.275 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:58.276 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:58.277 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:58.278 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:58.299 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:58.300 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:58.301 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:58.304 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.305 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:34:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:58.312 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:58.313 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:58.314 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:58.342 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:58.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:58.344 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:58.363 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:58.364 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:58.365 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:58.376 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:58.377 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:58.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.379 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:58.396 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:58.397 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:58.398 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:58.399 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:58.400 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:34:58.401 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:34:58.410 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.411 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 25, 2022-07-20T02:34:58.412 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:34:58.425 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:34:38.295386+0000", 2022-07-20T02:34:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:58.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:58.427 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:58.428 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:58.438 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:58.439 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:58.439 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:58.440 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:58.440 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:58.440 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:58.441 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:58.443 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:58.444 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:58.444 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:58.444 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:58.445 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:58.458 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:58.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.459 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "139", 2022-07-20T02:34:58.460 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:58.460 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:58.460 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:58.461 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:58.461 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:58.462 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:58.463 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:58.463 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:58.464 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:58.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:58.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:58.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:58.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:58.466 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:58.466 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:58.466 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:58.478 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:58.479 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:58.493 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:58.493 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:58.494 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:58.494 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:58.495 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:58.495 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:58.496 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:58.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:58.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:58.497 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:58.513 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:58.514 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:58.515 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:34:58.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:34:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:34:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:34:58.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:34:58.519 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:34:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 26, 2022-07-20T02:34:58.523 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:34:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:34:39.171843+0000", 2022-07-20T02:34:58.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:34:58.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:34:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:34:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:34:58.551 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:34:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:34:58.552 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:34:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:34:58.553 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:34:58.554 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:34:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:34:58.578 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:34:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:34:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:34:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:34:58.579 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:34:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:34:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:34:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:34:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:34:58.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "140", 2022-07-20T02:34:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:34:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:34:58.581 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:34:58.582 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:34:58.599 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:34:58.599 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:34:58.600 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:34:58.600 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:34:58.601 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:34:58.601 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:34:58.601 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:34:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:34:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:34:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:34:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:34:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:34:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:34:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:34:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:34:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:34:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:34:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:34:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:34:58.674 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:34:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:34:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:34:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.675 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:34:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:34:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:34:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:34:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:34:58.676 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:34:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:34:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:34:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:34:58.677 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:34:58.678 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:34:58.681 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:34:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:34:58.682 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:34:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:34:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.684 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:34:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.686 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:34:58.687 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:58.688 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:58.689 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:58.690 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:58.691 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:34:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:58.692 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:34:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:34:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.715 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:34:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:34:58.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.739 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:58.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:34:58.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:34:58.782 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.783 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:58.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:34:58.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:34:58.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:34:58.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.801 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:34:58.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:34:58.802 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:34:58.802 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:34:58.803 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:58.803 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:58.803 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:58.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.805 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:58.807 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:34:58.808 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:58.808 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:58.809 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:58.812 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:58.815 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:58.815 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:34:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:34:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:34:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:34:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:34:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:34:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:34:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:34:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:34:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:34:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:34:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:34:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:34:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:34:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:34:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:34:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:34:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:34:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:34:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:34:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:34:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:34:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:34:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:58.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:58.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:58.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:34:58.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:34:58.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:58.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:58.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:34:59.000 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:34:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:34:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:34:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.015 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:59.016 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:34:59.055 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:59.056 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:59.056 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:59.056 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:59.057 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:59.057 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:59.057 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:59.057 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:34:59.078 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:59.079 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:59.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:59.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:34:59.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:34:59.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:59.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:34:59.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:34:59.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.110 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:59.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:34:59.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:34:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:34:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:34:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:34:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:34:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:34:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:34:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:34:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.167 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.168 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:59.170 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:34:59.170 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:59.171 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:59.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:59.172 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:59.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:59.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:34:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:34:59.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:34:59.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.259 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:59.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:34:59.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:34:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:34:59.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:34:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:34:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:34:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:34:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:34:59.463 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:34:59.464 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:34:59.464 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:34:59.465 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.466 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.466 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.468 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:59.469 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:34:59.469 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:59.470 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:59.471 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:59.472 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:59.482 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:59.482 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:59.482 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:59.484 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:34:59.485 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:59.485 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:59.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:59.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.486 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:34:59.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:34:59.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.517 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:59.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:34:59.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:34:59.520 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.521 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:59.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:34:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:34:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:34:59.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:34:59.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:34:59.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:34:59.551 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:34:59.551 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:34:59.552 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:34:59.552 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.552 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.552 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.553 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:59.562 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:34:59.565 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:59.565 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:59.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:59.566 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:59.566 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:59.566 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:59.567 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:59.567 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:34:59.567 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:59.568 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:59.568 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:34:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:34:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:34:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:34:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:34:59.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:34:59.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.633 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:59.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:34:59.646 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:34:59.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:34:59.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:34:59.648 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:34:59.648 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:34:59.649 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:34:59.649 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.681 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.682 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.683 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:59.683 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:34:59.683 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:34:59.684 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:34:59.684 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:34:59.684 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:34:59.684 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:34:59.685 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:34:59.685 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:34:59.685 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:34:59.685 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:34:59.699 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:34:59.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:34:59.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:34:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:34:59.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.713 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:34:59.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:34:59.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:34:59.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.728 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:34:59.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:34:59.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:34:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.746 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:34:59.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:34:59.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:34:59.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:34:59.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.748 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:34:59.748 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:34:59.748 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:34:59.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.758 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:34:59.758 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:34:59.758 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:34:59.759 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:34:59.759 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:34:59.759 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:34:59.759 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:34:59.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:34:59.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.760 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:59.760 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:34:59.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.761 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:34:59.761 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.761 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.762 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.762 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.762 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:34:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:34:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.803 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.804 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.804 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.804 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.804 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:34:59.804 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.805 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:34:59.805 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.816 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.816 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.817 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.817 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.818 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:34:59.818 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.827 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:34:59.827 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.827 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.827 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.828 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.828 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.828 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:34:59.828 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:34:59.847 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.891 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:34:59.892 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:34:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.907 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.908 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:34:59.909 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.924 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:34:59.927 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.928 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.929 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.929 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.931 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:34:59.931 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:34:59.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:34:59.933 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:34:59.934 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:34:59.934 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:34:59.935 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:34:59.935 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:34:59.935 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:34:59.936 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:34:59.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:34:59.937 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:34:59.937 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:34:59.938 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:34:59.938 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:34:59.938 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:34:59.939 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:34:59.939 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:34:59.940 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:34:59.940 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:34:59.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:59.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:34:59.941 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:34:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:34:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:34:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:34:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:34:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:34:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:34:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:34:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:34:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:34:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:34:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:35:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:35:00.003 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:35:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:35:00.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:35:00.005 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:35:00.006 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:00.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 msgr2=0x7f4020055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 0x7f4020055e10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f403c01b000 tx=0x7f403c016000).stop 2022-07-20T02:35:00.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 msgr2=0x7f4040143630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f403000c000 tx=0x7f4030006df0).stop 2022-07-20T02:35:00.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 shutdown_connections 2022-07-20T02:35:00.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4020053950 0x7f4020055e10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040143b20 0x7f4040146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40400609f0 0x7f404015a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 --2- 172.21.15.26:0/2128432759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f404005d180 0x7f4040143630 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 >> 172.21.15.26:0/2128432759 conn(0x7f40400b2080 msgr2=0x7f4040144950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:00.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 shutdown_connections 2022-07-20T02:35:00.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:34:57.975+0000 7f404750b700 1 -- 172.21.15.26:0/2128432759 wait complete. 2022-07-20T02:35:00.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:00.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 Processor -- start 2022-07-20T02:35:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 -- start start 2022-07-20T02:35:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9c6005d180 0x7f9c600afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9c60060f00 0x7f9c600b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6005d560 con 0x7f9c600609f0 2022-07-20T02:35:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6014bd60 con 0x7f9c60060f00 2022-07-20T02:35:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c6555b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c60098b80 con 0x7f9c6005d180 2022-07-20T02:35:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9c60060f00 0x7f9c600b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9c60060f00 0x7f9c600b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43314/0 (socket says 172.21.15.26:43314) 2022-07-20T02:35:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 -- 172.21.15.26:0/4149686277 learned_addr learned my addr 172.21.15.26:0/4149686277 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c64d5a700 1 --1- 172.21.15.26:0/4149686277 >> v1:172.21.15.26:6789/0 conn(0x7f9c600609f0 0x7f9c6005f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42598/0 (socket says 172.21.15.26:42598) 2022-07-20T02:35:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5ffff700 1 --2- 172.21.15.26:0/4149686277 >> v2:172.21.15.120:3300/0 conn(0x7f9c6005d180 0x7f9c600afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 -- 172.21.15.26:0/4149686277 >> v2:172.21.15.120:3300/0 conn(0x7f9c6005d180 msgr2=0x7f9c600afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 --2- 172.21.15.26:0/4149686277 >> v2:172.21.15.120:3300/0 conn(0x7f9c6005d180 0x7f9c600afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 -- 172.21.15.26:0/4149686277 >> v1:172.21.15.26:6789/0 conn(0x7f9c600609f0 legacy=0x7f9c6005f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 -- 172.21.15.26:0/4149686277 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c600a8ff0 con 0x7f9c60060f00 2022-07-20T02:35:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5effd700 1 -- 172.21.15.26:0/4149686277 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080781539 0 0) 0x7f9c6005d560 con 0x7f9c600609f0 2022-07-20T02:35:00.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5f7fe700 1 --2- 172.21.15.26:0/4149686277 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c60060f00 0x7f9c600b03a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f9c6009afb0 tx=0x7f9c50006eb0).ready entity=mon.0 client_cookie=608fb567735fd010 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5effd700 1 -- 172.21.15.26:0/4149686277 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5000d610 con 0x7f9c60060f00 2022-07-20T02:35:00.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5effd700 1 -- 172.21.15.26:0/4149686277 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c50000e90 con 0x7f9c60060f00 2022-07-20T02:35:00.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.131+0000 7f9c5effd700 1 -- 172.21.15.26:0/4149686277 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5000d960 con 0x7f9c60060f00 2022-07-20T02:35:00.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- 172.21.15.26:0/4149686277 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c60060f00 msgr2=0x7f9c600b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- 172.21.15.26:0/4149686277 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c60060f00 0x7f9c600b03a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f9c6009afb0 tx=0x7f9c50006eb0).stop 2022-07-20T02:35:00.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- 172.21.15.26:0/4149686277 shutdown_connections 2022-07-20T02:35:00.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- 172.21.15.26:0/4149686277 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c60060f00 0x7f9c600b03a0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- 172.21.15.26:0/4149686277 >> v2:172.21.15.120:3300/0 conn(0x7f9c6005d180 0x7f9c600afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- 172.21.15.26:0/4149686277 >> 172.21.15.26:0/4149686277 conn(0x7f9c600b2080 msgr2=0x7f9c600b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:00.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- 172.21.15.26:0/4149686277 shutdown_connections 2022-07-20T02:35:00.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- 172.21.15.26:0/4149686277 wait complete. 2022-07-20T02:35:00.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 Processor -- start 2022-07-20T02:35:00.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- start start 2022-07-20T02:35:00.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c600609f0 0x7f9c600aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c60060f00 0x7f9c600abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c60157190 con 0x7f9c6005d180 2022-07-20T02:35:00.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c601573b0 con 0x7f9c600609f0 2022-07-20T02:35:00.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c6555b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c60144ef0 con 0x7f9c60060f00 2022-07-20T02:35:00.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43322/0 (socket says 172.21.15.26:43322) 2022-07-20T02:35:00.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 -- 172.21.15.26:0/3006914792 learned_addr learned my addr 172.21.15.26:0/3006914792 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:00.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c600609f0 0x7f9c600aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c64d5a700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c60060f00 0x7f9c600abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 -- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c60060f00 msgr2=0x7f9c600abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c60060f00 0x7f9c600abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 -- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c600609f0 msgr2=0x7f9c600aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c600609f0 0x7f9c600aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.135+0000 7f9c5ffff700 1 -- 172.21.15.26:0/3006914792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5000b1e0 con 0x7f9c6005d180 2022-07-20T02:35:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5ffff700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f9c60145080 tx=0x7f9c54008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c54011780 con 0x7f9c6005d180 2022-07-20T02:35:00.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c540118e0 con 0x7f9c6005d180 2022-07-20T02:35:00.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c60144c50 con 0x7f9c6005d180 2022-07-20T02:35:00.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c54011c40 con 0x7f9c6005d180 2022-07-20T02:35:00.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c60144820 con 0x7f9c6005d180 2022-07-20T02:35:00.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c60157810 con 0x7f9c6005d180 2022-07-20T02:35:00.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9c5401c460 con 0x7f9c6005d180 2022-07-20T02:35:00.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 0x7f9c40055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9c5406c0f0 con 0x7f9c6005d180 2022-07-20T02:35:00.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.139+0000 7f9c5f7fe700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 0x7f9c40055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.143+0000 7f9c5f7fe700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 0x7f9c40055e40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f9c5001c000 tx=0x7f9c50015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.147+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c60157810 con 0x7f9c6005d180 2022-07-20T02:35:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.315+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c60157810 con 0x7f9c6005d180 2022-07-20T02:35:00.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.315+0000 7f9c5d7fa700 1 -- 172.21.15.26:0/3006914792 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2495 (secure 0 0 0) 0x7f9c54042e50 con 0x7f9c6005d180 2022-07-20T02:35:00.379 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:00.379 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:34:41.123008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1171289829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1171289829},{"type":"v1","addr":"172.21.15.26:6837","nonce":1171289829}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:35:00.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 msgr2=0x7f9c40055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 0x7f9c40055e40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f9c5001c000 tx=0x7f9c50015000).stop 2022-07-20T02:35:00.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 msgr2=0x7f9c600a9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f9c60145080 tx=0x7f9c54008040).stop 2022-07-20T02:35:00.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 shutdown_connections 2022-07-20T02:35:00.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c40053980 0x7f9c40055e40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c60060f00 0x7f9c600abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c600609f0 0x7f9c600aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 --2- 172.21.15.26:0/3006914792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c6005d180 0x7f9c600a9e00 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 >> 172.21.15.26:0/3006914792 conn(0x7f9c600b2080 msgr2=0x7f9c60145700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:00.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 shutdown_connections 2022-07-20T02:35:00.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.319+0000 7f9c6555b700 1 -- 172.21.15.26:0/3006914792 wait complete. 2022-07-20T02:35:00.397 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 72 2022-07-20T02:35:00.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:35:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.507+0000 7f317f59e700 1 Processor -- start 2022-07-20T02:35:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.507+0000 7f317f59e700 1 -- start start 2022-07-20T02:35:00.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f31800b38b0 0x7f31800b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800a9f00 0x7f3180144e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318013ec80 con 0x7f31800667d0 2022-07-20T02:35:00.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3180066d50 con 0x7f31800b38b0 2022-07-20T02:35:00.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180144710 con 0x7f31800a9f00 2022-07-20T02:35:00.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f31800b38b0 0x7f31800b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f31800b38b0 0x7f31800b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43330/0 (socket says 172.21.15.26:43330) 2022-07-20T02:35:00.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 -- 172.21.15.26:0/114385138 learned_addr learned my addr 172.21.15.26:0/114385138 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:00.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317ed9d700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800a9f00 0x7f3180144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 -- 172.21.15.26:0/114385138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800a9f00 msgr2=0x7f3180144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:00.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800a9f00 0x7f3180144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 -- 172.21.15.26:0/114385138 >> v1:172.21.15.120:6789/0 conn(0x7f31800667d0 legacy=0x7f3180140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 -- 172.21.15.26:0/114385138 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31801449b0 con 0x7f31800b38b0 2022-07-20T02:35:00.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317e59c700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800b38b0 0x7f31800b3c90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f3170002f70 tx=0x7f3170006eb0).ready entity=mon.0 client_cookie=33b8a692709dd633 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317dd9b700 1 -- 172.21.15.26:0/114385138 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317000d570 con 0x7f31800b38b0 2022-07-20T02:35:00.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317dd9b700 1 -- 172.21.15.26:0/114385138 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3170000e90 con 0x7f31800b38b0 2022-07-20T02:35:00.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317dd9b700 1 -- 172.21.15.26:0/114385138 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317000d900 con 0x7f31800b38b0 2022-07-20T02:35:00.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- 172.21.15.26:0/114385138 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800b38b0 msgr2=0x7f31800b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.606 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800b38b0 0x7f31800b3c90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f3170002f70 tx=0x7f3170006eb0).stop 2022-07-20T02:35:00.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- 172.21.15.26:0/114385138 shutdown_connections 2022-07-20T02:35:00.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800a9f00 0x7f3180144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 --2- 172.21.15.26:0/114385138 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800b38b0 0x7f31800b3c90 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- 172.21.15.26:0/114385138 >> 172.21.15.26:0/114385138 conn(0x7f31800b71d0 msgr2=0x7f31800b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:00.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- 172.21.15.26:0/114385138 shutdown_connections 2022-07-20T02:35:00.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.511+0000 7f317f59e700 1 -- 172.21.15.26:0/114385138 wait complete. 2022-07-20T02:35:00.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 Processor -- start 2022-07-20T02:35:00.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 -- start start 2022-07-20T02:35:00.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31800a9f00 0x7f31800acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800b38b0 0x7f3180140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318013fbf0 con 0x7f31800667d0 2022-07-20T02:35:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800ac3f0 con 0x7f31800a9f00 2022-07-20T02:35:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317f59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318013fe20 con 0x7f31800b38b0 2022-07-20T02:35:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43334/0 (socket says 172.21.15.26:43334) 2022-07-20T02:35:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 -- 172.21.15.26:0/3166118534 learned_addr learned my addr 172.21.15.26:0/3166118534 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f3177fff700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31800a9f00 0x7f31800acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317ed9d700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800b38b0 0x7f3180140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 -- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800b38b0 msgr2=0x7f3180140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800b38b0 0x7f3180140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 -- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31800a9f00 msgr2=0x7f31800acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31800a9f00 0x7f31800acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 -- 172.21.15.26:0/3166118534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317000b1e0 con 0x7f31800667d0 2022-07-20T02:35:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.515+0000 7f317e59c700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f3170002f70 tx=0x7f3170015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3170011760 con 0x7f31800667d0 2022-07-20T02:35:00.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31700118c0 con 0x7f31800667d0 2022-07-20T02:35:00.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31800ac650 con 0x7f31800667d0 2022-07-20T02:35:00.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3170011c20 con 0x7f31800667d0 2022-07-20T02:35:00.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318013f860 con 0x7f31800667d0 2022-07-20T02:35:00.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3180145340 con 0x7f31800667d0 2022-07-20T02:35:00.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3170025030 con 0x7f31800667d0 2022-07-20T02:35:00.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 0x7f3164055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:00.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f3177fff700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 0x7f3164055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:00.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.519+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f317006d2c0 con 0x7f31800667d0 2022-07-20T02:35:00.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.523+0000 7f3177fff700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 0x7f3164055ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f316800d000 tx=0x7f3168007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:00.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.531+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3170047e60 con 0x7f31800667d0 2022-07-20T02:35:00.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:00.695+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f31800574e0 con 0x7f31800667d0 2022-07-20T02:35:01.611 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:35:01.607+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:35:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.619+0000 7f31777fe700 1 -- 172.21.15.26:0/3166118534 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3170023090 con 0x7f31800667d0 2022-07-20T02:35:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.623+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 msgr2=0x7f3164055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.623+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 0x7f3164055ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f316800d000 tx=0x7f3168007b90).stop 2022-07-20T02:35:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.623+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 msgr2=0x7f31800b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.623+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f3170002f70 tx=0x7f3170015000).stop 2022-07-20T02:35:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 shutdown_connections 2022-07-20T02:35:01.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f31640539e0 0x7f3164055ea0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31800b38b0 0x7f3180140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31800a9f00 0x7f31800acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 --2- 172.21.15.26:0/3166118534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f31800667d0 0x7f31800b5b40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.638 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 >> 172.21.15.26:0/3166118534 conn(0x7f31800b71d0 msgr2=0x7f3180141700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 shutdown_connections 2022-07-20T02:35:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.627+0000 7f317f59e700 1 -- 172.21.15.26:0/3166118534 wait complete. 2022-07-20T02:35:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:35:01.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:01.719 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:35:01.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 Processor -- start 2022-07-20T02:35:01.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 -- start start 2022-07-20T02:35:01.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fe805d180 0x7f0fe80afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:01.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe80b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:01.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe805d560 con 0x7f0fe80609f0 2022-07-20T02:35:01.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe814bd60 con 0x7f0fe805d180 2022-07-20T02:35:01.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0feed30700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8098b80 con 0x7f0fe8060f00 2022-07-20T02:35:01.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fee52f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0fe80609f0 0x7f0fe805f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42636/0 (socket says 172.21.15.26:42636) 2022-07-20T02:35:01.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fee52f700 1 -- 172.21.15.26:0/3012923744 learned_addr learned my addr 172.21.15.26:0/3012923744 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:01.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 --2- 172.21.15.26:0/3012923744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe80b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:01.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fedd2e700 1 --2- 172.21.15.26:0/3012923744 >> v2:172.21.15.120:3300/0 conn(0x7f0fe805d180 0x7f0fe80afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:01.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fecd2c700 1 -- 172.21.15.26:0/3012923744 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828955288 0 0) 0x7f0fe805d560 con 0x7f0fe80609f0 2022-07-20T02:35:01.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fecd2c700 1 -- 172.21.15.26:0/3012923744 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd0003260 con 0x7f0fe80609f0 2022-07-20T02:35:01.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 -- 172.21.15.26:0/3012923744 >> v2:172.21.15.120:3300/0 conn(0x7f0fe805d180 msgr2=0x7f0fe80afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:01.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 --2- 172.21.15.26:0/3012923744 >> v2:172.21.15.120:3300/0 conn(0x7f0fe805d180 0x7f0fe80afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 -- 172.21.15.26:0/3012923744 >> v1:172.21.15.26:6789/0 conn(0x7f0fe80609f0 legacy=0x7f0fe805f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 -- 172.21.15.26:0/3012923744 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80a8ff0 con 0x7f0fe8060f00 2022-07-20T02:35:01.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fed52d700 1 --2- 172.21.15.26:0/3012923744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe80b03a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0fe40032f0 tx=0x7f0fe40091d0).ready entity=mon.2 client_cookie=205240d7eef5a93e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:01.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fecd2c700 1 -- 172.21.15.26:0/3012923744 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b600 con 0x7f0fe8060f00 2022-07-20T02:35:01.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fecd2c700 1 -- 172.21.15.26:0/3012923744 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400b760 con 0x7f0fe8060f00 2022-07-20T02:35:01.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.787+0000 7f0fecd2c700 1 -- 172.21.15.26:0/3012923744 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b8d0 con 0x7f0fe8060f00 2022-07-20T02:35:01.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- 172.21.15.26:0/3012923744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 msgr2=0x7f0fe80b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- 172.21.15.26:0/3012923744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe80b03a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0fe40032f0 tx=0x7f0fe40091d0).stop 2022-07-20T02:35:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- 172.21.15.26:0/3012923744 shutdown_connections 2022-07-20T02:35:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- 172.21.15.26:0/3012923744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe80b03a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- 172.21.15.26:0/3012923744 >> v2:172.21.15.120:3300/0 conn(0x7f0fe805d180 0x7f0fe80afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- 172.21.15.26:0/3012923744 >> 172.21.15.26:0/3012923744 conn(0x7f0fe80b2080 msgr2=0x7f0fe80b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:01.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- 172.21.15.26:0/3012923744 shutdown_connections 2022-07-20T02:35:01.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- 172.21.15.26:0/3012923744 wait complete. 2022-07-20T02:35:01.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 Processor -- start 2022-07-20T02:35:01.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- start start 2022-07-20T02:35:01.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fe805d180 0x7f0fe814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:01.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:01.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:01.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8159d40 con 0x7f0fe80609f0 2022-07-20T02:35:01.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8145c30 con 0x7f0fe805d180 2022-07-20T02:35:01.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0feed30700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8159f70 con 0x7f0fe8060f00 2022-07-20T02:35:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43360/0 (socket says 172.21.15.26:43360) 2022-07-20T02:35:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 -- 172.21.15.26:0/2244009941 learned_addr learned my addr 172.21.15.26:0/2244009941 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fedd2e700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fe805d180 0x7f0fe814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fee52f700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 -- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 msgr2=0x7f0fe815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 -- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fe805d180 msgr2=0x7f0fe814e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fe805d180 0x7f0fe814e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 -- 172.21.15.26:0/2244009941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4009470 con 0x7f0fe80609f0 2022-07-20T02:35:01.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.791+0000 7f0fed52d700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f0fe40032f0 tx=0x7f0fe4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b8d0 con 0x7f0fe80609f0 2022-07-20T02:35:02.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4021630 con 0x7f0fe80609f0 2022-07-20T02:35:02.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8145ed0 con 0x7f0fe80609f0 2022-07-20T02:35:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b600 con 0x7f0fe80609f0 2022-07-20T02:35:02.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe814ed20 con 0x7f0fe80609f0 2022-07-20T02:35:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe809a490 con 0x7f0fe80609f0 2022-07-20T02:35:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0fe4028070 con 0x7f0fe80609f0 2022-07-20T02:35:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 0x7f0fc8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f0fe406d170 con 0x7f0fe80609f0 2022-07-20T02:35:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fedd2e700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 0x7f0fc8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.795+0000 7f0fedd2e700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 0x7f0fc8055e10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f0fd800b000 tx=0x7f0fd8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.807+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0fe4043ee0 con 0x7f0fe80609f0 2022-07-20T02:35:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.963+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fe80574e0 con 0x7f0fe80609f0 2022-07-20T02:35:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0fd6ffd700 1 -- 172.21.15.26:0/2244009941 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f0fe4015020 con 0x7f0fe80609f0 2022-07-20T02:35:02.041 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:02.042 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":73,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:34:40.117843+0000","modified":"2022-07-20T02:35:01.611349+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T02:35:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 msgr2=0x7f0fc8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 0x7f0fc8055e10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f0fd800b000 tx=0x7f0fd8006df0).stop 2022-07-20T02:35:02.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 msgr2=0x7f0fe8156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f0fe40032f0 tx=0x7f0fe4016000).stop 2022-07-20T02:35:02.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 shutdown_connections 2022-07-20T02:35:02.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fe80609f0 0x7f0fe8156750 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fc8053950 0x7f0fc8055e10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fe8060f00 0x7f0fe815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 --2- 172.21.15.26:0/2244009941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fe805d180 0x7f0fe814e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 >> 172.21.15.26:0/2244009941 conn(0x7f0fe80b2080 msgr2=0x7f0fe8099850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:02.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 shutdown_connections 2022-07-20T02:35:02.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:01.967+0000 7f0feed30700 1 -- 172.21.15.26:0/2244009941 wait complete. 2022-07-20T02:35:02.083 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 73 2022-07-20T02:35:02.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:35:02.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 Processor -- start 2022-07-20T02:35:02.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 -- start start 2022-07-20T02:35:02.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdcd40609f0 0x7fdcd40b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd405d560 con 0x7fdcd405d180 2022-07-20T02:35:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd40a8ff0 con 0x7fdcd4060f00 2022-07-20T02:35:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcda4cf700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd414bd60 con 0x7fdcd40609f0 2022-07-20T02:35:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcd94cd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdcd405d180 0x7fdcd40afeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42646/0 (socket says 172.21.15.26:42646) 2022-07-20T02:35:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcd94cd700 1 -- 172.21.15.26:0/2211047818 learned_addr learned my addr 172.21.15.26:0/2211047818 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:02.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.195+0000 7fdcd9cce700 1 --2- 172.21.15.26:0/2211047818 >> v2:172.21.15.26:3300/0 conn(0x7fdcd40609f0 0x7fdcd40b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172147358 0 0) 0x7fdcd405d560 con 0x7fdcd405d180 2022-07-20T02:35:02.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0003260 con 0x7fdcd405d180 2022-07-20T02:35:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669183905 0 0) 0x7fdcd40a8ff0 con 0x7fdcd4060f00 2022-07-20T02:35:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd405d560 con 0x7fdcd4060f00 2022-07-20T02:35:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd9cce700 1 -- 172.21.15.26:0/2211047818 >> v1:172.21.15.120:6789/0 conn(0x7fdcd4060f00 legacy=0x7fdcd414acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd9cce700 1 -- 172.21.15.26:0/2211047818 >> v1:172.21.15.26:6789/0 conn(0x7fdcd405d180 legacy=0x7fdcd40afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd9cce700 1 -- 172.21.15.26:0/2211047818 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd409a490 con 0x7fdcd40609f0 2022-07-20T02:35:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd9cce700 1 --2- 172.21.15.26:0/2211047818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd40609f0 0x7fdcd40b05c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fdcd409afb0 tx=0x7fdccc006df0).ready entity=mon.0 client_cookie=5191cd9230587c78 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdccc00d610 con 0x7fdcd40609f0 2022-07-20T02:35:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc000e90 con 0x7fdcd40609f0 2022-07-20T02:35:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcd3fff700 1 -- 172.21.15.26:0/2211047818 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdccc00d960 con 0x7fdcd40609f0 2022-07-20T02:35:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- 172.21.15.26:0/2211047818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd40609f0 msgr2=0x7fdcd40b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/2211047818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd40609f0 0x7fdcd40b05c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fdcd409afb0 tx=0x7fdccc006df0).stop 2022-07-20T02:35:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- 172.21.15.26:0/2211047818 shutdown_connections 2022-07-20T02:35:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/2211047818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd40609f0 0x7fdcd40b05c0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- 172.21.15.26:0/2211047818 >> 172.21.15.26:0/2211047818 conn(0x7fdcd40b2080 msgr2=0x7fdcd40b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- 172.21.15.26:0/2211047818 shutdown_connections 2022-07-20T02:35:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- 172.21.15.26:0/2211047818 wait complete. 2022-07-20T02:35:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 Processor -- start 2022-07-20T02:35:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.199+0000 7fdcda4cf700 1 -- start start 2022-07-20T02:35:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40609f0 0x7fdcd4145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd4060f00 0x7fdcd4148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b5390 con 0x7fdcd405d180 2022-07-20T02:35:02.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4149ec0 con 0x7fdcd40609f0 2022-07-20T02:35:02.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd414a160 con 0x7fdcd4060f00 2022-07-20T02:35:02.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43372/0 (socket says 172.21.15.26:43372) 2022-07-20T02:35:02.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd8ccc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40609f0 0x7fdcd4145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 -- 172.21.15.26:0/3000688663 learned_addr learned my addr 172.21.15.26:0/3000688663 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:02.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd9cce700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd4060f00 0x7fdcd4148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 -- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd4060f00 msgr2=0x7fdcd4148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd4060f00 0x7fdcd4148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 -- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40609f0 msgr2=0x7fdcd4145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40609f0 0x7fdcd4145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 -- 172.21.15.26:0/3000688663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc00b180 con 0x7fdcd405d180 2022-07-20T02:35:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd94cd700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fdcc8001490 tx=0x7fdcc8007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:02.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcc8005ba0 con 0x7fdcd405d180 2022-07-20T02:35:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8005d00 con 0x7fdcd405d180 2022-07-20T02:35:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd414a400 con 0x7fdcd405d180 2022-07-20T02:35:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcc80072c0 con 0x7fdcd405d180 2022-07-20T02:35:02.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd414a8a0 con 0x7fdcd405d180 2022-07-20T02:35:02.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.203+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdcc801a060 con 0x7fdcd405d180 2022-07-20T02:35:02.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.207+0000 7fdcd27fc700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 0x7fdcb4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.207+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd409a730 con 0x7fdcd405d180 2022-07-20T02:35:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.207+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fdcc806b750 con 0x7fdcd405d180 2022-07-20T02:35:02.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.207+0000 7fdcd8ccc700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 0x7fdcb4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.207+0000 7fdcd8ccc700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 0x7fdcb4055ed0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fdcc400d000 tx=0x7fdcc4007a20).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:02.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.215+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdcc8018070 con 0x7fdcd405d180 2022-07-20T02:35:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.371+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdcd405da10 con 0x7fdcd405d180 2022-07-20T02:35:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.623+0000 7fdcd27fc700 1 -- 172.21.15.26:0/3000688663 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdcc80424c0 con 0x7fdcd405d180 2022-07-20T02:35:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.627+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 msgr2=0x7fdcb4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.627+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 0x7fdcb4055ed0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fdcc400d000 tx=0x7fdcc4007a20).stop 2022-07-20T02:35:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.627+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 msgr2=0x7fdcd40af420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.627+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fdcc8001490 tx=0x7fdcc8007e50).stop 2022-07-20T02:35:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.627+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 shutdown_connections 2022-07-20T02:35:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdcb4053a10 0x7fdcb4055ed0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd4060f00 0x7fdcd4148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40609f0 0x7fdcd4145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 --2- 172.21.15.26:0/3000688663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcd405d180 0x7fdcd40af420 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 >> 172.21.15.26:0/3000688663 conn(0x7fdcd40b2080 msgr2=0x7fdcd414b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 shutdown_connections 2022-07-20T02:35:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.631+0000 7fdcda4cf700 1 -- 172.21.15.26:0/3000688663 wait complete. 2022-07-20T02:35:02.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:35:02.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 Processor -- start 2022-07-20T02:35:02.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 -- start start 2022-07-20T02:35:02.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d80061810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d801467a0 con 0x7f6d8013d220 2022-07-20T02:35:02.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d801469d0 con 0x7f6d801473b0 2022-07-20T02:35:02.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80146c70 con 0x7f6d800b5ad0 2022-07-20T02:35:02.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7effd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6d8013d220 0x7f6d800b6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42660/0 (socket says 172.21.15.26:42660) 2022-07-20T02:35:02.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7effd700 1 -- 172.21.15.26:0/2795613521 learned_addr learned my addr 172.21.15.26:0/2795613521 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d80061810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533722810 0 0) 0x7f6d801467a0 con 0x7f6d8013d220 2022-07-20T02:35:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60003260 con 0x7f6d8013d220 2022-07-20T02:35:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73605694 0 0) 0x7f6d60003260 con 0x7f6d8013d220 2022-07-20T02:35:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 msgr2=0x7f6d80061810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d80061810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 >> v1:172.21.15.120:6789/0 conn(0x7f6d801473b0 legacy=0x7f6d801431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d80146f10 con 0x7f6d8013d220 2022-07-20T02:35:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f6d70001660 con 0x7f6d8013d220 2022-07-20T02:35:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 >> v1:172.21.15.26:6789/0 conn(0x7f6d8013d220 legacy=0x7f6d800b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d60003780 0x7f6d60003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d60004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d60004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80146c70 con 0x7f6d600049f0 2022-07-20T02:35:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6000b780 con 0x7f6d60003780 2022-07-20T02:35:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6000b940 con 0x7f6d800b5ad0 2022-07-20T02:35:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d60004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7effd700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d60003780 0x7f6d60003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7e7fc700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d60004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 -- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 msgr2=0x7f6d60004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d800b5ad0 0x7f6d60004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 -- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d60003780 msgr2=0x7f6d60003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d60003780 0x7f6d60003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 -- 172.21.15.26:0/2795613521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60003260 con 0x7f6d600049f0 2022-07-20T02:35:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.763+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d60004e70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f6d7800c000 tx=0x7f6d780091d0).ready entity=mon.0 client_cookie=35a27235a1671e47 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7800e7e0 con 0x7f6d600049f0 2022-07-20T02:35:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7800e940 con 0x7f6d600049f0 2022-07-20T02:35:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7dffb700 1 -- 172.21.15.26:0/2795613521 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7800eca0 con 0x7f6d600049f0 2022-07-20T02:35:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- 172.21.15.26:0/2795613521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 msgr2=0x7f6d60004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/2795613521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d60004e70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f6d7800c000 tx=0x7f6d780091d0).stop 2022-07-20T02:35:03.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7f7fe700 1 -- 172.21.15.26:0/2795613521 reap_dead start 2022-07-20T02:35:03.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- 172.21.15.26:0/2795613521 shutdown_connections 2022-07-20T02:35:03.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- 172.21.15.26:0/2795613521 >> 172.21.15.26:0/2795613521 conn(0x7f6d80066f70 msgr2=0x7f6d8005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:03.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- 172.21.15.26:0/2795613521 shutdown_connections 2022-07-20T02:35:03.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- 172.21.15.26:0/2795613521 wait complete. 2022-07-20T02:35:03.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 Processor -- start 2022-07-20T02:35:03.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.767+0000 7f6d7ffff700 1 -- start start 2022-07-20T02:35:03.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d8013b210 0x7f6d80152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80153280 0x7f6d800b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800ba600 con 0x7f6d600049f0 2022-07-20T02:35:03.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d801526c0 con 0x7f6d80153280 2022-07-20T02:35:03.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b7f80 con 0x7f6d8013b210 2022-07-20T02:35:03.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43392/0 (socket says 172.21.15.26:43392) 2022-07-20T02:35:03.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 -- 172.21.15.26:0/3143419495 learned_addr learned my addr 172.21.15.26:0/3143419495 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:03.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7e7fc700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d8013b210 0x7f6d80152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7f7fe700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80153280 0x7f6d800b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 -- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d8013b210 msgr2=0x7f6d80152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d8013b210 0x7f6d80152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 -- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80153280 msgr2=0x7f6d800b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80153280 0x7f6d800b90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 -- 172.21.15.26:0/3143419495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d801528f0 con 0x7f6d600049f0 2022-07-20T02:35:03.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7effd700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f6d7000d000 tx=0x7f6d70001e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d700116b0 con 0x7f6d600049f0 2022-07-20T02:35:03.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70011810 con 0x7f6d600049f0 2022-07-20T02:35:03.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d800b8170 con 0x7f6d600049f0 2022-07-20T02:35:03.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d70011b70 con 0x7f6d600049f0 2022-07-20T02:35:03.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8013e290 con 0x7f6d600049f0 2022-07-20T02:35:03.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d7001b030 con 0x7f6d600049f0 2022-07-20T02:35:03.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.771+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8013b5f0 con 0x7f6d600049f0 2022-07-20T02:35:03.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.775+0000 7f6d77fff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 0x7f6d64055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.775+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f6d7006bbf0 con 0x7f6d600049f0 2022-07-20T02:35:03.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.775+0000 7f6d7e7fc700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 0x7f6d64055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.775+0000 7f6d7e7fc700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 0x7f6d64055ea0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f6d68003000 tx=0x7f6d68007a00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.779+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d70042960 con 0x7f6d600049f0 2022-07-20T02:35:03.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:02.907+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d800574e0 con 0x7f6d600049f0 2022-07-20T02:35:03.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d77fff700 1 -- 172.21.15.26:0/3143419495 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 185+0+0 (secure 0 0 0) 0x7f6d70019030 con 0x7f6d600049f0 2022-07-20T02:35:03.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 msgr2=0x7f6d64055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 0x7f6d64055ea0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f6d68003000 tx=0x7f6d68007a00).stop 2022-07-20T02:35:03.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 msgr2=0x7f6d800b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f6d7000d000 tx=0x7f6d70001e10).stop 2022-07-20T02:35:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 shutdown_connections 2022-07-20T02:35:03.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6d640539e0 0x7f6d64055ea0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d80153280 0x7f6d800b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d8013b210 0x7f6d80152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 --2- 172.21.15.26:0/3143419495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d600049f0 0x7f6d800b8970 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.635+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 >> 172.21.15.26:0/3143419495 conn(0x7f6d80066f70 msgr2=0x7f6d8013a690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:03.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.639+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 shutdown_connections 2022-07-20T02:35:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.639+0000 7f6d7ffff700 1 -- 172.21.15.26:0/3143419495 wait complete. 2022-07-20T02:35:03.657 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:35:03.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:35:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 Processor -- start 2022-07-20T02:35:03.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 -- start start 2022-07-20T02:35:03.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4ea405d140 0x7f4ea405c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea4140c20 0x7f4ea40b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea405d520 con 0x7f4ea405be30 2022-07-20T02:35:03.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea40b28d0 con 0x7f4ea405d140 2022-07-20T02:35:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.831+0000 7f4ea9b9f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40adb70 con 0x7f4ea4140c20 2022-07-20T02:35:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4ea405d140 0x7f4ea405c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea3fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4ea405be30 0x7f4ea4140510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42682/0 (socket says 172.21.15.26:42682) 2022-07-20T02:35:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea3fff700 1 -- 172.21.15.26:0/2550854127 learned_addr learned my addr 172.21.15.26:0/2550854127 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea939e700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea4140c20 0x7f4ea40b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2550854127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea4140c20 msgr2=0x7f4ea40b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea4140c20 0x7f4ea40b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2550854127 >> v1:172.21.15.26:6789/0 conn(0x7f4ea405be30 legacy=0x7f4ea4140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2550854127 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4146720 con 0x7f4ea405d140 2022-07-20T02:35:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea8b9d700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405d140 0x7f4ea405c600 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98006df0).ready entity=mon.0 client_cookie=fa50cef3b546816e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea37fe700 1 -- 172.21.15.26:0/2550854127 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9800d570 con 0x7f4ea405d140 2022-07-20T02:35:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea37fe700 1 -- 172.21.15.26:0/2550854127 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98000e90 con 0x7f4ea405d140 2022-07-20T02:35:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea37fe700 1 -- 172.21.15.26:0/2550854127 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9800d900 con 0x7f4ea405d140 2022-07-20T02:35:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2550854127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405d140 msgr2=0x7f4ea405c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405d140 0x7f4ea405c600 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98006df0).stop 2022-07-20T02:35:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2550854127 shutdown_connections 2022-07-20T02:35:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea4140c20 0x7f4ea40b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2550854127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405d140 0x7f4ea405c600 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2550854127 >> 172.21.15.26:0/2550854127 conn(0x7f4ea40b66a0 msgr2=0x7f4ea40b6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2550854127 shutdown_connections 2022-07-20T02:35:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2550854127 wait complete. 2022-07-20T02:35:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 Processor -- start 2022-07-20T02:35:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.835+0000 7f4ea9b9f700 1 -- start start 2022-07-20T02:35:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea405d140 0x7f4ea40ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 0x7f4ea4142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40ad240 con 0x7f4ea405be30 2022-07-20T02:35:03.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40aad50 con 0x7f4ea405d140 2022-07-20T02:35:03.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40ad470 con 0x7f4ea40ac710 2022-07-20T02:35:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43408/0 (socket says 172.21.15.26:43408) 2022-07-20T02:35:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2132783196 learned_addr learned my addr 172.21.15.26:0/2132783196 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea939e700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 0x7f4ea4142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea3fff700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea405d140 0x7f4ea40ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 msgr2=0x7f4ea4142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 0x7f4ea4142770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea405d140 msgr2=0x7f4ea40ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea405d140 0x7f4ea40ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 -- 172.21.15.26:0/2132783196 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9800b180 con 0x7f4ea405be30 2022-07-20T02:35:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea939e700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 0x7f4ea4142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea8b9d700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9800de40 con 0x7f4ea405be30 2022-07-20T02:35:03.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9801e400 con 0x7f4ea405be30 2022-07-20T02:35:03.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea4143390 con 0x7f4ea405be30 2022-07-20T02:35:03.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9801e760 con 0x7f4ea405be30 2022-07-20T02:35:03.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea40ab9c0 con 0x7f4ea405be30 2022-07-20T02:35:03.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea4143830 con 0x7f4ea405be30 2022-07-20T02:35:03.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.839+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e9801c070 con 0x7f4ea405be30 2022-07-20T02:35:03.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.843+0000 7f4ea1ffb700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 0x7f4e84055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:03.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.843+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f4e9806cac0 con 0x7f4ea405be30 2022-07-20T02:35:03.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.843+0000 7f4ea3fff700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 0x7f4e84055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:03.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.843+0000 7f4ea3fff700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 0x7f4e84055f60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f4e9400b000 tx=0x7f4e94009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:03.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.847+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ea4143830 con 0x7f4ea405be30 2022-07-20T02:35:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:03.971+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ea4143830 con 0x7f4ea405be30 2022-07-20T02:35:04.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.639+0000 7f4ea1ffb700 1 -- 172.21.15.26:0/2132783196 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4ea4143830 con 0x7f4ea405be30 2022-07-20T02:35:04.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 msgr2=0x7f4e84055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:04.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 0x7f4e84055f60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f4e9400b000 tx=0x7f4e94009040).stop 2022-07-20T02:35:04.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 msgr2=0x7f4ea4143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:04.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98014000).stop 2022-07-20T02:35:04.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 shutdown_connections 2022-07-20T02:35:04.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e84053aa0 0x7f4e84055f60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:04.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ea40ac710 0x7f4ea4142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:04.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ea405d140 0x7f4ea40ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:04.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 --2- 172.21.15.26:0/2132783196 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ea405be30 0x7f4ea4143ad0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:04.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 >> 172.21.15.26:0/2132783196 conn(0x7f4ea40b66a0 msgr2=0x7f4ea41402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:04.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 shutdown_connections 2022-07-20T02:35:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:04.647+0000 7f4ea9b9f700 1 -- 172.21.15.26:0/2132783196 wait complete. 2022-07-20T02:35:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:35:04.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:35:04.668 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:35:04.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:35:04.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T02:35:04.741 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2022-07-20T02:35:04.741 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2022-07-20T02:35:04.741 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:35:04.742 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2022-07-20T02:35:04.742 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.742 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2022-07-20T02:35:04.742 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.743 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2022-07-20T02:35:04.751 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.752 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2022-07-20T02:35:04.752 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2022-07-20T02:35:04.752 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.753 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2022-07-20T02:35:04.753 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2022-07-20T02:35:04.753 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.753 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:35:04.753 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2022-07-20T02:35:04.760 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.760 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:35:04.761 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:35:04.761 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.761 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2022-07-20T02:35:04.765 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.766 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2022-07-20T02:35:04.767 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2022-07-20T02:35:04.767 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:35:04.767 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:35:04.767 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2022-07-20T02:35:04.768 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:35:04.793 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2022-07-20T02:35:04.794 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:35:04.794 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:35:04.794 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.794 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:35:04.795 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.795 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2022-07-20T02:35:04.795 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.795 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2022-07-20T02:35:04.796 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.796 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2022-07-20T02:35:04.796 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2022-07-20T02:35:04.796 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2022-07-20T02:35:04.797 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:35:04.798 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:35:04.799 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.799 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2022-07-20T02:35:04.799 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2022-07-20T02:35:04.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.800 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2022-07-20T02:35:04.800 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2022-07-20T02:35:04.801 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.801 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2022-07-20T02:35:04.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2022-07-20T02:35:04.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.804 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2022-07-20T02:35:04.804 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2022-07-20T02:35:04.805 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2022-07-20T02:35:04.805 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2022-07-20T02:35:04.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2022-07-20T02:35:04.806 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.807 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:35:04.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.808 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2022-07-20T02:35:04.809 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.810 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2022-07-20T02:35:04.810 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.811 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:35:04.811 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2022-07-20T02:35:04.812 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.812 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2022-07-20T02:35:04.814 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2022-07-20T02:35:04.814 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2022-07-20T02:35:04.816 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression mode = aggressive 2022-07-20T02:35:04.816 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2022-07-20T02:35:04.817 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2022-07-20T02:35:04.817 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2022-07-20T02:35:04.817 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:35:04.818 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2022-07-20T02:35:04.818 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2022-07-20T02:35:04.819 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:35:04.820 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2022-07-20T02:35:04.821 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:35:04.821 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:35:04.821 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2022-07-20T02:35:04.822 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:35:04.825 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.825 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2022-07-20T02:35:04.825 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:35:04.826 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2022-07-20T02:35:04.827 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:35:04.827 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:35:04.828 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:04.828 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:04.829 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2022-07-20T02:35:04.829 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.829 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2022-07-20T02:35:04.830 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:35:04.831 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2022-07-20T02:35:04.831 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2022-07-20T02:35:04.832 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2022-07-20T02:35:04.833 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2022-07-20T02:35:04.836 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2022-07-20T02:35:04.836 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:04.837 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2022-07-20T02:35:04.852 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2022-07-20T02:35:04.852 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:04.853 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.854 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:35:04.881 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:35:04.881 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:35:04.883 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.884 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:35:04.884 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:35:04.884 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:35:04.884 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2022-07-20T02:35:04.885 INFO:teuthology.orchestra.run.smithi159.stdout: 2022-07-20T02:35:04.885 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2022-07-20T02:35:04.885 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2022-07-20T02:35:04.886 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2022-07-20T02:35:04.886 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2022-07-20T02:35:04.886 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:35:04.886 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:35:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2022-07-20T02:35:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2022-07-20T02:35:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:35:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:35:04.887 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:35:04.888 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2022-07-20T02:35:04.888 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2022-07-20T02:35:04.888 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2022-07-20T02:35:04.888 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2022-07-20T02:35:04.889 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2022-07-20T02:35:04.889 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2022-07-20T02:35:04.889 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2022-07-20T02:35:04.889 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2022-07-20T02:35:04.890 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2022-07-20T02:35:04.890 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal frag = True 2022-07-20T02:35:04.890 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal fragment size max = 10000 2022-07-20T02:35:04.890 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal merge size = 5 2022-07-20T02:35:04.890 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split bits = 3 2022-07-20T02:35:04.891 INFO:teuthology.orchestra.run.smithi159.stdout: mds bal split size = 100 2022-07-20T02:35:04.891 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2022-07-20T02:35:04.891 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2022-07-20T02:35:04.891 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2022-07-20T02:35:04.892 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2022-07-20T02:35:04.892 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2022-07-20T02:35:04.893 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 15m 2022-07-20T02:35:04.893 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 15m 2022-07-20T02:35:04.893 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2022-07-20T02:35:04.893 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2022-07-20T02:35:04.899 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:35:04.900 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2022-07-20T02:35:04.906 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.906 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2022-07-20T02:35:04.906 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.906 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2022-07-20T02:35:04.907 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.907 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2022-07-20T02:35:04.907 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2022-07-20T02:35:04.907 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.908 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2022-07-20T02:35:04.908 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2022-07-20T02:35:04.908 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.910 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:35:04.911 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2022-07-20T02:35:04.911 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.914 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:35:04.914 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:35:04.914 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.922 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2022-07-20T02:35:04.923 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.923 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2022-07-20T02:35:04.924 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2022-07-20T02:35:04.924 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:35:04.924 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:35:04.925 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2022-07-20T02:35:04.925 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:35:04.925 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2022-07-20T02:35:04.926 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:35:04.926 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:35:04.926 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.940 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:35:04.943 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.944 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2022-07-20T02:35:04.944 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.945 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2022-07-20T02:35:04.945 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:04.972 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2022-07-20T02:35:04.973 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2022-07-20T02:35:04.973 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2022-07-20T02:35:05.007 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:35:05.007 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:35:05.008 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.008 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2022-07-20T02:35:05.009 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2022-07-20T02:35:05.009 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.029 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2022-07-20T02:35:05.032 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2022-07-20T02:35:05.032 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.032 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2022-07-20T02:35:05.032 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2022-07-20T02:35:05.033 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.033 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2022-07-20T02:35:05.033 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2022-07-20T02:35:05.033 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2022-07-20T02:35:05.034 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.042 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2022-07-20T02:35:05.043 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2022-07-20T02:35:05.043 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.043 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:35:05.043 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.044 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2022-07-20T02:35:05.044 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.044 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2022-07-20T02:35:05.044 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.044 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:35:05.045 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2022-07-20T02:35:05.045 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.045 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2022-07-20T02:35:05.055 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2022-07-20T02:35:05.055 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2022-07-20T02:35:05.056 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore compression mode = aggressive 2022-07-20T02:35:05.056 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2022-07-20T02:35:05.057 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 20 2022-07-20T02:35:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 20 2022-07-20T02:35:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:35:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2022-07-20T02:35:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2022-07-20T02:35:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:35:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2022-07-20T02:35:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:35:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:35:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2022-07-20T02:35:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:35:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.105 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2022-07-20T02:35:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:35:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2022-07-20T02:35:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:35:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:35:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2022-07-20T02:35:05.119 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.119 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2022-07-20T02:35:05.120 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:35:05.120 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2022-07-20T02:35:05.120 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2022-07-20T02:35:05.120 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2022-07-20T02:35:05.121 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2022-07-20T02:35:05.121 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2022-07-20T02:35:05.121 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.121 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2022-07-20T02:35:05.122 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2022-07-20T02:35:05.122 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.122 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.122 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:35:05.122 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:35:05.123 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:35:05.123 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.123 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:35:05.123 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:35:05.124 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:35:05.124 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2022-07-20T02:35:05.124 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-07-20T02:35:05.124 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2022-07-20T02:35:05.124 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2022-07-20T02:35:05.125 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2022-07-20T02:35:05.125 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2022-07-20T02:35:05.125 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:35:05.125 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:35:05.126 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2022-07-20T02:35:05.126 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2022-07-20T02:35:05.126 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:35:05.126 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.126 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.127 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2022-07-20T02:35:05.127 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2022-07-20T02:35:05.145 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2022-07-20T02:35:05.146 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2022-07-20T02:35:05.149 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2022-07-20T02:35:05.150 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2022-07-20T02:35:05.150 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2022-07-20T02:35:05.151 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2022-07-20T02:35:05.151 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2022-07-20T02:35:05.151 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal frag = True 2022-07-20T02:35:05.152 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal fragment size max = 10000 2022-07-20T02:35:05.152 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal merge size = 5 2022-07-20T02:35:05.153 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split bits = 3 2022-07-20T02:35:05.153 INFO:teuthology.orchestra.run.smithi120.stdout: mds bal split size = 100 2022-07-20T02:35:05.153 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2022-07-20T02:35:05.154 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2022-07-20T02:35:05.154 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2022-07-20T02:35:05.155 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2022-07-20T02:35:05.155 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2022-07-20T02:35:05.157 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.157 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.158 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2022-07-20T02:35:05.159 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2022-07-20T02:35:05.159 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:35:05.159 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2022-07-20T02:35:05.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.161 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2022-07-20T02:35:05.161 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.161 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2022-07-20T02:35:05.162 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.162 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2022-07-20T02:35:05.163 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2022-07-20T02:35:05.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.164 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2022-07-20T02:35:05.168 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2022-07-20T02:35:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:35:05.212 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2022-07-20T02:35:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:35:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:35:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2022-07-20T02:35:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2022-07-20T02:35:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2022-07-20T02:35:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:35:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:35:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2022-07-20T02:35:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:35:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2022-07-20T02:35:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:35:05.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:35:05.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.233 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:35:05.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.252 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2022-07-20T02:35:05.252 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.253 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2022-07-20T02:35:05.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.254 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2022-07-20T02:35:05.254 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2022-07-20T02:35:05.255 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2022-07-20T02:35:05.255 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:35:05.255 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:35:05.256 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.256 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2022-07-20T02:35:05.256 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2022-07-20T02:35:05.264 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.264 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2022-07-20T02:35:05.264 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2022-07-20T02:35:05.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.265 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2022-07-20T02:35:05.265 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2022-07-20T02:35:05.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2022-07-20T02:35:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2022-07-20T02:35:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2022-07-20T02:35:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2022-07-20T02:35:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2022-07-20T02:35:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:35:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2022-07-20T02:35:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2022-07-20T02:35:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:35:05.269 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2022-07-20T02:35:05.269 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.269 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2022-07-20T02:35:05.269 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2022-07-20T02:35:05.278 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2022-07-20T02:35:05.278 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2022-07-20T02:35:05.278 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2022-07-20T02:35:05.279 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2022-07-20T02:35:05.279 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2022-07-20T02:35:05.279 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:35:05.279 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2022-07-20T02:35:05.280 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2022-07-20T02:35:05.280 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:35:05.280 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2022-07-20T02:35:05.280 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:35:05.281 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:35:05.281 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2022-07-20T02:35:05.281 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:35:05.281 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.325 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2022-07-20T02:35:05.350 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:35:05.350 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2022-07-20T02:35:05.351 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:35:05.351 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:35:05.352 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.352 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.373 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2022-07-20T02:35:05.374 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.375 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2022-07-20T02:35:05.375 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:35:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2022-07-20T02:35:05.376 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2022-07-20T02:35:05.377 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2022-07-20T02:35:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2022-07-20T02:35:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2022-07-20T02:35:05.380 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.381 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2022-07-20T02:35:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2022-07-20T02:35:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.382 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:35:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:35:05.383 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:35:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.384 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:35:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:35:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:35:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: mon op complaint time = 120 2022-07-20T02:35:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:05.401 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2022-07-20T02:35:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2022-07-20T02:35:05.402 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2022-07-20T02:35:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2022-07-20T02:35:05.414 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:35:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:35:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: client mount timeout = 600 2022-07-20T02:35:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: debug client = 20 2022-07-20T02:35:05.415 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:35:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.416 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.416 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2022-07-20T02:35:05.416 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2022-07-20T02:35:05.417 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2022-07-20T02:35:05.417 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.a] 2022-07-20T02:35:05.417 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.c] 2022-07-20T02:35:05.417 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.b] 2022-07-20T02:35:05.418 INFO:teuthology.orchestra.run.smithi026.stdout:[mds] 2022-07-20T02:35:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: debug mds = 20 2022-07-20T02:35:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2022-07-20T02:35:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal frag = True 2022-07-20T02:35:05.418 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal fragment size max = 10000 2022-07-20T02:35:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal merge size = 5 2022-07-20T02:35:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split bits = 3 2022-07-20T02:35:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: mds bal split size = 100 2022-07-20T02:35:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug frag = True 2022-07-20T02:35:05.419 INFO:teuthology.orchestra.run.smithi026.stdout: mds debug scatterstat = True 2022-07-20T02:35:05.420 INFO:teuthology.orchestra.run.smithi026.stdout: mds op complaint time = 180 2022-07-20T02:35:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: mds verify scatter = True 2022-07-20T02:35:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: osd op complaint time = 180 2022-07-20T02:35:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.424 INFO:teuthology.orchestra.run.smithi026.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.431 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2022-07-20T02:35:05.431 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2022-07-20T02:35:05.432 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T02:35:05.435 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2022-07-20T02:35:05.435 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.436 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2022-07-20T02:35:05.451 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.457 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2022-07-20T02:35:05.458 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.458 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2022-07-20T02:35:05.459 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2022-07-20T02:35:05.459 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.460 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2022-07-20T02:35:05.460 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2022-07-20T02:35:05.465 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.466 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T02:35:05.466 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2022-07-20T02:35:05.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.472 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T02:35:05.473 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T02:35:05.474 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.474 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2022-07-20T02:35:05.474 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.475 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2022-07-20T02:35:05.509 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2022-07-20T02:35:05.509 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2022-07-20T02:35:05.509 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2022-07-20T02:35:05.510 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2022-07-20T02:35:05.510 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2022-07-20T02:35:05.511 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2022-07-20T02:35:05.511 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T02:35:05.511 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T02:35:05.512 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.513 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T02:35:05.533 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.547 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2022-07-20T02:35:05.568 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.569 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2022-07-20T02:35:05.569 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.570 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2022-07-20T02:35:05.570 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2022-07-20T02:35:05.571 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2022-07-20T02:35:05.571 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.26,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2022-07-20T02:35:05.571 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 36c84aa7-9150-4850-bbd5-e28fb0ff0eb0 2022-07-20T02:35:05.572 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.572 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2022-07-20T02:35:05.630 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2022-07-20T02:35:05.631 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.631 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2022-07-20T02:35:05.631 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2022-07-20T02:35:05.631 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.632 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2022-07-20T02:35:05.632 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2022-07-20T02:35:05.632 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.632 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2022-07-20T02:35:05.633 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2022-07-20T02:35:05.646 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2022-07-20T02:35:05.647 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.647 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2022-07-20T02:35:05.651 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2022-07-20T02:35:05.652 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.652 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2022-07-20T02:35:05.653 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.653 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2022-07-20T02:35:05.653 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.654 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2022-07-20T02:35:05.654 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.655 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2022-07-20T02:35:05.655 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2022-07-20T02:35:05.655 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.656 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2022-07-20T02:35:05.656 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2022-07-20T02:35:05.656 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2022-07-20T02:35:05.657 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2022-07-20T02:35:05.657 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2022-07-20T02:35:05.658 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2022-07-20T02:35:05.689 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2022-07-20T02:35:05.690 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:35:05.690 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2022-07-20T02:35:05.690 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2022-07-20T02:35:05.691 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2022-07-20T02:35:05.691 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2022-07-20T02:35:05.691 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2022-07-20T02:35:05.692 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2022-07-20T02:35:05.693 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2022-07-20T02:35:05.693 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:35:05.693 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.693 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2022-07-20T02:35:05.703 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:35:05.703 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2022-07-20T02:35:05.703 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:35:05.704 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:35:05.704 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.704 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.704 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2022-07-20T02:35:05.704 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.705 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2022-07-20T02:35:05.724 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:35:05.725 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2022-07-20T02:35:05.725 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2022-07-20T02:35:05.736 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2022-07-20T02:35:05.746 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2022-07-20T02:35:05.747 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2022-07-20T02:35:05.747 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2022-07-20T02:35:05.747 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2022-07-20T02:35:05.747 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2022-07-20T02:35:05.748 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2022-07-20T02:35:05.748 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.748 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T02:35:05.748 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T02:35:05.748 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T02:35:05.749 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.749 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T02:35:05.749 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T02:35:05.749 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T02:35:05.749 INFO:teuthology.orchestra.run.smithi176.stdout: mon op complaint time = 120 2022-07-20T02:35:05.769 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-07-20T02:35:05.774 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2022-07-20T02:35:05.775 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2022-07-20T02:35:05.775 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2022-07-20T02:35:05.776 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2022-07-20T02:35:05.778 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T02:35:05.778 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T02:35:05.779 INFO:teuthology.orchestra.run.smithi176.stdout: client mount timeout = 600 2022-07-20T02:35:05.780 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2022-07-20T02:35:05.780 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:35:05.781 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.781 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.782 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2022-07-20T02:35:05.789 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2022-07-20T02:35:05.790 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2022-07-20T02:35:05.791 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2022-07-20T02:35:05.791 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2022-07-20T02:35:05.793 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2022-07-20T02:35:05.806 INFO:teuthology.orchestra.run.smithi176.stdout:[mds] 2022-07-20T02:35:05.807 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds = 20 2022-07-20T02:35:05.807 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2022-07-20T02:35:05.807 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal frag = True 2022-07-20T02:35:05.807 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal fragment size max = 10000 2022-07-20T02:35:05.808 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal merge size = 5 2022-07-20T02:35:05.808 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split bits = 3 2022-07-20T02:35:05.808 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split size = 100 2022-07-20T02:35:05.808 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug frag = True 2022-07-20T02:35:05.808 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug scatterstat = True 2022-07-20T02:35:05.809 INFO:teuthology.orchestra.run.smithi176.stdout: mds op complaint time = 180 2022-07-20T02:35:05.809 INFO:teuthology.orchestra.run.smithi176.stdout: mds verify scatter = True 2022-07-20T02:35:05.809 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2022-07-20T02:35:05.809 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 15m 2022-07-20T02:35:05.809 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 15m 2022-07-20T02:35:05.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:35:06.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 Processor -- start 2022-07-20T02:35:06.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 -- start start 2022-07-20T02:35:06.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11240b5ef0 0x7f11241474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f11241479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f11240a8c40 0x7f11240aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1124147eb0 con 0x7f11240b5ef0 2022-07-20T02:35:06.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11240b62d0 con 0x7f11240a8c40 2022-07-20T02:35:06.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1129be0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112414aac0 con 0x7f11240a7fc0 2022-07-20T02:35:06.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11240b5ef0 0x7f11241474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f11240b5ef0 0x7f11241474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43414/0 (socket says 172.21.15.26:43414) 2022-07-20T02:35:06.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 -- 172.21.15.26:0/2548322514 learned_addr learned my addr 172.21.15.26:0/2548322514 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:06.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1123fff700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f11241479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f11293df700 1 --2- 172.21.15.26:0/2548322514 >> v2:172.21.15.120:3300/0 conn(0x7f11240a8c40 0x7f11240aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 -- 172.21.15.26:0/2548322514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 msgr2=0x7f11241479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f11241479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 -- 172.21.15.26:0/2548322514 >> v2:172.21.15.120:3300/0 conn(0x7f11240a8c40 msgr2=0x7f11240aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 --2- 172.21.15.26:0/2548322514 >> v2:172.21.15.120:3300/0 conn(0x7f11240a8c40 0x7f11240aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 -- 172.21.15.26:0/2548322514 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112414ad60 con 0x7f11240b5ef0 2022-07-20T02:35:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.987+0000 7f1128bde700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241474d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f1118002f70 tx=0x7f1118006eb0).ready entity=mon.0 client_cookie=4c5776fed331cec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f11237fe700 1 -- 172.21.15.26:0/2548322514 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f111800d570 con 0x7f11240b5ef0 2022-07-20T02:35:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f11237fe700 1 -- 172.21.15.26:0/2548322514 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118000e90 con 0x7f11240b5ef0 2022-07-20T02:35:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f11237fe700 1 -- 172.21.15.26:0/2548322514 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f111800d900 con 0x7f11240b5ef0 2022-07-20T02:35:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- 172.21.15.26:0/2548322514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 msgr2=0x7f11241474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241474d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f1118002f70 tx=0x7f1118006eb0).stop 2022-07-20T02:35:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- 172.21.15.26:0/2548322514 shutdown_connections 2022-07-20T02:35:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 --2- 172.21.15.26:0/2548322514 >> v2:172.21.15.120:3300/0 conn(0x7f11240a8c40 0x7f11240aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f11241479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 --2- 172.21.15.26:0/2548322514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241474d0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- 172.21.15.26:0/2548322514 >> 172.21.15.26:0/2548322514 conn(0x7f1124060310 msgr2=0x7f11240b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- 172.21.15.26:0/2548322514 shutdown_connections 2022-07-20T02:35:06.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- 172.21.15.26:0/2548322514 wait complete. 2022-07-20T02:35:06.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 Processor -- start 2022-07-20T02:35:06.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.991+0000 7f1129be0700 1 -- start start 2022-07-20T02:35:06.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f1124063900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11240a8c40 0x7f1124147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1124149390 con 0x7f11240b5ef0 2022-07-20T02:35:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1124066100 con 0x7f11240a8c40 2022-07-20T02:35:06.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1124064cb0 con 0x7f11240a7fc0 2022-07-20T02:35:06.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43424/0 (socket says 172.21.15.26:43424) 2022-07-20T02:35:06.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 -- 172.21.15.26:0/2010406619 learned_addr learned my addr 172.21.15.26:0/2010406619 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:06.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1128bde700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f1124063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1123fff700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11240a8c40 0x7f1124147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 -- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 msgr2=0x7f1124063900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f1124063900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 -- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11240a8c40 msgr2=0x7f1124147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11240a8c40 0x7f1124147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 -- 172.21.15.26:0/2010406619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111800b1e0 con 0x7f11240b5ef0 2022-07-20T02:35:06.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f11293df700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f111c00c000 tx=0x7f111c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f111c00c670 con 0x7f11240b5ef0 2022-07-20T02:35:06.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c00c7d0 con 0x7f11240b5ef0 2022-07-20T02:35:06.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1124062e10 con 0x7f11240b5ef0 2022-07-20T02:35:06.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f111c00cb30 con 0x7f11240b5ef0 2022-07-20T02:35:06.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11240630b0 con 0x7f11240b5ef0 2022-07-20T02:35:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112414b000 con 0x7f11240b5ef0 2022-07-20T02:35:06.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.995+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f111c022070 con 0x7f11240b5ef0 2022-07-20T02:35:06.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.999+0000 7f1121ffb700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 0x7f1104055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.999+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f111c06c340 con 0x7f11240b5ef0 2022-07-20T02:35:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.999+0000 7f1128bde700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 0x7f1104055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:05.999+0000 7f1128bde700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 0x7f1104055ed0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f111801c000 tx=0x7f1118015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.007+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f111c0433e0 con 0x7f11240b5ef0 2022-07-20T02:35:06.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.135+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f11240574e0 con 0x7f11240b5ef0 2022-07-20T02:35:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.139+0000 7f1121ffb700 1 -- 172.21.15.26:0/2010406619 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f111c046a00 con 0x7f11240b5ef0 2022-07-20T02:35:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 msgr2=0x7f1104055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 0x7f1104055ed0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f111801c000 tx=0x7f1118015000).stop 2022-07-20T02:35:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 msgr2=0x7f11241567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f111c00c000 tx=0x7f111c0091d0).stop 2022-07-20T02:35:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 shutdown_connections 2022-07-20T02:35:06.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11240b5ef0 0x7f11241567e0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1104053a10 0x7f1104055ed0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11240a8c40 0x7f1124147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 --2- 172.21.15.26:0/2010406619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11240a7fc0 0x7f1124063900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 >> 172.21.15.26:0/2010406619 conn(0x7f1124060310 msgr2=0x7f1124149a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 shutdown_connections 2022-07-20T02:35:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.143+0000 7f1129be0700 1 -- 172.21.15.26:0/2010406619 wait complete. 2022-07-20T02:35:06.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-20T02:35:06.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 Processor -- start 2022-07-20T02:35:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 -- start start 2022-07-20T02:35:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9cd40a8c40 0x7f9cd40af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd40b62d0 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd414aac0 con 0x7f9cd40a8c40 2022-07-20T02:35:06.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.427+0000 7f9cd9975700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd414ad60 con 0x7f9cd40b5ef0 2022-07-20T02:35:06.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9cd40a8c40 0x7f9cd40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd3fff700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9cd40a7fc0 0x7f9cd40aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:56920/0 (socket says 172.21.15.26:56920) 2022-07-20T02:35:06.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd3fff700 1 -- 172.21.15.26:0/782094250 learned_addr learned my addr 172.21.15.26:0/782094250 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:06.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd8973700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 -- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 msgr2=0x7f9cd41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd41474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 -- 172.21.15.26:0/782094250 >> v1:172.21.15.120:6789/0 conn(0x7f9cd40a7fc0 legacy=0x7f9cd40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 -- 172.21.15.26:0/782094250 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd414b000 con 0x7f9cd40a8c40 2022-07-20T02:35:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd8973700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd41474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9174700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40af550 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9ccc0032f0 tx=0x7f9ccc0091d0).ready entity=mon.1 client_cookie=55c79eba71eeafff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd37fe700 1 -- 172.21.15.26:0/782094250 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ccc00b6c0 con 0x7f9cd40a8c40 2022-07-20T02:35:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd37fe700 1 -- 172.21.15.26:0/782094250 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00b820 con 0x7f9cd40a8c40 2022-07-20T02:35:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd37fe700 1 -- 172.21.15.26:0/782094250 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ccc00b990 con 0x7f9cd40a8c40 2022-07-20T02:35:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 -- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a8c40 msgr2=0x7f9cd40af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40af550 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9ccc0032f0 tx=0x7f9ccc0091d0).stop 2022-07-20T02:35:06.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 -- 172.21.15.26:0/782094250 shutdown_connections 2022-07-20T02:35:06.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40af550 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 --2- 172.21.15.26:0/782094250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd41474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 -- 172.21.15.26:0/782094250 >> 172.21.15.26:0/782094250 conn(0x7f9cd4060310 msgr2=0x7f9cd40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 -- 172.21.15.26:0/782094250 shutdown_connections 2022-07-20T02:35:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.431+0000 7f9cd9975700 1 -- 172.21.15.26:0/782094250 wait complete. 2022-07-20T02:35:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 Processor -- start 2022-07-20T02:35:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 -- start start 2022-07-20T02:35:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 0x7f9cd40aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd4156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40ab230 con 0x7f9cd40a8c40 2022-07-20T02:35:06.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40ab490 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd9975700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40ab730 con 0x7f9cd40b5ef0 2022-07-20T02:35:06.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43436/0 (socket says 172.21.15.26:43436) 2022-07-20T02:35:06.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.435+0000 7f9cd3fff700 1 -- 172.21.15.26:0/1186567769 learned_addr learned my addr 172.21.15.26:0/1186567769 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:06.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 0x7f9cd40aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd9174700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd4156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 -- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 msgr2=0x7f9cd4156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd4156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 -- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 msgr2=0x7f9cd40b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 -- 172.21.15.26:0/1186567769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc800b180 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd3fff700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:06.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd8973700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 0x7f9cd40aabf0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f9cc800c000 tx=0x7f9cc8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cc8010780 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc80108e0 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ccc009470 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cc8010c40 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd4147d40 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd414b2a0 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.439+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9cc801c460 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.443+0000 7f9cd1ffb700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 0x7f9cb4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.443+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f9cc806c1a0 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.443+0000 7f9cd3fff700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 0x7f9cb4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.443+0000 7f9cd3fff700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 0x7f9cb4055ed0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f9cc400d000 tx=0x7f9cc4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.447+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cc8043140 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.619+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f9cd40574e0 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.671+0000 7f9cd1ffb700 1 -- 172.21.15.26:0/1186567769 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v446) v1 ==== 128+0+0 (secure 0 0 0) 0x7f9cc8046760 con 0x7f9cd40a7fc0 2022-07-20T02:35:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 msgr2=0x7f9cb4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 0x7f9cb4055ed0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f9cc400d000 tx=0x7f9cc4009040).stop 2022-07-20T02:35:06.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 msgr2=0x7f9cd40aabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 0x7f9cd40aabf0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f9cc800c000 tx=0x7f9cc8006df0).stop 2022-07-20T02:35:06.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 shutdown_connections 2022-07-20T02:35:06.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9cd40a8c40 0x7f9cd40b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9cb4053a10 0x7f9cb4055ed0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd40b5ef0 0x7f9cd4156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 --2- 172.21.15.26:0/1186567769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd40a7fc0 0x7f9cd40aabf0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 >> 172.21.15.26:0/1186567769 conn(0x7f9cd4060310 msgr2=0x7f9cd40aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:06.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 shutdown_connections 2022-07-20T02:35:06.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.675+0000 7f9cd9975700 1 -- 172.21.15.26:0/1186567769 wait complete. 2022-07-20T02:35:06.702 INFO:tasks.cephfs_test_runner:test_getattr_caps (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:35:06.702 INFO:tasks.cephfs_test_runner:Starting test: test_root_rctime (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:35:06.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-20T02:35:06.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 Processor -- start 2022-07-20T02:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 -- start start 2022-07-20T02:35:06.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8008060550 0x7f8008066c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.886 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080609c0 con 0x7f800805f9e0 2022-07-20T02:35:06.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080b13f0 con 0x7f8008060550 2022-07-20T02:35:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e9c2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080b1690 con 0x7f80080b0f50 2022-07-20T02:35:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800e1c1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8008060550 0x7f8008066c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:36728/0 (socket says 172.21.15.26:36728) 2022-07-20T02:35:06.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 -- 172.21.15.26:0/2463462187 learned_addr learned my addr 172.21.15.26:0/2463462187 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:06.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 -- 172.21.15.26:0/2463462187 >> v2:172.21.15.120:3300/0 conn(0x7f8008060550 msgr2=0x7f8008066c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:06.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 --2- 172.21.15.26:0/2463462187 >> v2:172.21.15.120:3300/0 conn(0x7f8008060550 0x7f8008066c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 -- 172.21.15.26:0/2463462187 >> v1:172.21.15.120:6789/0 conn(0x7f800805f9e0 legacy=0x7f80080b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 -- 172.21.15.26:0/2463462187 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080b1930 con 0x7f80080b0f50 2022-07-20T02:35:06.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800d1bf700 1 --2- 172.21.15.26:0/2463462187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7ff80032f0 tx=0x7f7ff8009290).ready entity=mon.2 client_cookie=5bf159069453b5ef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800c9be700 1 -- 172.21.15.26:0/2463462187 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff800b8a0 con 0x7f80080b0f50 2022-07-20T02:35:06.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800c9be700 1 -- 172.21.15.26:0/2463462187 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff800ba00 con 0x7f80080b0f50 2022-07-20T02:35:06.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.827+0000 7f800c9be700 1 -- 172.21.15.26:0/2463462187 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff800bb70 con 0x7f80080b0f50 2022-07-20T02:35:06.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- 172.21.15.26:0/2463462187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 msgr2=0x7f80081473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2463462187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7ff80032f0 tx=0x7f7ff8009290).stop 2022-07-20T02:35:06.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- 172.21.15.26:0/2463462187 shutdown_connections 2022-07-20T02:35:06.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2463462187 >> v2:172.21.15.120:3300/0 conn(0x7f8008060550 0x7f8008066c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2463462187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080b0f50 0x7f80081473e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- 172.21.15.26:0/2463462187 >> 172.21.15.26:0/2463462187 conn(0x7f80080b3050 msgr2=0x7f80080b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:06.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- 172.21.15.26:0/2463462187 shutdown_connections 2022-07-20T02:35:06.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- 172.21.15.26:0/2463462187 wait complete. 2022-07-20T02:35:06.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 Processor -- start 2022-07-20T02:35:06.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- start start 2022-07-20T02:35:06.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 0x7f80080ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80080b0f50 0x7f80080ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008159f20 con 0x7f8008060550 2022-07-20T02:35:06.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008064a20 con 0x7f80080b0f50 2022-07-20T02:35:06.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e9c2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8008063df0 con 0x7f800805f9e0 2022-07-20T02:35:06.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800d1bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 0x7f80080ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800d9c0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800d9c0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:36734/0 (socket says 172.21.15.26:36734) 2022-07-20T02:35:06.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800d9c0700 1 -- 172.21.15.26:0/2173694868 learned_addr learned my addr 172.21.15.26:0/2173694868 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.831+0000 7f800e1c1700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80080b0f50 0x7f80080ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 -- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 msgr2=0x7f80081455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 -- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80080b0f50 msgr2=0x7f80080ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80080b0f50 0x7f80080ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 -- 172.21.15.26:0/2173694868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009530 con 0x7f8008060550 2022-07-20T02:35:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d9c0700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d1bf700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 0x7f80080ac390 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f7ff80032f0 tx=0x7f7ff8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff800b8a0 con 0x7f8008060550 2022-07-20T02:35:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff8021440 con 0x7f8008060550 2022-07-20T02:35:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8008064090 con 0x7f8008060550 2022-07-20T02:35:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff80217a0 con 0x7f8008060550 2022-07-20T02:35:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080abff0 con 0x7f8008060550 2022-07-20T02:35:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8008064be0 con 0x7f8008060550 2022-07-20T02:35:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7ff800f070 con 0x7f8008060550 2022-07-20T02:35:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 0x7f7fe8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f7ff806d110 con 0x7f8008060550 2022-07-20T02:35:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.835+0000 7f800d9c0700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 0x7f7fe8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.839+0000 7f800d9c0700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 0x7f7fe8055ea0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f7ffc00d000 tx=0x7f7ffc009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:06.843+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7ff80441b0 con 0x7f8008060550 2022-07-20T02:35:07.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.007+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f80080574e0 con 0x7f8008060550 2022-07-20T02:35:07.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.675+0000 7f8006ffd700 1 -- 172.21.15.26:0/2173694868 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v447) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7ff8015020 con 0x7f8008060550 2022-07-20T02:35:07.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 msgr2=0x7f7fe8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 0x7f7fe8055ea0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f7ffc00d000 tx=0x7f7ffc009040).stop 2022-07-20T02:35:07.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 msgr2=0x7f80080ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 0x7f80080ac390 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f7ff80032f0 tx=0x7f7ff8016000).stop 2022-07-20T02:35:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 shutdown_connections 2022-07-20T02:35:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8008060550 0x7f80080ac390 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7fe80539e0 0x7f7fe8055ea0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80080b0f50 0x7f80080ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 --2- 172.21.15.26:0/2173694868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f800805f9e0 0x7f80081455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 >> 172.21.15.26:0/2173694868 conn(0x7f80080b3050 msgr2=0x7f8008066bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:07.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 shutdown_connections 2022-07-20T02:35:07.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.679+0000 7f800e9c2700 1 -- 172.21.15.26:0/2173694868 wait complete. 2022-07-20T02:35:07.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:35:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.815+0000 7f849ef6a700 1 Processor -- start 2022-07-20T02:35:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.815+0000 7f849ef6a700 1 -- start start 2022-07-20T02:35:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f8498098eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8498097d80 0x7f84980b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 0x7f849809b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84980b1f00 con 0x7f8498097d80 2022-07-20T02:35:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8498098370 con 0x7f849805fef0 2022-07-20T02:35:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8498098610 con 0x7f84980b1b20 2022-07-20T02:35:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8498097d80 0x7f84980b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8498097d80 0x7f84980b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43460/0 (socket says 172.21.15.26:43460) 2022-07-20T02:35:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 -- 172.21.15.26:0/296419193 learned_addr learned my addr 172.21.15.26:0/296419193 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849e769700 1 --2- 172.21.15.26:0/296419193 >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 0x7f849809b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849df68700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f8498098eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 -- 172.21.15.26:0/296419193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 msgr2=0x7f8498098eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f8498098eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 -- 172.21.15.26:0/296419193 >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 msgr2=0x7f849809b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 --2- 172.21.15.26:0/296419193 >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 0x7f849809b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 -- 172.21.15.26:0/296419193 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980b2a00 con 0x7f8498097d80 2022-07-20T02:35:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849e769700 1 --2- 172.21.15.26:0/296419193 >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 0x7f849809b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849d767700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8498097d80 0x7f84980b2ea0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f8488002f70 tx=0x7f8488006eb0).ready entity=mon.0 client_cookie=d8f735f33814a1ee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:07.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849cf66700 1 -- 172.21.15.26:0/296419193 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848800d570 con 0x7f8498097d80 2022-07-20T02:35:07.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849cf66700 1 -- 172.21.15.26:0/296419193 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488000e90 con 0x7f8498097d80 2022-07-20T02:35:07.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849cf66700 1 -- 172.21.15.26:0/296419193 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848800d900 con 0x7f8498097d80 2022-07-20T02:35:07.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- 172.21.15.26:0/296419193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8498097d80 msgr2=0x7f84980b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8498097d80 0x7f84980b2ea0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f8488002f70 tx=0x7f8488006eb0).stop 2022-07-20T02:35:07.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- 172.21.15.26:0/296419193 shutdown_connections 2022-07-20T02:35:07.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- 172.21.15.26:0/296419193 >> v2:172.21.15.120:3300/0 conn(0x7f849805fef0 0x7f849809b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8498097d80 0x7f84980b2ea0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 --2- 172.21.15.26:0/296419193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f8498098eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- 172.21.15.26:0/296419193 >> 172.21.15.26:0/296419193 conn(0x7f8498063030 msgr2=0x7f8498063410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:07.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- 172.21.15.26:0/296419193 shutdown_connections 2022-07-20T02:35:07.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.819+0000 7f849ef6a700 1 -- 172.21.15.26:0/296419193 wait complete. 2022-07-20T02:35:07.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 Processor -- start 2022-07-20T02:35:07.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- start start 2022-07-20T02:35:07.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f84980666b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8498066ba0 0x7f8498143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8498148d40 con 0x7f849805fef0 2022-07-20T02:35:07.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980661b0 con 0x7f8498066ba0 2022-07-20T02:35:07.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980ac8a0 con 0x7f84980b1b20 2022-07-20T02:35:07.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43464/0 (socket says 172.21.15.26:43464) 2022-07-20T02:35:07.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 -- 172.21.15.26:0/1084522493 learned_addr learned my addr 172.21.15.26:0/1084522493 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:07.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849e769700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8498066ba0 0x7f8498143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849d767700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f84980666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 -- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 msgr2=0x7f84980666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f84980666b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 -- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8498066ba0 msgr2=0x7f8498143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8498066ba0 0x7f8498143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 -- 172.21.15.26:0/1084522493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848800b1e0 con 0x7f849805fef0 2022-07-20T02:35:07.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849df68700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f84940030b0 tx=0x7f8494008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:07.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8494011780 con 0x7f849805fef0 2022-07-20T02:35:07.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84940118e0 con 0x7f849805fef0 2022-07-20T02:35:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84980654a0 con 0x7f849805fef0 2022-07-20T02:35:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8494011c40 con 0x7f849805fef0 2022-07-20T02:35:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.823+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8498146420 con 0x7f849805fef0 2022-07-20T02:35:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84980aca60 con 0x7f849805fef0 2022-07-20T02:35:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8494012420 con 0x7f849805fef0 2022-07-20T02:35:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f8486ffd700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 0x7f8478055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f849406bbd0 con 0x7f849805fef0 2022-07-20T02:35:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f849d767700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 0x7f8478055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.827+0000 7f849d767700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 0x7f8478055fc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f848801c000 tx=0x7f8488015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.835+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8494042b70 con 0x7f849805fef0 2022-07-20T02:35:07.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.959+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f84980b2720 con 0x7f849805fef0 2022-07-20T02:35:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.963+0000 7f8486ffd700 1 -- 172.21.15.26:0/1084522493 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8494046190 con 0x7f849805fef0 2022-07-20T02:35:07.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.963+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 msgr2=0x7f8478055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.963+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 0x7f8478055fc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f848801c000 tx=0x7f8488015000).stop 2022-07-20T02:35:07.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.963+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 msgr2=0x7f84980af4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:07.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.963+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f84940030b0 tx=0x7f8494008040).stop 2022-07-20T02:35:07.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 shutdown_connections 2022-07-20T02:35:07.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8478053b00 0x7f8478055fc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8498066ba0 0x7f8498143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84980b1b20 0x7f84980666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 --2- 172.21.15.26:0/1084522493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f849805fef0 0x7f84980af4d0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:07.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 >> 172.21.15.26:0/1084522493 conn(0x7f8498063030 msgr2=0x7f849809b400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:07.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 shutdown_connections 2022-07-20T02:35:07.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:07.967+0000 7f849ef6a700 1 -- 172.21.15.26:0/1084522493 wait complete. 2022-07-20T02:35:08.010 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:35:08.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:08.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:08.105 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:08.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:08.178 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:35:08.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:08.179 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:08.186 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:08.200 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:35:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:35:08.215 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:35:08.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:08.222 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:08.223 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:35:08.246 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:35:08.246 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:08.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:35:08.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 Processor -- start 2022-07-20T02:35:08.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 -- start start 2022-07-20T02:35:08.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcfac0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0b62d0 con 0x7fcfac147be0 2022-07-20T02:35:08.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac14aac0 con 0x7fcfac0b5ef0 2022-07-20T02:35:08.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb302e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac14ad60 con 0x7fcfac0a7fc0 2022-07-20T02:35:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb182b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcfac147be0 0x7fcfac0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42752/0 (socket says 172.21.15.26:42752) 2022-07-20T02:35:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb182b700 1 -- 172.21.15.26:0/2775100999 learned_addr learned my addr 172.21.15.26:0/2775100999 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:08.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb282d700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcfac0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337828410 0 0) 0x7fcfac0b62d0 con 0x7fcfac147be0 2022-07-20T02:35:08.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf94003260 con 0x7fcfac147be0 2022-07-20T02:35:08.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607005069 0 0) 0x7fcf94003260 con 0x7fcfac147be0 2022-07-20T02:35:08.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 msgr2=0x7fcfac0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:08.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcfac0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:08.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 >> v1:172.21.15.120:6789/0 conn(0x7fcfac0b5ef0 legacy=0x7fcfac1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:08.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac14b000 con 0x7fcfac147be0 2022-07-20T02:35:08.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7fcfa80019e0 con 0x7fcfac147be0 2022-07-20T02:35:08.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 >> v1:172.21.15.26:6789/0 conn(0x7fcfac147be0 legacy=0x7fcfac0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:08.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94003780 0x7fcf94003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf94004500 0x7fcf94004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcf94004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac14ad60 con 0x7fcf94004500 2022-07-20T02:35:08.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9400b750 con 0x7fcf94003780 2022-07-20T02:35:08.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9400b8c0 con 0x7fcfac0a7fc0 2022-07-20T02:35:08.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb182b700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf94004500 0x7fcf94004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb202c700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94003780 0x7fcf94003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.379+0000 7fcfb282d700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcf94004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 -- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 msgr2=0x7fcf94004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:08.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0a7fc0 0x7fcf94004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:08.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 -- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94003780 msgr2=0x7fcf94003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:08.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94003780 0x7fcf94003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:08.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 -- 172.21.15.26:0/2775100999 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94003260 con 0x7fcf94004500 2022-07-20T02:35:08.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb182b700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf94004500 0x7fcf94004980 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7fcfa80016f0 tx=0x7fcfac0ace20).ready entity=mon.0 client_cookie=57aecb39ca86cd4e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:08.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa800cce0 con 0x7fcf94004500 2022-07-20T02:35:08.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa800ce40 con 0x7fcf94004500 2022-07-20T02:35:08.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb102a700 1 -- 172.21.15.26:0/2775100999 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa8005b80 con 0x7fcf94004500 2022-07-20T02:35:08.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- 172.21.15.26:0/2775100999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf94004500 msgr2=0x7fcf94004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:08.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 --2- 172.21.15.26:0/2775100999 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcf94004500 0x7fcf94004980 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7fcfa80016f0 tx=0x7fcfac0ace20).stop 2022-07-20T02:35:08.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb282d700 1 -- 172.21.15.26:0/2775100999 reap_dead start 2022-07-20T02:35:08.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- 172.21.15.26:0/2775100999 shutdown_connections 2022-07-20T02:35:08.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- 172.21.15.26:0/2775100999 >> 172.21.15.26:0/2775100999 conn(0x7fcfac060310 msgr2=0x7fcfac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:08.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- 172.21.15.26:0/2775100999 shutdown_connections 2022-07-20T02:35:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- 172.21.15.26:0/2775100999 wait complete. 2022-07-20T02:35:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 Processor -- start 2022-07-20T02:35:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.383+0000 7fcfb302e700 1 -- start start 2022-07-20T02:35:08.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94004500 0x7fcfac0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 0x7fcfac156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac065000 con 0x7fcfac064b60 2022-07-20T02:35:08.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0652a0 con 0x7fcf94004500 2022-07-20T02:35:08.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac065540 con 0x7fcfac0b42d0 2022-07-20T02:35:08.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43484/0 (socket says 172.21.15.26:43484) 2022-07-20T02:35:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 -- 172.21.15.26:0/383710809 learned_addr learned my addr 172.21.15.26:0/383710809 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb202c700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94004500 0x7fcfac0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb282d700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 0x7fcfac156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 -- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 msgr2=0x7fcfac156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 0x7fcfac156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 -- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94004500 msgr2=0x7fcfac0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94004500 0x7fcfac0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 -- 172.21.15.26:0/383710809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac1565c0 con 0x7fcfac064b60 2022-07-20T02:35:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb282d700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 0x7fcfac156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb182b700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fcfa80016f0 tx=0x7fcfa8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa8001f70 con 0x7fcfac064b60 2022-07-20T02:35:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa80074d0 con 0x7fcfac064b60 2022-07-20T02:35:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac0a92b0 con 0x7fcfac064b60 2022-07-20T02:35:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa8007830 con 0x7fcfac064b60 2022-07-20T02:35:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac147880 con 0x7fcfac064b60 2022-07-20T02:35:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcfa8010070 con 0x7fcfac064b60 2022-07-20T02:35:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.387+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac14b2a0 con 0x7fcfac064b60 2022-07-20T02:35:08.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.391+0000 7fcf9b7fe700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 0x7fcf8c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.391+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fcfa806ce00 con 0x7fcfac064b60 2022-07-20T02:35:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.391+0000 7fcfb202c700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 0x7fcf8c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.391+0000 7fcfb202c700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 0x7fcf8c055f60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fcf9c00c000 tx=0x7fcf9c007a00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.395+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcfa8043ea0 con 0x7fcfac064b60 2022-07-20T02:35:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.519+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fcfac0574e0 con 0x7fcfac064b60 2022-07-20T02:35:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.523+0000 7fcf9b7fe700 1 -- 172.21.15.26:0/383710809 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+29277 (secure 0 0 0) 0x7fcfa80474c0 con 0x7fcfac064b60 2022-07-20T02:35:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:08.567 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:35:08.567 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 143, 2022-07-20T02:35:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:35:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:35:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:35:04.632703+0000", 2022-07-20T02:35:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:08.568 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:35:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:35:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:35:08.569 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:35:08.577 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:35:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:35:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:35:08.578 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:35:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:35:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:35:08.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:35:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:35:08.580 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 26, 2022-07-20T02:35:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:35:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:35:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:35:08.581 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:35:08.589 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:35:08.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.589 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:35:08.590 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:35:08.590 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:35:08.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:08.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:08.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:08.591 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:08.591 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:08.591 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:08.592 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:08.592 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:08.596 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:35:08.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:35:08.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:35:08.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:35:08.598 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:35:08.598 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:08.598 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:08.598 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:08.599 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:08.599 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:08.599 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:08.599 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:08.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.600 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:35:08.600 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:08.600 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:08.601 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:08.601 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:08.602 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:08.602 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:08.602 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:08.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:08.605 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:08.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:08.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:35:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:35:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.639 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:35:08.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:35:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:35:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:35:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:35:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:35:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:35:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:35:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:35:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:35:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:08.682 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:08.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:08.684 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:08.685 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:08.686 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:08.687 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:08.688 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:08.689 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:08.690 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:08.692 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:08.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.703 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:08.704 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:08.705 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:08.712 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:08.713 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:35:08.714 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:35:08.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.716 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:35:08.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:08.718 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:35:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:08.720 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:08.721 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:08.722 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:35:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:08.723 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:08.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:35:08.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:35:08.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.755 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:08.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:35:08.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:35:08.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.771 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:08.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:35:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:35:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:35:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:35:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:35:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:35:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:35:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:35:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.803 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:08.803 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:35:08.804 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:35:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:35:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:35:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:35:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:35:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:35:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:35:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:35:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:35:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:35:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:35:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:35:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:35:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:35:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:35:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:35:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:35:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:35:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:35:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:35:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:35:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:35:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:35:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:35:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:35:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:35:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:35:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:35:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:35:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:35:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:35:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:35:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:35:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:35:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:35:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:35:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:35:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.978 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:35:08.978 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:35:08.978 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:35:08.979 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:35:08.979 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:08.979 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:08.979 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:08.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:08.981 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:35:08.981 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:08.981 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:08.982 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:08.982 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:08.982 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:08.982 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:08.983 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:08.983 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:35:08.983 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:08.983 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:08.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:08.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:08.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:35:08.991 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:08.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:08.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:08.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:35:08.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:08.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:08.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:08.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:08.993 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:08.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:08.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:35:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:35:09.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:35:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:35:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:35:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:35:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:09.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:35:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:35:09.037 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:35:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:35:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:09.038 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.040 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:35:09.065 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:09.067 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:09.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:09.069 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:09.072 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 130, 2022-07-20T02:35:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:35:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:35:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:35:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:35:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:35:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:35:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:35:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:35:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:35:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:35:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:35:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:35:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:35:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:35:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:35:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:35:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:35:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:35:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:35:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:35:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:35:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:35:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:35:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:35:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:35:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:35:09.268 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:09.269 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:09.269 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:09.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.271 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:09.271 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:35:09.275 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:09.275 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:09.278 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:09.278 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:09.279 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:09.279 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:09.279 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:09.285 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 137, 2022-07-20T02:35:09.286 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:09.286 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:09.286 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:09.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.287 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:35:09.287 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.298 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:35:09.298 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.306 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.309 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:09.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.310 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:35:09.310 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.311 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:35:09.311 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.312 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.312 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:09.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:35:09.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:35:09.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.375 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:09.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:09.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:09.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:35:09.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:09.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:35:09.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:09.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:35:09.388 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:35:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:35:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:35:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:09.390 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:09.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:09.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.390 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:35:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:35:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:35:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.437 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:09.437 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.438 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.438 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.438 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.439 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.439 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:35:09.439 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.440 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:09.440 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.441 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.441 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.441 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.441 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.442 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:35:09.442 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.452 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:09.453 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.453 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.453 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.453 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.454 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.454 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:35:09.454 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.455 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:35:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.457 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:09.458 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.459 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.460 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.460 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.460 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:35:09.460 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:09.461 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:09.462 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:09.462 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:09.462 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:09.462 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:09.463 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:35:09.463 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:09.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.463 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:09.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:35:09.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:35:09.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:35:09.464 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:35:09.464 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:35:09.465 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/1171289829": "2022-07-21T02:35:00.698481+0000", 2022-07-20T02:35:09.465 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/1171289829": "2022-07-21T02:35:00.698481+0000" 2022-07-20T02:35:09.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.465 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:35:09.466 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:35:09.466 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:35:09.466 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:09.466 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:35:09.467 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:35:09.467 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:09.467 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:35:09.467 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:09.467 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:09.468 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:35:09.468 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:35:09.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.468 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:35:09.469 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:09.469 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:09.469 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:35:09.469 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:09.469 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:09.480 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:35:09.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:09.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:09.481 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:35:09.481 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:35:09.482 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:35:09.482 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:35:09.482 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:35:09.482 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 msgr2=0x7fcf8c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 0x7fcf8c055f60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fcf9c00c000 tx=0x7fcf9c007a00).stop 2022-07-20T02:35:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 msgr2=0x7fcfac147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fcfa80016f0 tx=0x7fcfa8015000).stop 2022-07-20T02:35:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 shutdown_connections 2022-07-20T02:35:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcfac064b60 0x7fcfac147390 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcf8c053aa0 0x7fcf8c055f60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfac0b42d0 0x7fcfac156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 --2- 172.21.15.26:0/383710809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf94004500 0x7fcfac0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 >> 172.21.15.26:0/383710809 conn(0x7fcfac060310 msgr2=0x7fcfac0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 shutdown_connections 2022-07-20T02:35:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:08.527+0000 7fcfb302e700 1 -- 172.21.15.26:0/383710809 wait complete. 2022-07-20T02:35:09.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 Processor -- start 2022-07-20T02:35:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 -- start start 2022-07-20T02:35:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f45f40b5ef0 0x7f45f41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45f40af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f40b62d0 con 0x7f45f40a7fc0 2022-07-20T02:35:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f414aac0 con 0x7f45f40b5ef0 2022-07-20T02:35:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45fa71f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f414ad60 con 0x7f45f40a8c40 2022-07-20T02:35:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45f8f1c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f45f40a7fc0 0x7f45f40aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42770/0 (socket says 172.21.15.26:42770) 2022-07-20T02:35:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2866664498 learned_addr learned my addr 172.21.15.26:0/2866664498 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.599+0000 7f45f971d700 1 --2- 172.21.15.26:0/2866664498 >> v2:172.21.15.26:3300/0 conn(0x7f45f40b5ef0 0x7f45f41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f9f1e700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45f40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714538262 0 0) 0x7f45f40b62d0 con 0x7f45f40a7fc0 2022-07-20T02:35:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e0003260 con 0x7f45f40a7fc0 2022-07-20T02:35:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3607645664 0 0) 0x7f45e0003260 con 0x7f45f40a7fc0 2022-07-20T02:35:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 msgr2=0x7f45f40af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45f40af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 >> v2:172.21.15.26:3300/0 conn(0x7f45f40b5ef0 msgr2=0x7f45f41474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 --2- 172.21.15.26:0/2866664498 >> v2:172.21.15.26:3300/0 conn(0x7f45f40b5ef0 0x7f45f41474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f414b000 con 0x7f45f40a7fc0 2022-07-20T02:35:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f45e40019e0 con 0x7f45f40a7fc0 2022-07-20T02:35:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 >> v1:172.21.15.26:6789/0 conn(0x7f45f40a7fc0 legacy=0x7f45f40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e0003750 0x7f45e0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45e0004500 0x7f45e0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45e0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f414aac0 con 0x7f45e0004500 2022-07-20T02:35:09.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f414ad60 con 0x7f45e0003750 2022-07-20T02:35:09.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e000b750 con 0x7f45f40a8c40 2022-07-20T02:35:09.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45e0004500 0x7f45e0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f9f1e700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45e0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f971d700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e0003750 0x7f45e0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 msgr2=0x7f45e0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:09.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40a8c40 0x7f45e0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e0003750 msgr2=0x7f45e0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e0003750 0x7f45e0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2866664498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0003260 con 0x7f45e0004500 2022-07-20T02:35:09.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f971d700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e0003750 0x7f45e0003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45e0004500 0x7f45e0004980 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f45e40016f0 tx=0x7f45e40021d0).ready entity=mon.0 client_cookie=2f00668a4a97370c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e400cc50 con 0x7f45e0004500 2022-07-20T02:35:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e400cdb0 con 0x7f45e0004500 2022-07-20T02:35:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f3fff700 1 -- 172.21.15.26:0/2866664498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e4005b80 con 0x7f45e0004500 2022-07-20T02:35:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 -- 172.21.15.26:0/2866664498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45e0004500 msgr2=0x7f45e0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2866664498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45e0004500 0x7f45e0004980 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f45e40016f0 tx=0x7f45e40021d0).stop 2022-07-20T02:35:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45f9f1e700 1 -- 172.21.15.26:0/2866664498 reap_dead start 2022-07-20T02:35:09.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 -- 172.21.15.26:0/2866664498 shutdown_connections 2022-07-20T02:35:09.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 -- 172.21.15.26:0/2866664498 >> 172.21.15.26:0/2866664498 conn(0x7f45f4060310 msgr2=0x7f45f40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:09.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 -- 172.21.15.26:0/2866664498 shutdown_connections 2022-07-20T02:35:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.603+0000 7f45fa71f700 1 -- 172.21.15.26:0/2866664498 wait complete. 2022-07-20T02:35:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 Processor -- start 2022-07-20T02:35:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 -- start start 2022-07-20T02:35:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45e0004500 0x7f45f40b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f415a380 0x7f45f4066990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.607+0000 7f45fa71f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4159aa0 con 0x7f45f40aad00 2022-07-20T02:35:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45fa71f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40ab420 con 0x7f45f415a380 2022-07-20T02:35:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45fa71f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40626f0 con 0x7f45e0004500 2022-07-20T02:35:09.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f971d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45e0004500 0x7f45f40b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43504/0 (socket says 172.21.15.26:43504) 2022-07-20T02:35:09.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2271777155 learned_addr learned my addr 172.21.15.26:0/2271777155 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:09.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f9f1e700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f415a380 0x7f45f4066990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45e0004500 msgr2=0x7f45f40b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45e0004500 0x7f45f40b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f415a380 msgr2=0x7f45f4066990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f415a380 0x7f45f4066990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 -- 172.21.15.26:0/2271777155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40ab650 con 0x7f45f40aad00 2022-07-20T02:35:09.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f8f1c700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f45e40016f0 tx=0x7f45e4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e4001350 con 0x7f45f40aad00 2022-07-20T02:35:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e40072c0 con 0x7f45f40aad00 2022-07-20T02:35:09.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f40ab8f0 con 0x7f45f40aad00 2022-07-20T02:35:09.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e4007620 con 0x7f45f40aad00 2022-07-20T02:35:09.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f40aa900 con 0x7f45f40aad00 2022-07-20T02:35:09.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f40aba80 con 0x7f45f40aad00 2022-07-20T02:35:09.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.611+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f45e401d070 con 0x7f45f40aad00 2022-07-20T02:35:09.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.615+0000 7f45f27fc700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 0x7f45d4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.615+0000 7f45f971d700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 0x7f45d4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.615+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f45e406cca0 con 0x7f45f40aad00 2022-07-20T02:35:09.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.615+0000 7f45f971d700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 0x7f45d4055e10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f45e800b000 tx=0x7f45e8008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:09.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.619+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45e4043c30 con 0x7f45f40aad00 2022-07-20T02:35:09.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.787+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45f414b2a0 con 0x7f45f40aad00 2022-07-20T02:35:09.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.787+0000 7f45f27fc700 1 -- 172.21.15.26:0/2271777155 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f45e401a030 con 0x7f45f40aad00 2022-07-20T02:35:09.814 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:09.815 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/3723018029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3723018029},{"type":"v1","addr":"172.21.15.26:6837","nonce":3723018029}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[]} 2022-07-20T02:35:09.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 msgr2=0x7f45d4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 0x7f45d4055e10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f45e800b000 tx=0x7f45e8008040).stop 2022-07-20T02:35:09.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 msgr2=0x7f45f4159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f45e40016f0 tx=0x7f45e4015000).stop 2022-07-20T02:35:09.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 shutdown_connections 2022-07-20T02:35:09.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f45f40aad00 0x7f45f4159e90 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f45d4053950 0x7f45d4055e10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f415a380 0x7f45f4066990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 --2- 172.21.15.26:0/2271777155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45e0004500 0x7f45f40b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 >> 172.21.15.26:0/2271777155 conn(0x7f45f4060310 msgr2=0x7f45f4067950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:09.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 shutdown_connections 2022-07-20T02:35:09.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.791+0000 7f45fa71f700 1 -- 172.21.15.26:0/2271777155 wait complete. 2022-07-20T02:35:09.850 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 75 2022-07-20T02:35:09.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:09.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.971+0000 7f30d4bd0700 1 Processor -- start 2022-07-20T02:35:09.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.971+0000 7f30d4bd0700 1 -- start start 2022-07-20T02:35:09.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f30d00b5ef0 0x7f30d01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:09.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d00b62d0 con 0x7f30d00a7fc0 2022-07-20T02:35:09.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d014aac0 con 0x7f30d00a8c40 2022-07-20T02:35:09.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d014ad60 con 0x7f30d00b5ef0 2022-07-20T02:35:09.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30cf7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f30d00b5ef0 0x7f30d01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:09.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ceffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f30d00a7fc0 0x7f30d00aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42790/0 (socket says 172.21.15.26:42790) 2022-07-20T02:35:09.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ceffd700 1 -- 172.21.15.26:0/3474222587 learned_addr learned my addr 172.21.15.26:0/3474222587 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:09.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ce7fc700 1 -- 172.21.15.26:0/3474222587 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149819953 0 0) 0x7f30d00b62d0 con 0x7f30d00a7fc0 2022-07-20T02:35:09.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ce7fc700 1 -- 172.21.15.26:0/3474222587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30bc003260 con 0x7f30d00a7fc0 2022-07-20T02:35:09.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30cf7fe700 1 -- 172.21.15.26:0/3474222587 >> v1:172.21.15.120:6789/0 conn(0x7f30d00a8c40 legacy=0x7f30d0151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30cf7fe700 1 -- 172.21.15.26:0/3474222587 >> v1:172.21.15.26:6789/0 conn(0x7f30d00a7fc0 legacy=0x7f30d00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30cf7fe700 1 -- 172.21.15.26:0/3474222587 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d014b000 con 0x7f30d00b5ef0 2022-07-20T02:35:09.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30cf7fe700 1 --2- 172.21.15.26:0/3474222587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00b5ef0 0x7f30d01474d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f30c4002f70 tx=0x7f30c4006df0).ready entity=mon.0 client_cookie=eb528c9a43d33608 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:09.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ce7fc700 1 -- 172.21.15.26:0/3474222587 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c400d610 con 0x7f30d00b5ef0 2022-07-20T02:35:09.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ce7fc700 1 -- 172.21.15.26:0/3474222587 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4000e90 con 0x7f30d00b5ef0 2022-07-20T02:35:09.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30ce7fc700 1 -- 172.21.15.26:0/3474222587 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c400d960 con 0x7f30d00b5ef0 2022-07-20T02:35:09.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3474222587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00b5ef0 msgr2=0x7f30d01474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:09.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3474222587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00b5ef0 0x7f30d01474d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f30c4002f70 tx=0x7f30c4006df0).stop 2022-07-20T02:35:09.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3474222587 shutdown_connections 2022-07-20T02:35:09.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3474222587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00b5ef0 0x7f30d01474d0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:09.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3474222587 >> 172.21.15.26:0/3474222587 conn(0x7f30d0060310 msgr2=0x7f30d00b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:09.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3474222587 shutdown_connections 2022-07-20T02:35:09.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3474222587 wait complete. 2022-07-20T02:35:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 Processor -- start 2022-07-20T02:35:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.975+0000 7f30d4bd0700 1 -- start start 2022-07-20T02:35:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d00a7fc0 0x7f30d015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d00b5ef0 0x7f30d00ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0148440 con 0x7f30d00a8c40 2022-07-20T02:35:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d00abe00 con 0x7f30d00a7fc0 2022-07-20T02:35:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0159bc0 con 0x7f30d00b5ef0 2022-07-20T02:35:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43518/0 (socket says 172.21.15.26:43518) 2022-07-20T02:35:10.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 -- 172.21.15.26:0/3791200404 learned_addr learned my addr 172.21.15.26:0/3791200404 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:10.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30cffff700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d00b5ef0 0x7f30d00ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30cf7fe700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d00a7fc0 0x7f30d015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 -- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d00b5ef0 msgr2=0x7f30d00ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d00b5ef0 0x7f30d00ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 -- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d00a7fc0 msgr2=0x7f30d015a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d00a7fc0 0x7f30d015a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 -- 172.21.15.26:0/3791200404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c400b180 con 0x7f30d00a8c40 2022-07-20T02:35:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ceffd700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f30c0001490 tx=0x7f30c0007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c0005ba0 con 0x7f30d00a8c40 2022-07-20T02:35:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c0005d00 con 0x7f30d00a8c40 2022-07-20T02:35:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d0159e60 con 0x7f30d00a8c40 2022-07-20T02:35:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c0007300 con 0x7f30d00a8c40 2022-07-20T02:35:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.979+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d0065e30 con 0x7f30d00a8c40 2022-07-20T02:35:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d014b2a0 con 0x7f30d00a8c40 2022-07-20T02:35:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f30c0010030 con 0x7f30d00a8c40 2022-07-20T02:35:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30ccff9700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 0x7f30b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f30c006be90 con 0x7f30d00a8c40 2022-07-20T02:35:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30cf7fe700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 0x7f30b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.983+0000 7f30cf7fe700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 0x7f30b0055fc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f30c4018000 tx=0x7f30c4014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:09.991+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f30c0042d40 con 0x7f30d00a8c40 2022-07-20T02:35:10.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30d00574e0 con 0x7f30d00a8c40 2022-07-20T02:35:10.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30ccff9700 1 -- 172.21.15.26:0/3791200404 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+1562 (secure 0 0 0) 0x7f30c0046360 con 0x7f30d00a8c40 2022-07-20T02:35:10.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:10.160 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7888,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/3448149855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3448149855},{"type":"v1","addr":"172.21.15.120:6833","nonce":3448149855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":7900,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6835/971480621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":971480621},{"type":"v1","addr":"172.21.15.26:6835","nonce":971480621}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/3723018029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":3723018029},{"type":"v1","addr":"172.21.15.26:6837","nonce":3723018029}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[]} 2022-07-20T02:35:10.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 msgr2=0x7f30b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 0x7f30b0055fc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f30c4018000 tx=0x7f30c4014000).stop 2022-07-20T02:35:10.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 msgr2=0x7f30d00af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f30c0001490 tx=0x7f30c0007e90).stop 2022-07-20T02:35:10.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 shutdown_connections 2022-07-20T02:35:10.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f30d00a8c40 0x7f30d00af1c0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.175 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f30b0053b00 0x7f30b0055fc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30d00b5ef0 0x7f30d00ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 --2- 172.21.15.26:0/3791200404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30d00a7fc0 0x7f30d015a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 >> 172.21.15.26:0/3791200404 conn(0x7f30d0060310 msgr2=0x7f30d0148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:10.176 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 shutdown_connections 2022-07-20T02:35:10.177 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.155+0000 7f30d4bd0700 1 -- 172.21.15.26:0/3791200404 wait complete. 2022-07-20T02:35:10.177 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 75 2022-07-20T02:35:10.179 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:35:10.180 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:35:10.181 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:35:10.181 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:10.181 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:35:10.182 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:35:10.182 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:35:10.182 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:10.183 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:35:10.183 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:35:10.183 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:35:10.184 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:10.198 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:35:10.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:35:10.200 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:35:10.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:35:10.221 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:35:10.222 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:10.223 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:35:10.223 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:35:10.224 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:35:10.225 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:10.238 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:35:10.239 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:10.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:35:10.270 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:35:10.294 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:35:10.306 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:35:10.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.347+0000 7febe625a700 1 Processor -- start 2022-07-20T02:35:10.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.347+0000 7febe625a700 1 -- start start 2022-07-20T02:35:10.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7febe00b3260 0x7febe0060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7febe0095b10 0x7febe0095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0067dc0 con 0x7febe00a95a0 2022-07-20T02:35:10.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe00b3640 con 0x7febe0095b10 2022-07-20T02:35:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe00615c0 con 0x7febe00b3260 2022-07-20T02:35:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7febe0095b10 0x7febe0095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe4a57700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7febe00a95a0 0x7febe00676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42828/0 (socket says 172.21.15.26:42828) 2022-07-20T02:35:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe4a57700 1 -- 172.21.15.26:0/267548007 learned_addr learned my addr 172.21.15.26:0/267548007 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7febe0095b10 0x7febe0095f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43552/0 (socket says 172.21.15.26:43552) 2022-07-20T02:35:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5258700 1 --2- 172.21.15.26:0/267548007 >> v2:172.21.15.120:3300/0 conn(0x7febe00b3260 0x7febe0060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 -- 172.21.15.26:0/267548007 >> v2:172.21.15.120:3300/0 conn(0x7febe00b3260 msgr2=0x7febe0060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 --2- 172.21.15.26:0/267548007 >> v2:172.21.15.120:3300/0 conn(0x7febe00b3260 0x7febe0060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febcffff700 1 -- 172.21.15.26:0/267548007 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853348443 0 0) 0x7febe0067dc0 con 0x7febe00a95a0 2022-07-20T02:35:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 -- 172.21.15.26:0/267548007 >> v1:172.21.15.26:6789/0 conn(0x7febe00a95a0 legacy=0x7febe00676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 -- 172.21.15.26:0/267548007 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0061860 con 0x7febe0095b10 2022-07-20T02:35:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe5a59700 1 --2- 172.21.15.26:0/267548007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe0095f50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7febd80032f0 tx=0x7febd800b470).ready entity=mon.0 client_cookie=7356e410cdd45e78 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febcffff700 1 -- 172.21.15.26:0/267548007 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800dd40 con 0x7febe0095b10 2022-07-20T02:35:10.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febcffff700 1 -- 172.21.15.26:0/267548007 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8010470 con 0x7febe0095b10 2022-07-20T02:35:10.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febcffff700 1 -- 172.21.15.26:0/267548007 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800dea0 con 0x7febe0095b10 2022-07-20T02:35:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- 172.21.15.26:0/267548007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 msgr2=0x7febe0095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 --2- 172.21.15.26:0/267548007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe0095f50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7febd80032f0 tx=0x7febd800b470).stop 2022-07-20T02:35:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- 172.21.15.26:0/267548007 shutdown_connections 2022-07-20T02:35:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 --2- 172.21.15.26:0/267548007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe0095f50 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 --2- 172.21.15.26:0/267548007 >> v2:172.21.15.120:3300/0 conn(0x7febe00b3260 0x7febe0060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- 172.21.15.26:0/267548007 >> 172.21.15.26:0/267548007 conn(0x7febe00b67f0 msgr2=0x7febe00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:10.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- 172.21.15.26:0/267548007 shutdown_connections 2022-07-20T02:35:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.351+0000 7febe625a700 1 -- 172.21.15.26:0/267548007 wait complete. 2022-07-20T02:35:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 Processor -- start 2022-07-20T02:35:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- start start 2022-07-20T02:35:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 0x7febe00aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe00b3260 0x7febe0152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01522c0 con 0x7febe0095b10 2022-07-20T02:35:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00ab420 con 0x7febe00a95a0 2022-07-20T02:35:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe0066c60 con 0x7febe00b3260 2022-07-20T02:35:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43556/0 (socket says 172.21.15.26:43556) 2022-07-20T02:35:10.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 -- 172.21.15.26:0/545133137 learned_addr learned my addr 172.21.15.26:0/545133137 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:10.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5a59700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe00b3260 0x7febe0152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe4a57700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 0x7febe00aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 -- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe00b3260 msgr2=0x7febe0152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe00b3260 0x7febe0152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 -- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 msgr2=0x7febe00aa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:10.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 0x7febe00aa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:10.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 -- 172.21.15.26:0/545133137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd800b710 con 0x7febe0095b10 2022-07-20T02:35:10.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe4a57700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 0x7febe00aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:10.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe5258700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7febe00ab560 tx=0x7febdc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:10.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc010780 con 0x7febe0095b10 2022-07-20T02:35:10.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc0108e0 con 0x7febe0095b10 2022-07-20T02:35:10.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe01524b0 con 0x7febe0095b10 2022-07-20T02:35:10.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe00ad1e0 con 0x7febe0095b10 2022-07-20T02:35:10.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.355+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc010c40 con 0x7febe0095b10 2022-07-20T02:35:10.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe0152950 con 0x7febe0095b10 2022-07-20T02:35:10.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7febdc01c460 con 0x7febe0095b10 2022-07-20T02:35:10.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febce7fc700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 0x7febc0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:10.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5686+0+0 (secure 0 0 0) 0x7febdc06c2a0 con 0x7febe0095b10 2022-07-20T02:35:10.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febe4a57700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 0x7febc0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:10.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.359+0000 7febe4a57700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 0x7febc0055e10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7febd000b000 tx=0x7febd0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:10.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.367+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febe0152950 con 0x7febe0095b10 2022-07-20T02:35:10.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:10.503+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7febe0152950 con 0x7febe0095b10 2022-07-20T02:35:11.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.315+0000 7febce7fc700 1 -- 172.21.15.26:0/545133137 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v146) v1 ==== 97+0+0 (secure 0 0 0) 0x7febe0152950 con 0x7febe0095b10 2022-07-20T02:35:11.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.319+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 msgr2=0x7febc0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.319+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 0x7febc0055e10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7febd000b000 tx=0x7febd0009040).stop 2022-07-20T02:35:11.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 msgr2=0x7febe00947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7febe00ab560 tx=0x7febdc006df0).stop 2022-07-20T02:35:11.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 shutdown_connections 2022-07-20T02:35:11.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7febc0053950 0x7febc0055e10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febe00b3260 0x7febe0152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe00a95a0 0x7febe00aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 --2- 172.21.15.26:0/545133137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7febe0095b10 0x7febe00947e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 >> 172.21.15.26:0/545133137 conn(0x7febe00b67f0 msgr2=0x7febe00b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:11.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 shutdown_connections 2022-07-20T02:35:11.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.323+0000 7febe625a700 1 -- 172.21.15.26:0/545133137 wait complete. 2022-07-20T02:35:11.337 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:35:11.347 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:35:11.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:35:11.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 Processor -- start 2022-07-20T02:35:11.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- start start 2022-07-20T02:35:11.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f61b0143090 0x7f61b0098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f61b00614b0 0x7f61b005c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b0143470 con 0x7f61b00b0810 2022-07-20T02:35:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b005fd90 con 0x7f61b00614b0 2022-07-20T02:35:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b0097d50 con 0x7f61b0143090 2022-07-20T02:35:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f61b00614b0 0x7f61b005c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f61b00614b0 0x7f61b005c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43568/0 (socket says 172.21.15.26:43568) 2022-07-20T02:35:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 -- 172.21.15.26:0/2770341005 learned_addr learned my addr 172.21.15.26:0/2770341005 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b5b6e700 1 --2- 172.21.15.26:0/2770341005 >> v2:172.21.15.120:3300/0 conn(0x7f61b0143090 0x7f61b0098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 -- 172.21.15.26:0/2770341005 >> v2:172.21.15.120:3300/0 conn(0x7f61b0143090 msgr2=0x7f61b0098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 --2- 172.21.15.26:0/2770341005 >> v2:172.21.15.120:3300/0 conn(0x7f61b0143090 0x7f61b0098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 -- 172.21.15.26:0/2770341005 >> v1:172.21.15.120:6789/0 conn(0x7f61b00b0810 legacy=0x7f61b00950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 -- 172.21.15.26:0/2770341005 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0097ff0 con 0x7f61b00614b0 2022-07-20T02:35:11.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b536d700 1 --2- 172.21.15.26:0/2770341005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b00614b0 0x7f61b005c580 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f61ac0032f0 tx=0x7f61ac009290).ready entity=mon.0 client_cookie=2b3c18afc001a0e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:11.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b4b6c700 1 -- 172.21.15.26:0/2770341005 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ac00b870 con 0x7f61b00614b0 2022-07-20T02:35:11.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b4b6c700 1 -- 172.21.15.26:0/2770341005 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac00b9d0 con 0x7f61b00614b0 2022-07-20T02:35:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b4b6c700 1 -- 172.21.15.26:0/2770341005 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ac00bb40 con 0x7f61b00614b0 2022-07-20T02:35:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2770341005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b00614b0 msgr2=0x7f61b005c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2770341005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b00614b0 0x7f61b005c580 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f61ac0032f0 tx=0x7f61ac009290).stop 2022-07-20T02:35:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2770341005 shutdown_connections 2022-07-20T02:35:11.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2770341005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b00614b0 0x7f61b005c580 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2770341005 >> v2:172.21.15.120:3300/0 conn(0x7f61b0143090 0x7f61b0098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2770341005 >> 172.21.15.26:0/2770341005 conn(0x7f61b00b5e70 msgr2=0x7f61b00b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:11.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2770341005 shutdown_connections 2022-07-20T02:35:11.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.511+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2770341005 wait complete. 2022-07-20T02:35:11.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 Processor -- start 2022-07-20T02:35:11.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 -- start start 2022-07-20T02:35:11.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b00614b0 0x7f61b01502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00b0810 0x7f61b00adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b0064580 con 0x7f61b0143090 2022-07-20T02:35:11.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00ad950 con 0x7f61b00b0810 2022-07-20T02:35:11.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b7dd2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00a99a0 con 0x7f61b00614b0 2022-07-20T02:35:11.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43576/0 (socket says 172.21.15.26:43576) 2022-07-20T02:35:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 -- 172.21.15.26:0/2875603716 learned_addr learned my addr 172.21.15.26:0/2875603716 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b5b6e700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b00614b0 0x7f61b01502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b536d700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00b0810 0x7f61b00adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 -- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b00614b0 msgr2=0x7f61b01502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b00614b0 0x7f61b01502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 -- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00b0810 msgr2=0x7f61b00adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00b0810 0x7f61b00adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:11.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 -- 172.21.15.26:0/2875603716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac009530 con 0x7f61b0143090 2022-07-20T02:35:11.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61b636f700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f61b0068970 tx=0x7f61a8007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a8012400 con 0x7f61b0143090 2022-07-20T02:35:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.515+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8012560 con 0x7f61b0143090 2022-07-20T02:35:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b009bf00 con 0x7f61b0143090 2022-07-20T02:35:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b00ae1c0 con 0x7f61b0143090 2022-07-20T02:35:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a80128c0 con 0x7f61b0143090 2022-07-20T02:35:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c003410 con 0x7f61b0143090 2022-07-20T02:35:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61a8019030 con 0x7f61b0143090 2022-07-20T02:35:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.519+0000 7f61a6ffd700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 0x7f6198055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.523+0000 7f61b5b6e700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 0x7f6198055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.523+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f61a806afe0 con 0x7f61b0143090 2022-07-20T02:35:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.523+0000 7f61b5b6e700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 0x7f6198055f30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f61a000b000 tx=0x7f61a0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.527+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61a8042060 con 0x7f61b0143090 2022-07-20T02:35:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.695+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f618c0036c0 con 0x7f61b0143090 2022-07-20T02:35:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.699+0000 7f61a6ffd700 1 -- 172.21.15.26:0/2875603716 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f61a8045680 con 0x7f61b0143090 2022-07-20T02:35:11.755 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:11.756 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:35:11.756 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:35:11.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.757 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:35:11.816 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:35:11.817 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.820 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:35:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:35:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.881 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:35:11.881 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:35:11.882 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:11.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.890 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:35:11.890 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:35:11.890 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.897 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:35:11.899 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:35:11.899 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.902 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:35:11.902 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:35:11.903 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.903 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.905 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:35:11.905 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:35:11.906 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.907 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.908 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:35:11.908 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:35:11.909 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.913 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:35:11.913 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:35:11.914 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:35:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:35:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:35:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:35:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:35:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:35:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:35:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:35:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:35:11.946 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:35:11.946 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:11.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:35:11.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:35:11.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.950 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:35:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:35:11.968 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:11.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.970 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:35:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:35:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:11.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:35:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:11.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:35:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:35:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:11.973 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:11.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:35:11.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:11.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:11.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:11.987 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:11.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 msgr2=0x7f6198055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 0x7f6198055f30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f61a000b000 tx=0x7f61a0006df0).stop 2022-07-20T02:35:11.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 msgr2=0x7f61b0064930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:11.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f61b0068970 tx=0x7f61a8007bc0).stop 2022-07-20T02:35:12.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 shutdown_connections 2022-07-20T02:35:12.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61b0143090 0x7f61b0064930 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6198053a70 0x7f6198055f30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00b0810 0x7f61b00adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 --2- 172.21.15.26:0/2875603716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b00614b0 0x7f61b01502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 >> 172.21.15.26:0/2875603716 conn(0x7f61b00b5e70 msgr2=0x7f61b00b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 shutdown_connections 2022-07-20T02:35:12.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:11.703+0000 7f61b7dd2700 1 -- 172.21.15.26:0/2875603716 wait complete. 2022-07-20T02:35:12.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 Processor -- start 2022-07-20T02:35:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 -- start start 2022-07-20T02:35:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb4340b5d50 0x7fb434067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 0x7fb4341434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb434146750 con 0x7fb434140320 2022-07-20T02:35:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4341469f0 con 0x7fb4340b5d50 2022-07-20T02:35:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43af0e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb434146c90 con 0x7fb434143070 2022-07-20T02:35:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43a70d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb434140320 0x7fb434140790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42862/0 (socket says 172.21.15.26:42862) 2022-07-20T02:35:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43a70d700 1 -- 172.21.15.26:0/781584139 learned_addr learned my addr 172.21.15.26:0/781584139 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb439f0c700 1 --2- 172.21.15.26:0/781584139 >> v2:172.21.15.26:3300/0 conn(0x7fb4340b5d50 0x7fb434067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb43970b700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 0x7fb4341434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769147419 0 0) 0x7fb434146750 con 0x7fb434140320 2022-07-20T02:35:12.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.119+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb41c003260 con 0x7fb434140320 2022-07-20T02:35:12.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb439f0c700 1 -- 172.21.15.26:0/781584139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 msgr2=0x7fb4341434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:12.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb439f0c700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 0x7fb4341434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb439f0c700 1 -- 172.21.15.26:0/781584139 >> v1:172.21.15.26:6789/0 conn(0x7fb434140320 legacy=0x7fb434140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2282439963 0 0) 0x7fb41c003260 con 0x7fb434140320 2022-07-20T02:35:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb439f0c700 1 -- 172.21.15.26:0/781584139 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434146f30 con 0x7fb4340b5d50 2022-07-20T02:35:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43970b700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 0x7fb4341434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb439f0c700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4340b5d50 0x7fb434067490 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fb42c001b80 tx=0x7fb424006df0).ready entity=mon.0 client_cookie=5c03ff7724e36407 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c0019e0 con 0x7fb4340b5d50 2022-07-20T02:35:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb424000e90 con 0x7fb4340b5d50 2022-07-20T02:35:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb438f0a700 1 -- 172.21.15.26:0/781584139 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb424012460 con 0x7fb4340b5d50 2022-07-20T02:35:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 -- 172.21.15.26:0/781584139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4340b5d50 msgr2=0x7fb434067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4340b5d50 0x7fb434067490 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fb42c001b80 tx=0x7fb424006df0).stop 2022-07-20T02:35:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 -- 172.21.15.26:0/781584139 shutdown_connections 2022-07-20T02:35:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb434143070 0x7fb4341434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 --2- 172.21.15.26:0/781584139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4340b5d50 0x7fb434067490 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 -- 172.21.15.26:0/781584139 >> 172.21.15.26:0/781584139 conn(0x7fb434060310 msgr2=0x7fb4340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 -- 172.21.15.26:0/781584139 shutdown_connections 2022-07-20T02:35:12.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.123+0000 7fb43af0e700 1 -- 172.21.15.26:0/781584139 wait complete. 2022-07-20T02:35:12.254 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 Processor -- start 2022-07-20T02:35:12.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- start start 2022-07-20T02:35:12.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4340b5d50 0x7fb4340a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb434140320 0x7fb434155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341565a0 con 0x7fb434143070 2022-07-20T02:35:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340ab0b0 con 0x7fb434140320 2022-07-20T02:35:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340b3a60 con 0x7fb4340b5d50 2022-07-20T02:35:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43590/0 (socket says 172.21.15.26:43590) 2022-07-20T02:35:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 -- 172.21.15.26:0/1846758825 learned_addr learned my addr 172.21.15.26:0/1846758825 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43970b700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb434140320 0x7fb434155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb439f0c700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4340b5d50 0x7fb4340a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 -- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4340b5d50 msgr2=0x7fb4340a95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4340b5d50 0x7fb4340a95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 -- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb434140320 msgr2=0x7fb434155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb434140320 0x7fb434155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 -- 172.21.15.26:0/1846758825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43000b180 con 0x7fb434143070 2022-07-20T02:35:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43a70d700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fb4341566e0 tx=0x7fb42c0015b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c004ab0 con 0x7fb434143070 2022-07-20T02:35:12.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c004c10 con 0x7fb434143070 2022-07-20T02:35:12.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42400b180 con 0x7fb434143070 2022-07-20T02:35:12.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c007170 con 0x7fb434143070 2022-07-20T02:35:12.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.127+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4340ab580 con 0x7fb434143070 2022-07-20T02:35:12.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4340ab740 con 0x7fb434143070 2022-07-20T02:35:12.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb42c0072d0 con 0x7fb434143070 2022-07-20T02:35:12.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb422ffd700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 0x7fb414055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb42c06bb60 con 0x7fb434143070 2022-07-20T02:35:12.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb439f0c700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 0x7fb414055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.131+0000 7fb439f0c700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 0x7fb414055f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fb42401a000 tx=0x7fb424014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.135+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb42c042b50 con 0x7fb434143070 2022-07-20T02:35:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.259+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4341439b0 con 0x7fb434143070 2022-07-20T02:35:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.259+0000 7fb422ffd700 1 -- 172.21.15.26:0/1846758825 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+13472 (secure 0 0 0) 0x7fb42c046170 con 0x7fb434143070 2022-07-20T02:35:12.300 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:12.301 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":146,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:11.311917+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:12.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 msgr2=0x7fb414055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 0x7fb414055f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fb42401a000 tx=0x7fb424014000).stop 2022-07-20T02:35:12.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 msgr2=0x7fb4341560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fb4341566e0 tx=0x7fb42c0015b0).stop 2022-07-20T02:35:12.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 shutdown_connections 2022-07-20T02:35:12.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb434143070 0x7fb4341560b0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb414053a70 0x7fb414055f30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb434140320 0x7fb434155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 --2- 172.21.15.26:0/1846758825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4340b5d50 0x7fb4340a95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 >> 172.21.15.26:0/1846758825 conn(0x7fb434060310 msgr2=0x7fb43405ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 shutdown_connections 2022-07-20T02:35:12.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.263+0000 7fb43af0e700 1 -- 172.21.15.26:0/1846758825 wait complete. 2022-07-20T02:35:12.312 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:35:12.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:12.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.431+0000 7f610251e700 1 Processor -- start 2022-07-20T02:35:12.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.431+0000 7f610251e700 1 -- start start 2022-07-20T02:35:12.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60fc05cfd0 0x7f60fc1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 0x7f60fc140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60fc05d3b0 con 0x7f60fc146e80 2022-07-20T02:35:12.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60fc1435b0 con 0x7f60fc05cfd0 2022-07-20T02:35:12.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60fc0b4bc0 con 0x7f60fc0b1160 2022-07-20T02:35:12.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60fc05cfd0 0x7f60fc1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f60fc05cfd0 0x7f60fc1407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43594/0 (socket says 172.21.15.26:43594) 2022-07-20T02:35:12.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f6100abb700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60fc146e80 0x7f60fc147290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42878/0 (socket says 172.21.15.26:42878) 2022-07-20T02:35:12.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 -- 172.21.15.26:0/782942430 learned_addr learned my addr 172.21.15.26:0/782942430 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:12.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fb7fe700 1 --2- 172.21.15.26:0/782942430 >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 0x7f60fc140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60faffd700 1 -- 172.21.15.26:0/782942430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574286936 0 0) 0x7f60fc05d3b0 con 0x7f60fc146e80 2022-07-20T02:35:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60faffd700 1 -- 172.21.15.26:0/782942430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e8003260 con 0x7f60fc146e80 2022-07-20T02:35:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 -- 172.21.15.26:0/782942430 >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 msgr2=0x7f60fc140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:12.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 --2- 172.21.15.26:0/782942430 >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 0x7f60fc140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 -- 172.21.15.26:0/782942430 >> v1:172.21.15.26:6789/0 conn(0x7f60fc146e80 legacy=0x7f60fc147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 -- 172.21.15.26:0/782942430 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc0b4e60 con 0x7f60fc05cfd0 2022-07-20T02:35:12.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fb7fe700 1 --2- 172.21.15.26:0/782942430 >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 0x7f60fc140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:12.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60fbfff700 1 --2- 172.21.15.26:0/782942430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc05cfd0 0x7f60fc1407c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f60f0002f70 tx=0x7f60f0006eb0).ready entity=mon.0 client_cookie=6c6414220b3cef41 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60faffd700 1 -- 172.21.15.26:0/782942430 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f000d610 con 0x7f60fc05cfd0 2022-07-20T02:35:12.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60faffd700 1 -- 172.21.15.26:0/782942430 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f0000e90 con 0x7f60fc05cfd0 2022-07-20T02:35:12.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f60faffd700 1 -- 172.21.15.26:0/782942430 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f000d9a0 con 0x7f60fc05cfd0 2022-07-20T02:35:12.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- 172.21.15.26:0/782942430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc05cfd0 msgr2=0x7f60fc1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 --2- 172.21.15.26:0/782942430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc05cfd0 0x7f60fc1407c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f60f0002f70 tx=0x7f60f0006eb0).stop 2022-07-20T02:35:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- 172.21.15.26:0/782942430 shutdown_connections 2022-07-20T02:35:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 --2- 172.21.15.26:0/782942430 >> v2:172.21.15.120:3300/0 conn(0x7f60fc0b1160 0x7f60fc140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 --2- 172.21.15.26:0/782942430 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc05cfd0 0x7f60fc1407c0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- 172.21.15.26:0/782942430 >> 172.21.15.26:0/782942430 conn(0x7f60fc066f70 msgr2=0x7f60fc05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- 172.21.15.26:0/782942430 shutdown_connections 2022-07-20T02:35:12.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.435+0000 7f610251e700 1 -- 172.21.15.26:0/782942430 wait complete. 2022-07-20T02:35:12.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 Processor -- start 2022-07-20T02:35:12.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- start start 2022-07-20T02:35:12.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 0x7f60fc13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc146e80 0x7f60fc156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0b9260 con 0x7f60fc0b1160 2022-07-20T02:35:12.528 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc155f20 con 0x7f60fc05cfd0 2022-07-20T02:35:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1463f0 con 0x7f60fc146e80 2022-07-20T02:35:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43602/0 (socket says 172.21.15.26:43602) 2022-07-20T02:35:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 -- 172.21.15.26:0/3750005561 learned_addr learned my addr 172.21.15.26:0/3750005561 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fbfff700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 0x7f60fc13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f6100abb700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc146e80 0x7f60fc156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 -- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc146e80 msgr2=0x7f60fc156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc146e80 0x7f60fc156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 -- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 msgr2=0x7f60fc13cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 0x7f60fc13cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 -- 172.21.15.26:0/3750005561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec00b180 con 0x7f60fc0b1160 2022-07-20T02:35:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fbfff700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 0x7f60fc13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60fb7fe700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f60fc0ad950 tx=0x7f60ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ec00ca40 con 0x7f60fc0b1160 2022-07-20T02:35:12.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec00cba0 con 0x7f60fc0b1160 2022-07-20T02:35:12.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f000b1e0 con 0x7f60fc0b1160 2022-07-20T02:35:12.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ec0123f0 con 0x7f60fc0b1160 2022-07-20T02:35:12.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.439+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc146150 con 0x7f60fc0b1160 2022-07-20T02:35:12.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc145620 con 0x7f60fc0b1160 2022-07-20T02:35:12.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60ec012550 con 0x7f60fc0b1160 2022-07-20T02:35:12.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f60f97fa700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 0x7f60dc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f60ec06c350 con 0x7f60fc0b1160 2022-07-20T02:35:12.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f60fbfff700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 0x7f60dc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.443+0000 7f60fbfff700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 0x7f60dc055ea0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f60f001a000 tx=0x7f60f0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.447+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60ec0433f0 con 0x7f60fc0b1160 2022-07-20T02:35:12.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.575+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f60fc0574e0 con 0x7f60fc0b1160 2022-07-20T02:35:12.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.575+0000 7f60f97fa700 1 -- 172.21.15.26:0/3750005561 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v146) v1 ==== 110+0+10 (secure 0 0 0) 0x7f60ec046a10 con 0x7f60fc0b1160 2022-07-20T02:35:12.603 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:12.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 msgr2=0x7f60dc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 0x7f60dc055ea0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f60f001a000 tx=0x7f60f0015000).stop 2022-07-20T02:35:12.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 msgr2=0x7f60fc152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f60fc0ad950 tx=0x7f60ec006df0).stop 2022-07-20T02:35:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 shutdown_connections 2022-07-20T02:35:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60fc0b1160 0x7f60fc152930 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f60dc0539e0 0x7f60dc055ea0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc146e80 0x7f60fc156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 --2- 172.21.15.26:0/3750005561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc05cfd0 0x7f60fc13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 >> 172.21.15.26:0/3750005561 conn(0x7f60fc066f70 msgr2=0x7f60fc0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 shutdown_connections 2022-07-20T02:35:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.579+0000 7f610251e700 1 -- 172.21.15.26:0/3750005561 wait complete. 2022-07-20T02:35:12.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 Processor -- start 2022-07-20T02:35:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 -- start start 2022-07-20T02:35:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e9413f660 0x7f4e9413faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4e940a9ea0 0x7f4e940aa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e940a8cc0 con 0x7f4e940ac940 2022-07-20T02:35:12.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e940aba20 con 0x7f4e940a9ea0 2022-07-20T02:35:12.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.743+0000 7f4e9a1e2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e940ab4e0 con 0x7f4e9413f660 2022-07-20T02:35:12.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4e940a9ea0 0x7f4e940aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e991e0700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4e940ac940 0x7f4e940acd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42888/0 (socket says 172.21.15.26:42888) 2022-07-20T02:35:12.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e991e0700 1 -- 172.21.15.26:0/3626929107 learned_addr learned my addr 172.21.15.26:0/3626929107 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:12.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e989df700 1 --2- 172.21.15.26:0/3626929107 >> v2:172.21.15.120:3300/0 conn(0x7f4e9413f660 0x7f4e9413faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 -- 172.21.15.26:0/3626929107 >> v2:172.21.15.120:3300/0 conn(0x7f4e9413f660 msgr2=0x7f4e9413faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:12.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 --2- 172.21.15.26:0/3626929107 >> v2:172.21.15.120:3300/0 conn(0x7f4e9413f660 0x7f4e9413faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 -- 172.21.15.26:0/3626929107 >> v1:172.21.15.26:6789/0 conn(0x7f4e940ac940 legacy=0x7f4e940acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e83fff700 1 -- 172.21.15.26:0/3626929107 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258371175 0 0) 0x7f4e940a8cc0 con 0x7f4e940ac940 2022-07-20T02:35:12.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 -- 172.21.15.26:0/3626929107 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9405ce10 con 0x7f4e940a9ea0 2022-07-20T02:35:12.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e999e1700 1 --2- 172.21.15.26:0/3626929107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e940aa330 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f4e8c0032f0 tx=0x7f4e8c00a370).ready entity=mon.0 client_cookie=1269f1f1897608 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e83fff700 1 -- 172.21.15.26:0/3626929107 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e8c00cb70 con 0x7f4e940a9ea0 2022-07-20T02:35:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e83fff700 1 -- 172.21.15.26:0/3626929107 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c00ccd0 con 0x7f4e940a9ea0 2022-07-20T02:35:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e83fff700 1 -- 172.21.15.26:0/3626929107 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e8c010600 con 0x7f4e940a9ea0 2022-07-20T02:35:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/3626929107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 msgr2=0x7f4e940aa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/3626929107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e940aa330 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f4e8c0032f0 tx=0x7f4e8c00a370).stop 2022-07-20T02:35:12.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/3626929107 shutdown_connections 2022-07-20T02:35:12.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/3626929107 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e940aa330 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/3626929107 >> v2:172.21.15.120:3300/0 conn(0x7f4e9413f660 0x7f4e9413faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/3626929107 >> 172.21.15.26:0/3626929107 conn(0x7f4e94060310 msgr2=0x7f4e940b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/3626929107 shutdown_connections 2022-07-20T02:35:12.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.747+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/3626929107 wait complete. 2022-07-20T02:35:12.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 Processor -- start 2022-07-20T02:35:12.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- start start 2022-07-20T02:35:12.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e940ac940 0x7f4e94064f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9413f660 0x7f4e940ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e94137680 con 0x7f4e940a9ea0 2022-07-20T02:35:12.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940b0730 con 0x7f4e940ac940 2022-07-20T02:35:12.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940630b0 con 0x7f4e9413f660 2022-07-20T02:35:12.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43614/0 (socket says 172.21.15.26:43614) 2022-07-20T02:35:12.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 -- 172.21.15.26:0/2542710600 learned_addr learned my addr 172.21.15.26:0/2542710600 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:12.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e989df700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e940ac940 0x7f4e94064f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e999e1700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9413f660 0x7f4e940ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 -- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9413f660 msgr2=0x7f4e940ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9413f660 0x7f4e940ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 -- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e940ac940 msgr2=0x7f4e94064f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e940ac940 0x7f4e94064f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 -- 172.21.15.26:0/2542710600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c00a610 con 0x7f4e940a9ea0 2022-07-20T02:35:12.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e991e0700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f4e940b05f0 tx=0x7f4e90007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e900070d0 con 0x7f4e940a9ea0 2022-07-20T02:35:12.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e90007230 con 0x7f4e940a9ea0 2022-07-20T02:35:12.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e940b0960 con 0x7f4e940a9ea0 2022-07-20T02:35:12.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e90005610 con 0x7f4e940a9ea0 2022-07-20T02:35:12.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.751+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e940af260 con 0x7f4e940a9ea0 2022-07-20T02:35:12.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e9001a060 con 0x7f4e940a9ea0 2022-07-20T02:35:12.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e94063270 con 0x7f4e940a9ea0 2022-07-20T02:35:12.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e827fc700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 0x7f4e74055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:12.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f4e9006b1f0 con 0x7f4e940a9ea0 2022-07-20T02:35:12.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e989df700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 0x7f4e74055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:12.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.755+0000 7f4e989df700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 0x7f4e74055e90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f4e8400b000 tx=0x7f4e84006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:12.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.759+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e900421e0 con 0x7f4e940a9ea0 2022-07-20T02:35:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4e940ab240 con 0x7f4e940a9ea0 2022-07-20T02:35:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e827fc700 1 -- 172.21.15.26:0/2542710600 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v146) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e90018020 con 0x7f4e940a9ea0 2022-07-20T02:35:12.909 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 msgr2=0x7f4e74055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 0x7f4e74055e90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f4e8400b000 tx=0x7f4e84006df0).stop 2022-07-20T02:35:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 msgr2=0x7f4e94063530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f4e940b05f0 tx=0x7f4e90007af0).stop 2022-07-20T02:35:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 shutdown_connections 2022-07-20T02:35:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4e740539d0 0x7f4e74055e90 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e9413f660 0x7f4e940ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e940ac940 0x7f4e94064f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 --2- 172.21.15.26:0/2542710600 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e940a9ea0 0x7f4e94063530 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 >> 172.21.15.26:0/2542710600 conn(0x7f4e94060310 msgr2=0x7f4e9413b930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 shutdown_connections 2022-07-20T02:35:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:12.887+0000 7f4e9a1e2700 1 -- 172.21.15.26:0/2542710600 wait complete. 2022-07-20T02:35:12.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:35:13.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 Processor -- start 2022-07-20T02:35:13.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 -- start start 2022-07-20T02:35:13.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 0x7fe5fc14a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5fc14b470 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc064650 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc0b41a0 con 0x7fe5fc149ce0 2022-07-20T02:35:13.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43624/0 (socket says 172.21.15.26:43624) 2022-07-20T02:35:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/2469293810 learned_addr learned my addr 172.21.15.26:0/2469293810 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:13.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fad9d700 1 --2- 172.21.15.26:0/2469293810 >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 0x7fe5fc14a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/2469293810 >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 msgr2=0x7fe5fc14a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 --2- 172.21.15.26:0/2469293810 >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 0x7fe5fc14a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/2469293810 >> v1:172.21.15.120:6789/0 conn(0x7fe5fc0b7d80 legacy=0x7fe5fc14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/2469293810 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0b4440 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fad9d700 1 --2- 172.21.15.26:0/2469293810 >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 0x7fe5fc14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:35:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f9d9b700 1 --2- 172.21.15.26:0/2469293810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0006eb0).ready entity=mon.0 client_cookie=21c0c0df646a837 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f959a700 1 -- 172.21.15.26:0/2469293810 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f000d610 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f959a700 1 -- 172.21.15.26:0/2469293810 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0000e90 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5f959a700 1 -- 172.21.15.26:0/2469293810 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f000d9a0 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 -- 172.21.15.26:0/2469293810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0aaf00 msgr2=0x7fe5fc0b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.031+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/2469293810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0006eb0).stop 2022-07-20T02:35:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- 172.21.15.26:0/2469293810 shutdown_connections 2022-07-20T02:35:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/2469293810 >> v2:172.21.15.120:3300/0 conn(0x7fe5fc149ce0 0x7fe5fc14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/2469293810 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b4b80 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- 172.21.15.26:0/2469293810 >> 172.21.15.26:0/2469293810 conn(0x7fe5fc05f840 msgr2=0x7fe5fc05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- 172.21.15.26:0/2469293810 shutdown_connections 2022-07-20T02:35:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- 172.21.15.26:0/2469293810 wait complete. 2022-07-20T02:35:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 Processor -- start 2022-07-20T02:35:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- start start 2022-07-20T02:35:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5fc149ce0 0x7fe5fc09f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0aeec0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0aec60 con 0x7fe5fc0aaf00 2022-07-20T02:35:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fb59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0b08e0 con 0x7fe5fc149ce0 2022-07-20T02:35:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43630/0 (socket says 172.21.15.26:43630) 2022-07-20T02:35:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/1946880716 learned_addr learned my addr 172.21.15.26:0/1946880716 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fad9d700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5fc149ce0 0x7fe5fc09f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5fa59c700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5fc149ce0 msgr2=0x7fe5fc09f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5fc149ce0 0x7fe5fc09f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5fc0aaf00 msgr2=0x7fe5fc0b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 -- 172.21.15.26:0/1946880716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f000b1e0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.035+0000 7fe5f9d9b700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f00118a0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0011a00 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc0b0b80 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f0011d60 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc0ae240 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc0b46e0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe5f001a040 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 0x7fe5d8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe5f006cdb0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.039+0000 7fe5fa59c700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 0x7fe5d8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.043+0000 7fe5fa59c700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 0x7fe5d8055f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fe5ec00d000 tx=0x7fe5ec007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.051+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5f0071070 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.179+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe5fc0574e0 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.179+0000 7fe5eb7fe700 1 -- 172.21.15.26:0/1946880716 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v146) v1 ==== 84+0+143 (secure 0 0 0) 0x7fe5f0023020 con 0x7fe5fc0b7d80 2022-07-20T02:35:13.189 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:13.190 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:35:13.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:13.190 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:35:13.190 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:35:13.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:13.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:13.191 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:35:13.191 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:35:13.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:13.192 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:35:13.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 msgr2=0x7fe5d8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 0x7fe5d8055f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fe5ec00d000 tx=0x7fe5ec007b90).stop 2022-07-20T02:35:13.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 msgr2=0x7fe5fc09ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0015000).stop 2022-07-20T02:35:13.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 shutdown_connections 2022-07-20T02:35:13.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe5fc0b7d80 0x7fe5fc09ee10 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe5d8053a70 0x7fe5d8055f30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5fc149ce0 0x7fe5fc09f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 --2- 172.21.15.26:0/1946880716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5fc0aaf00 0x7fe5fc0b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 >> 172.21.15.26:0/1946880716 conn(0x7fe5fc05f840 msgr2=0x7fe5fc0b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:13.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 shutdown_connections 2022-07-20T02:35:13.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.183+0000 7fe5fb59e700 1 -- 172.21.15.26:0/1946880716 wait complete. 2022-07-20T02:35:13.211 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:35:13.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:35:13.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.351+0000 7faa70401700 1 Processor -- start 2022-07-20T02:35:13.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.351+0000 7faa70401700 1 -- start start 2022-07-20T02:35:13.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.351+0000 7faa70401700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.351+0000 7faa70401700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa680b1160 0x7faa68140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa68146e80 0x7faa681472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa681435b0 con 0x7faa6805cfd0 2022-07-20T02:35:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6805d3b0 con 0x7faa68146e80 2022-07-20T02:35:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680b4bc0 con 0x7faa680b1160 2022-07-20T02:35:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e19d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa68146e80 0x7faa681472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6d99c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa680b1160 0x7faa68140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa68146e80 0x7faa681472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35098/0 (socket says 172.21.15.26:35098) 2022-07-20T02:35:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 -- 172.21.15.26:0/518667773 learned_addr learned my addr 172.21.15.26:0/518667773 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e19d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43636/0 (socket says 172.21.15.26:43636) 2022-07-20T02:35:13.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 -- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa680b1160 msgr2=0x7faa68140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:13.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 --2- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa680b1160 0x7faa68140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 -- 172.21.15.26:0/518667773 >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 msgr2=0x7faa681407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:13.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 --2- 172.21.15.26:0/518667773 >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 -- 172.21.15.26:0/518667773 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680b4e60 con 0x7faa68146e80 2022-07-20T02:35:13.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e19d700 1 --2- 172.21.15.26:0/518667773 >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:13.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6e99e700 1 --2- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa681472f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7faa600032f0 tx=0x7faa60009290).ready entity=mon.1 client_cookie=a38ef13becc5e2a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6d19b700 1 -- 172.21.15.26:0/518667773 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa6000b680 con 0x7faa68146e80 2022-07-20T02:35:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6d19b700 1 -- 172.21.15.26:0/518667773 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6000b7e0 con 0x7faa68146e80 2022-07-20T02:35:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa6d19b700 1 -- 172.21.15.26:0/518667773 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa6000b950 con 0x7faa68146e80 2022-07-20T02:35:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 msgr2=0x7faa681472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 --2- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa681472f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7faa600032f0 tx=0x7faa60009290).stop 2022-07-20T02:35:13.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- 172.21.15.26:0/518667773 shutdown_connections 2022-07-20T02:35:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 --2- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa681472f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 --2- 172.21.15.26:0/518667773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa680b1160 0x7faa68140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 --2- 172.21.15.26:0/518667773 >> v2:172.21.15.26:3300/0 conn(0x7faa6805cfd0 0x7faa681407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- 172.21.15.26:0/518667773 >> 172.21.15.26:0/518667773 conn(0x7faa68066f70 msgr2=0x7faa6805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:13.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- 172.21.15.26:0/518667773 shutdown_connections 2022-07-20T02:35:13.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.355+0000 7faa70401700 1 -- 172.21.15.26:0/518667773 wait complete. 2022-07-20T02:35:13.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 Processor -- start 2022-07-20T02:35:13.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- start start 2022-07-20T02:35:13.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa6805cfd0 0x7faa680b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa680aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68153670 con 0x7faa680b1160 2022-07-20T02:35:13.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680b77f0 con 0x7faa68146e80 2022-07-20T02:35:13.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680ad670 con 0x7faa6805cfd0 2022-07-20T02:35:13.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43642/0 (socket says 172.21.15.26:43642) 2022-07-20T02:35:13.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 -- 172.21.15.26:0/373215772 learned_addr learned my addr 172.21.15.26:0/373215772 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:13.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6e99e700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa680aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6e19d700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa6805cfd0 0x7faa680b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 -- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa6805cfd0 msgr2=0x7faa680b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa6805cfd0 0x7faa680b90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 -- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 msgr2=0x7faa680aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa680aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 -- 172.21.15.26:0/373215772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6400b180 con 0x7faa680b1160 2022-07-20T02:35:13.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa6d99c700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7faa5c00c000 tx=0x7faa5c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa5c011750 con 0x7faa680b1160 2022-07-20T02:35:13.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c0118b0 con 0x7faa680b1160 2022-07-20T02:35:13.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa60009530 con 0x7faa680b1160 2022-07-20T02:35:13.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa5c011c10 con 0x7faa680b1160 2022-07-20T02:35:13.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.359+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa680b7eb0 con 0x7faa680b1160 2022-07-20T02:35:13.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa680b7960 con 0x7faa680b1160 2022-07-20T02:35:13.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa5c011d70 con 0x7faa680b1160 2022-07-20T02:35:13.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa5b7fe700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 0x7faa48055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:13.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5600+0+0 (secure 0 0 0) 0x7faa5c06c170 con 0x7faa680b1160 2022-07-20T02:35:13.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa6e19d700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 0x7faa48055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.363+0000 7faa6e19d700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 0x7faa48055ed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7faa6400b000 tx=0x7faa64006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:13.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.367+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa5c043160 con 0x7faa680b1160 2022-07-20T02:35:13.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.491+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7faa680574e0 con 0x7faa680b1160 2022-07-20T02:35:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.707+0000 7faa5b7fe700 1 -- 172.21.15.26:0/373215772 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v147) v1 ==== 133+0+0 (secure 0 0 0) 0x7faa5c046780 con 0x7faa680b1160 2022-07-20T02:35:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 msgr2=0x7faa48055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 0x7faa48055ed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7faa6400b000 tx=0x7faa64006df0).stop 2022-07-20T02:35:13.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 msgr2=0x7faa68153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:13.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7faa5c00c000 tx=0x7faa5c006df0).stop 2022-07-20T02:35:13.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 shutdown_connections 2022-07-20T02:35:13.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa680b1160 0x7faa68153180 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faa48053a10 0x7faa48055ed0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa68146e80 0x7faa680aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 --2- 172.21.15.26:0/373215772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa6805cfd0 0x7faa680b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:13.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 >> 172.21.15.26:0/373215772 conn(0x7faa68066f70 msgr2=0x7faa68143830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:13.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 shutdown_connections 2022-07-20T02:35:13.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.711+0000 7faa70401700 1 -- 172.21.15.26:0/373215772 wait complete. 2022-07-20T02:35:13.760 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:35:13.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:35:14.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 Processor -- start 2022-07-20T02:35:14.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 -- start start 2022-07-20T02:35:14.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab900618a0 0x7fab90147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab900b22d0 0x7fab90143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab90061c80 con 0x7fab90143d60 2022-07-20T02:35:14.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab900b2810 con 0x7fab900618a0 2022-07-20T02:35:14.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab95d03700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab900b2ab0 con 0x7fab900b22d0 2022-07-20T02:35:14.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab900618a0 0x7fab90147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fab900618a0 0x7fab90147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43650/0 (socket says 172.21.15.26:43650) 2022-07-20T02:35:14.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 -- 172.21.15.26:0/2075327491 learned_addr learned my addr 172.21.15.26:0/2075327491 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:14.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab8ffff700 1 --2- 172.21.15.26:0/2075327491 >> v2:172.21.15.120:3300/0 conn(0x7fab900b22d0 0x7fab90143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 -- 172.21.15.26:0/2075327491 >> v2:172.21.15.120:3300/0 conn(0x7fab900b22d0 msgr2=0x7fab90143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:14.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 --2- 172.21.15.26:0/2075327491 >> v2:172.21.15.120:3300/0 conn(0x7fab900b22d0 0x7fab90143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 -- 172.21.15.26:0/2075327491 >> v1:172.21.15.120:6789/0 conn(0x7fab90143d60 legacy=0x7fab900b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 -- 172.21.15.26:0/2075327491 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab900b2d50 con 0x7fab900618a0 2022-07-20T02:35:14.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab94d01700 1 --2- 172.21.15.26:0/2075327491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab900618a0 0x7fab90147250 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fab78002f70 tx=0x7fab78006eb0).ready entity=mon.0 client_cookie=a45289acbc51c71b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:14.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab8f7fe700 1 -- 172.21.15.26:0/2075327491 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab7800d5c0 con 0x7fab900618a0 2022-07-20T02:35:14.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab8f7fe700 1 -- 172.21.15.26:0/2075327491 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78000e90 con 0x7fab900618a0 2022-07-20T02:35:14.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.927+0000 7fab8f7fe700 1 -- 172.21.15.26:0/2075327491 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab7800d950 con 0x7fab900618a0 2022-07-20T02:35:14.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- 172.21.15.26:0/2075327491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab900618a0 msgr2=0x7fab90147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- 172.21.15.26:0/2075327491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab900618a0 0x7fab90147250 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fab78002f70 tx=0x7fab78006eb0).stop 2022-07-20T02:35:14.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- 172.21.15.26:0/2075327491 shutdown_connections 2022-07-20T02:35:14.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- 172.21.15.26:0/2075327491 >> v2:172.21.15.120:3300/0 conn(0x7fab900b22d0 0x7fab90143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- 172.21.15.26:0/2075327491 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab900618a0 0x7fab90147250 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- 172.21.15.26:0/2075327491 >> 172.21.15.26:0/2075327491 conn(0x7fab90066f70 msgr2=0x7fab90067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:14.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- 172.21.15.26:0/2075327491 shutdown_connections 2022-07-20T02:35:14.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- 172.21.15.26:0/2075327491 wait complete. 2022-07-20T02:35:14.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 Processor -- start 2022-07-20T02:35:14.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- start start 2022-07-20T02:35:14.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 0x7fab900b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab900b22d0 0x7fab90146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9013d9d0 con 0x7fab90143d60 2022-07-20T02:35:14.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab90145580 con 0x7fab900b22d0 2022-07-20T02:35:14.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95d03700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab901457b0 con 0x7fab900618a0 2022-07-20T02:35:14.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95502700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95502700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43660/0 (socket says 172.21.15.26:43660) 2022-07-20T02:35:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab95502700 1 -- 172.21.15.26:0/350437312 learned_addr learned my addr 172.21.15.26:0/350437312 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab8ffff700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab900b22d0 0x7fab90146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.931+0000 7fab94d01700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 0x7fab900b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 -- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 msgr2=0x7fab900b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 0x7fab900b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 -- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab900b22d0 msgr2=0x7fab90146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab900b22d0 0x7fab90146860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 -- 172.21.15.26:0/350437312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7800b1e0 con 0x7fab90143d60 2022-07-20T02:35:14.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab94d01700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 0x7fab900b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:14.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95502700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fab900ad5e0 tx=0x7fab88007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:14.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab88006aa0 con 0x7fab90143d60 2022-07-20T02:35:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab88006c00 con 0x7fab90143d60 2022-07-20T02:35:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab90147240 con 0x7fab90143d60 2022-07-20T02:35:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab88007030 con 0x7fab90143d60 2022-07-20T02:35:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab90145150 con 0x7fab90143d60 2022-07-20T02:35:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9013d050 con 0x7fab90143d60 2022-07-20T02:35:14.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fab88010070 con 0x7fab90143d60 2022-07-20T02:35:14.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.935+0000 7fab8dffb700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 0x7fab74055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.939+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5954+0+0 (secure 0 0 0) 0x7fab8806baa0 con 0x7fab90143d60 2022-07-20T02:35:14.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.939+0000 7fab94d01700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 0x7fab74055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.939+0000 7fab94d01700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 0x7fab74055ea0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7fab7801a000 tx=0x7fab78015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:14.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:13.947+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab9013d050 con 0x7fab90143d60 2022-07-20T02:35:14.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.075+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fab9013d050 con 0x7fab90143d60 2022-07-20T02:35:14.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.327+0000 7fab8dffb700 1 -- 172.21.15.26:0/350437312 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v148) v1 ==== 125+0+0 (secure 0 0 0) 0x7fab9013d050 con 0x7fab90143d60 2022-07-20T02:35:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 msgr2=0x7fab74055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 0x7fab74055ea0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7fab7801a000 tx=0x7fab78015000).stop 2022-07-20T02:35:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 msgr2=0x7fab90146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fab900ad5e0 tx=0x7fab88007f70).stop 2022-07-20T02:35:14.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 shutdown_connections 2022-07-20T02:35:14.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab90143d60 0x7fab90146d50 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fab740539e0 0x7fab74055ea0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab900b22d0 0x7fab90146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 --2- 172.21.15.26:0/350437312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab900618a0 0x7fab900b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 >> 172.21.15.26:0/350437312 conn(0x7fab90066f70 msgr2=0x7fab9012c280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:14.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 shutdown_connections 2022-07-20T02:35:14.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.335+0000 7fab95d03700 1 -- 172.21.15.26:0/350437312 wait complete. 2022-07-20T02:35:14.367 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:35:14.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:35:14.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 Processor -- start 2022-07-20T02:35:14.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 -- start start 2022-07-20T02:35:14.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 0x7f9ea41432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ea412c530 0x7f9ea40b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea4061e00 con 0x7f9ea4061a20 2022-07-20T02:35:14.705 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea4147040 con 0x7f9ea412c530 2022-07-20T02:35:14.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9eab534700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea41472e0 con 0x7f9ea41468c0 2022-07-20T02:35:14.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.503+0000 7f9ea92d0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 0x7f9ea41432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea8acf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ea412c530 0x7f9ea40b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea92d0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 0x7f9ea41432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43664/0 (socket says 172.21.15.26:43664) 2022-07-20T02:35:14.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea8acf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:36950/0 (socket says 172.21.15.26:36950) 2022-07-20T02:35:14.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ea412c530 0x7f9ea40b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35128/0 (socket says 172.21.15.26:35128) 2022-07-20T02:35:14.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea92d0700 1 -- 172.21.15.26:0/4241720814 learned_addr learned my addr 172.21.15.26:0/4241720814 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:14.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 -- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 msgr2=0x7f9ea412c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:14.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 -- 172.21.15.26:0/4241720814 >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 msgr2=0x7f9ea41432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:14.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 --2- 172.21.15.26:0/4241720814 >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 0x7f9ea41432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 -- 172.21.15.26:0/4241720814 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea4147580 con 0x7f9ea412c530 2022-07-20T02:35:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea8acf700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9ea9ad1700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea412c530 0x7f9ea40b2720 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f9e9c0032f0 tx=0x7f9e9c0091d0).ready entity=mon.1 client_cookie=98b0acd98b87ab89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9e93fff700 1 -- 172.21.15.26:0/4241720814 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e9c00b600 con 0x7f9ea412c530 2022-07-20T02:35:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9e93fff700 1 -- 172.21.15.26:0/4241720814 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c00b760 con 0x7f9ea412c530 2022-07-20T02:35:14.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9e93fff700 1 -- 172.21.15.26:0/4241720814 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e9c00b8d0 con 0x7f9ea412c530 2022-07-20T02:35:14.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 -- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea412c530 msgr2=0x7f9ea40b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea412c530 0x7f9ea40b2720 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f9e9c0032f0 tx=0x7f9e9c0091d0).stop 2022-07-20T02:35:14.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 -- 172.21.15.26:0/4241720814 shutdown_connections 2022-07-20T02:35:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea412c530 0x7f9ea40b2720 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 --2- 172.21.15.26:0/4241720814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea41468c0 0x7f9ea412c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 --2- 172.21.15.26:0/4241720814 >> v2:172.21.15.26:3300/0 conn(0x7f9ea4061a20 0x7f9ea41432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 -- 172.21.15.26:0/4241720814 >> 172.21.15.26:0/4241720814 conn(0x7f9ea405faf0 msgr2=0x7f9ea4066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:14.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 -- 172.21.15.26:0/4241720814 shutdown_connections 2022-07-20T02:35:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.507+0000 7f9eab534700 1 -- 172.21.15.26:0/4241720814 wait complete. 2022-07-20T02:35:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 Processor -- start 2022-07-20T02:35:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- start start 2022-07-20T02:35:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 0x7f9ea40b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea41468c0 0x7f9ea40b9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:14.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b0b80 con 0x7f9ea412c530 2022-07-20T02:35:14.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b0640 con 0x7f9ea41468c0 2022-07-20T02:35:14.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b08e0 con 0x7f9ea4061a20 2022-07-20T02:35:14.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43672/0 (socket says 172.21.15.26:43672) 2022-07-20T02:35:14.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 -- 172.21.15.26:0/3339918682 learned_addr learned my addr 172.21.15.26:0/3339918682 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:14.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea92d0700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 0x7f9ea40b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea9ad1700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea41468c0 0x7f9ea40b9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 -- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 msgr2=0x7f9ea40b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 0x7f9ea40b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 -- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea41468c0 msgr2=0x7f9ea40b9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea41468c0 0x7f9ea40b9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 -- 172.21.15.26:0/3339918682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea000b180 con 0x7f9ea412c530 2022-07-20T02:35:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea92d0700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 0x7f9ea40b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9ea8acf700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f9ea000c000 tx=0x7f9ea0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ea000ca40 con 0x7f9ea412c530 2022-07-20T02:35:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c009470 con 0x7f9ea412c530 2022-07-20T02:35:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea000cba0 con 0x7f9ea412c530 2022-07-20T02:35:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ea00123f0 con 0x7f9ea412c530 2022-07-20T02:35:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea40b9fa0 con 0x7f9ea412c530 2022-07-20T02:35:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.511+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea4146d20 con 0x7f9ea412c530 2022-07-20T02:35:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.515+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9ea0012550 con 0x7f9ea412c530 2022-07-20T02:35:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.515+0000 7f9e927fc700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 0x7f9e84055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.515+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f9ea006d1c0 con 0x7f9ea412c530 2022-07-20T02:35:15.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.515+0000 7f9ea92d0700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 0x7f9e84055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.515+0000 7f9ea92d0700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 0x7f9e84055ff0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f9e94008000 tx=0x7f9e94008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.523+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ea0043fa0 con 0x7f9ea412c530 2022-07-20T02:35:15.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.687+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f9ea40574e0 con 0x7f9ea412c530 2022-07-20T02:35:15.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.695+0000 7f9e927fc700 1 -- 172.21.15.26:0/3339918682 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v77) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9ea00475c0 con 0x7f9ea412c530 2022-07-20T02:35:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 msgr2=0x7f9e84055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 0x7f9e84055ff0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f9e94008000 tx=0x7f9e94008040).stop 2022-07-20T02:35:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 msgr2=0x7f9ea412a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f9ea000c000 tx=0x7f9ea0006df0).stop 2022-07-20T02:35:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 shutdown_connections 2022-07-20T02:35:15.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ea412c530 0x7f9ea412a3d0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9e84053b30 0x7f9e84055ff0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ea41468c0 0x7f9ea40b9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 --2- 172.21.15.26:0/3339918682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ea4061a20 0x7f9ea40b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 >> 172.21.15.26:0/3339918682 conn(0x7f9ea405faf0 msgr2=0x7f9ea40b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:15.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.699+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 shutdown_connections 2022-07-20T02:35:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:14.703+0000 7f9eab534700 1 -- 172.21.15.26:0/3339918682 wait complete. 2022-07-20T02:35:15.059 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 27 and data pool 28 2022-07-20T02:35:15.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:15.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 Processor -- start 2022-07-20T02:35:15.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 -- start start 2022-07-20T02:35:15.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03a00b2ab0 0x7f03a0061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a00aa1e0 con 0x7f03a00af6d0 2022-07-20T02:35:15.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a00ae110 con 0x7f03a00add30 2022-07-20T02:35:15.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.170+0000 7f03a69ae700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a00a9f40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03a00b2ab0 0x7f03a0061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f03a00b2ab0 0x7f03a0061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43690/0 (socket says 172.21.15.26:43690) 2022-07-20T02:35:15.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a61ad700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f03a00af6d0 0x7f03a00afb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42972/0 (socket says 172.21.15.26:42972) 2022-07-20T02:35:15.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 -- 172.21.15.26:0/390279029 learned_addr learned my addr 172.21.15.26:0/390279029 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:15.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 -- 172.21.15.26:0/390279029 >> v1:172.21.15.120:6789/0 conn(0x7f03a00add30 legacy=0x7f03a0060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 -- 172.21.15.26:0/390279029 >> v1:172.21.15.26:6789/0 conn(0x7f03a00af6d0 legacy=0x7f03a00afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 -- 172.21.15.26:0/390279029 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a00b1110 con 0x7f03a00b2ab0 2022-07-20T02:35:15.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a51ab700 1 --2- 172.21.15.26:0/390279029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0061060 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f039c002f70 tx=0x7f039c006eb0).ready entity=mon.0 client_cookie=d3105666def317c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a49aa700 1 -- 172.21.15.26:0/390279029 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039c00fa60 con 0x7f03a00b2ab0 2022-07-20T02:35:15.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a49aa700 1 -- 172.21.15.26:0/390279029 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c000e90 con 0x7f03a00b2ab0 2022-07-20T02:35:15.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a49aa700 1 -- 172.21.15.26:0/390279029 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039c00fdf0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- 172.21.15.26:0/390279029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 msgr2=0x7f03a0061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 --2- 172.21.15.26:0/390279029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0061060 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f039c002f70 tx=0x7f039c006eb0).stop 2022-07-20T02:35:15.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- 172.21.15.26:0/390279029 shutdown_connections 2022-07-20T02:35:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 --2- 172.21.15.26:0/390279029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0061060 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- 172.21.15.26:0/390279029 >> 172.21.15.26:0/390279029 conn(0x7f03a00b3050 msgr2=0x7f03a00b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- 172.21.15.26:0/390279029 shutdown_connections 2022-07-20T02:35:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- 172.21.15.26:0/390279029 wait complete. 2022-07-20T02:35:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 Processor -- start 2022-07-20T02:35:15.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 -- start start 2022-07-20T02:35:15.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 0x7f03a013eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.174+0000 7f03a69ae700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03a00af6d0 0x7f03a0145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a013a190 con 0x7f03a00b2ab0 2022-07-20T02:35:15.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a0146790 con 0x7f03a00add30 2022-07-20T02:35:15.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a013f310 con 0x7f03a00af6d0 2022-07-20T02:35:15.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43698/0 (socket says 172.21.15.26:43698) 2022-07-20T02:35:15.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 -- 172.21.15.26:0/1879123805 learned_addr learned my addr 172.21.15.26:0/1879123805 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:15.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a51ab700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03a00af6d0 0x7f03a0145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a59ac700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 0x7f03a013eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 -- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03a00af6d0 msgr2=0x7f03a0145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03a00af6d0 0x7f03a0145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 -- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 msgr2=0x7f03a013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 0x7f03a013eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 -- 172.21.15.26:0/1879123805 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c00b1e0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a59ac700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 0x7f03a013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:15.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a61ad700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f03a00630a0 tx=0x7f0398007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0398005ae0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a013f5b0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398005c40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a0064710 con 0x7f03a00b2ab0 2022-07-20T02:35:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03980070d0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a0138b40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.178+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0398010070 con 0x7f03a00b2ab0 2022-07-20T02:35:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.182+0000 7f038effd700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 0x7f0380055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.182+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f039806b9e0 con 0x7f03a00b2ab0 2022-07-20T02:35:15.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.182+0000 7f03a59ac700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 0x7f0380055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.182+0000 7f03a59ac700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 0x7f0380055fc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f039000b000 tx=0x7f0390009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.190+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03a0138b40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.318+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03a0138b40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.318+0000 7f038effd700 1 -- 172.21.15.26:0/1879123805 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f03a0138b40 con 0x7f03a00b2ab0 2022-07-20T02:35:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:15.407 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":149,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:14.690756+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:13.545762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:14.132638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:15.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 msgr2=0x7f0380055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 0x7f0380055fc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f039000b000 tx=0x7f0390009040).stop 2022-07-20T02:35:15.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 msgr2=0x7f03a0138de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f03a00630a0 tx=0x7f0398007f40).stop 2022-07-20T02:35:15.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 shutdown_connections 2022-07-20T02:35:15.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f03a00b2ab0 0x7f03a0138de0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0380053b00 0x7f0380055fc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03a00af6d0 0x7f03a0145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 --2- 172.21.15.26:0/1879123805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a00add30 0x7f03a013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 >> 172.21.15.26:0/1879123805 conn(0x7f03a00b3050 msgr2=0x7f03a01445c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:15.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 shutdown_connections 2022-07-20T02:35:15.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.322+0000 7f03a69ae700 1 -- 172.21.15.26:0/1879123805 wait complete. 2022-07-20T02:35:15.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:15.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 Processor -- start 2022-07-20T02:35:15.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- start start 2022-07-20T02:35:15.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f819c1484a0 0x7f819c066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0b1bc0 con 0x7f819c061050 2022-07-20T02:35:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c147700 con 0x7f819c0b2920 2022-07-20T02:35:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c14b040 con 0x7f819c1484a0 2022-07-20T02:35:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f819c1484a0 0x7f819c066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a10c1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f819c061050 0x7f819c066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42986/0 (socket says 172.21.15.26:42986) 2022-07-20T02:35:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a10c1700 1 -- 172.21.15.26:0/666222170 learned_addr learned my addr 172.21.15.26:0/666222170 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819b7fe700 1 -- 172.21.15.26:0/666222170 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646562642 0 0) 0x7f819c0b1bc0 con 0x7f819c061050 2022-07-20T02:35:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819b7fe700 1 -- 172.21.15.26:0/666222170 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c003260 con 0x7f819c061050 2022-07-20T02:35:15.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819bfff700 1 -- 172.21.15.26:0/666222170 >> v1:172.21.15.120:6789/0 conn(0x7f819c0b2920 legacy=0x7f819c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819bfff700 1 -- 172.21.15.26:0/666222170 >> v1:172.21.15.26:6789/0 conn(0x7f819c061050 legacy=0x7f819c066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819bfff700 1 -- 172.21.15.26:0/666222170 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c147930 con 0x7f819c1484a0 2022-07-20T02:35:15.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819bfff700 1 --2- 172.21.15.26:0/666222170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c1484a0 0x7f819c066a80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f818c0032f0 tx=0x7f818c0091d0).ready entity=mon.0 client_cookie=ad97a49fc715f3db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819b7fe700 1 -- 172.21.15.26:0/666222170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c00b6c0 con 0x7f819c1484a0 2022-07-20T02:35:15.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819b7fe700 1 -- 172.21.15.26:0/666222170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c00b820 con 0x7f819c1484a0 2022-07-20T02:35:15.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f819b7fe700 1 -- 172.21.15.26:0/666222170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c00b990 con 0x7f819c1484a0 2022-07-20T02:35:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- 172.21.15.26:0/666222170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c1484a0 msgr2=0x7f819c066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 --2- 172.21.15.26:0/666222170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c1484a0 0x7f819c066a80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f818c0032f0 tx=0x7f818c0091d0).stop 2022-07-20T02:35:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- 172.21.15.26:0/666222170 shutdown_connections 2022-07-20T02:35:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 --2- 172.21.15.26:0/666222170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c1484a0 0x7f819c066a80 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- 172.21.15.26:0/666222170 >> 172.21.15.26:0/666222170 conn(0x7f819c0b60a0 msgr2=0x7f819c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:15.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- 172.21.15.26:0/666222170 shutdown_connections 2022-07-20T02:35:15.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.642+0000 7f81a2b24700 1 -- 172.21.15.26:0/666222170 wait complete. 2022-07-20T02:35:15.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 Processor -- start 2022-07-20T02:35:15.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 -- start start 2022-07-20T02:35:15.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819c061050 0x7f819c0ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 0x7f819c1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 0x7f819c157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:15.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c156b10 con 0x7f819c0b2920 2022-07-20T02:35:15.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c0af510 con 0x7f819c1484a0 2022-07-20T02:35:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a2b24700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c156d00 con 0x7f819c061050 2022-07-20T02:35:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f819bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 0x7f819c1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f819bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 0x7f819c1571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43712/0 (socket says 172.21.15.26:43712) 2022-07-20T02:35:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f819bfff700 1 -- 172.21.15.26:0/1481016660 learned_addr learned my addr 172.21.15.26:0/1481016660 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 0x7f819c157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a08c0700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819c061050 0x7f819c0ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 -- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819c061050 msgr2=0x7f819c0ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819c061050 0x7f819c0ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 -- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 msgr2=0x7f819c1571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 0x7f819c1571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.646+0000 7f81a10c1700 1 -- 172.21.15.26:0/1481016660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c009470 con 0x7f819c1484a0 2022-07-20T02:35:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f81a10c1700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 0x7f819c157690 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f8194001110 tx=0x7f8194007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f819400ce80 con 0x7f819c1484a0 2022-07-20T02:35:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81940070d0 con 0x7f819c1484a0 2022-07-20T02:35:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8194007470 con 0x7f819c1484a0 2022-07-20T02:35:16.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c065330 con 0x7f819c1484a0 2022-07-20T02:35:16.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c0a9b10 con 0x7f819c1484a0 2022-07-20T02:35:16.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c0af370 con 0x7f819c1484a0 2022-07-20T02:35:16.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8194010070 con 0x7f819c1484a0 2022-07-20T02:35:16.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 0x7f8180055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f819406b7d0 con 0x7f819c1484a0 2022-07-20T02:35:16.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.650+0000 7f81a08c0700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 0x7f8180055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.654+0000 7f81a08c0700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 0x7f8180055f30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f818400d000 tx=0x7f8184007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:16.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.658+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f819c0af370 con 0x7f819c1484a0 2022-07-20T02:35:16.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.790+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f819c0b4d10 con 0x7f819c1484a0 2022-07-20T02:35:16.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.790+0000 7f8199ffb700 1 -- 172.21.15.26:0/1481016660 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f819c0b4d10 con 0x7f819c1484a0 2022-07-20T02:35:16.100 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:16.100 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":150,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:15.715598+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:13.545762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:14.132638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:16.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 msgr2=0x7f8180055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 0x7f8180055f30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f818400d000 tx=0x7f8184007b90).stop 2022-07-20T02:35:16.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 msgr2=0x7f819c157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 0x7f819c157690 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f8194001110 tx=0x7f8194007a40).stop 2022-07-20T02:35:16.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 shutdown_connections 2022-07-20T02:35:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f819c0b2920 0x7f819c1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8180053a70 0x7f8180055f30 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819c1484a0 0x7f819c157690 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 --2- 172.21.15.26:0/1481016660 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819c061050 0x7f819c0ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 >> 172.21.15.26:0/1481016660 conn(0x7f819c0b60a0 msgr2=0x7f819c0b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:16.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 shutdown_connections 2022-07-20T02:35:16.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:15.794+0000 7f81a2b24700 1 -- 172.21.15.26:0/1481016660 wait complete. 2022-07-20T02:35:16.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:35:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 Processor -- start 2022-07-20T02:35:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 -- start start 2022-07-20T02:35:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1dac05f840 0x7f1dac146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0b1ae0 con 0x7f1dac0b24c0 2022-07-20T02:35:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0b1d80 con 0x7f1dac1465b0 2022-07-20T02:35:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1dab59e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0b2e20 con 0x7f1dac05f840 2022-07-20T02:35:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1dac05f840 0x7f1dac146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1dac05f840 0x7f1dac146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43718/0 (socket says 172.21.15.26:43718) 2022-07-20T02:35:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 -- 172.21.15.26:0/2745154000 learned_addr learned my addr 172.21.15.26:0/2745154000 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:16.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 -- 172.21.15.26:0/2745154000 >> v1:172.21.15.120:6789/0 conn(0x7f1dac1465b0 legacy=0x7f1dac066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 -- 172.21.15.26:0/2745154000 >> v1:172.21.15.26:6789/0 conn(0x7f1dac0b24c0 legacy=0x7f1dac0604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1da959a700 1 -- 172.21.15.26:0/2745154000 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409614909 0 0) 0x7f1dac0b1ae0 con 0x7f1dac0b24c0 2022-07-20T02:35:16.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.242+0000 7f1daad9d700 1 -- 172.21.15.26:0/2745154000 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0b2020 con 0x7f1dac05f840 2022-07-20T02:35:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1daad9d700 1 --2- 172.21.15.26:0/2745154000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac05f840 0x7f1dac146990 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f1da40032f0 tx=0x7f1da4009290).ready entity=mon.0 client_cookie=475bfc79b2ca776a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1da959a700 1 -- 172.21.15.26:0/2745154000 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da400b8a0 con 0x7f1dac05f840 2022-07-20T02:35:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1da959a700 1 -- 172.21.15.26:0/2745154000 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da400ba00 con 0x7f1dac05f840 2022-07-20T02:35:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1da959a700 1 -- 172.21.15.26:0/2745154000 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da400bb70 con 0x7f1dac05f840 2022-07-20T02:35:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- 172.21.15.26:0/2745154000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac05f840 msgr2=0x7f1dac146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2745154000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac05f840 0x7f1dac146990 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f1da40032f0 tx=0x7f1da4009290).stop 2022-07-20T02:35:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- 172.21.15.26:0/2745154000 shutdown_connections 2022-07-20T02:35:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2745154000 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac05f840 0x7f1dac146990 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- 172.21.15.26:0/2745154000 >> 172.21.15.26:0/2745154000 conn(0x7f1dac0b3050 msgr2=0x7f1dac0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- 172.21.15.26:0/2745154000 shutdown_connections 2022-07-20T02:35:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- 172.21.15.26:0/2745154000 wait complete. 2022-07-20T02:35:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 Processor -- start 2022-07-20T02:35:16.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.246+0000 7f1dab59e700 1 -- start start 2022-07-20T02:35:16.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 0x7f1dac0ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1465b0 0x7f1dac1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac152e30 con 0x7f1dac0aa340 2022-07-20T02:35:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac1530d0 con 0x7f1dac05f840 2022-07-20T02:35:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0aaf90 con 0x7f1dac1465b0 2022-07-20T02:35:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43724/0 (socket says 172.21.15.26:43724) 2022-07-20T02:35:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 -- 172.21.15.26:0/2959593840 learned_addr learned my addr 172.21.15.26:0/2959593840 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:16.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1da9d9b700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1465b0 0x7f1dac1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daa59c700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 0x7f1dac0ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 -- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1465b0 msgr2=0x7f1dac1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1465b0 0x7f1dac1419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 -- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 msgr2=0x7f1dac0ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 0x7f1dac0ad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 -- 172.21.15.26:0/2959593840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009530 con 0x7f1dac0aa340 2022-07-20T02:35:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daa59c700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 0x7f1dac0ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1daad9d700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f1da40032f0 tx=0x7f1da4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da400b8a0 con 0x7f1dac0aa340 2022-07-20T02:35:16.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da401f3f0 con 0x7f1dac0aa340 2022-07-20T02:35:16.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da401f750 con 0x7f1dac0aa340 2022-07-20T02:35:16.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac0ab430 con 0x7f1dac0aa340 2022-07-20T02:35:16.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac0afc10 con 0x7f1dac0aa340 2022-07-20T02:35:16.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac0674c0 con 0x7f1dac0aa340 2022-07-20T02:35:16.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.250+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1da4026030 con 0x7f1dac0aa340 2022-07-20T02:35:16.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.254+0000 7f1d9b7fe700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 0x7f1d88055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:16.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.254+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1da406ce50 con 0x7f1dac0aa340 2022-07-20T02:35:16.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.254+0000 7f1daa59c700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 0x7f1d88055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:16.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.254+0000 7f1daa59c700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 0x7f1d88055ff0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f1da000b000 tx=0x7f1da0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:16.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.262+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1da4043c10 con 0x7f1dac0aa340 2022-07-20T02:35:16.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.426+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1dac0ab160 con 0x7f1dac0aa340 2022-07-20T02:35:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.738+0000 7f1d9b7fe700 1 -- 172.21.15.26:0/2959593840 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1da4024020 con 0x7f1dac0aa340 2022-07-20T02:35:16.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 msgr2=0x7f1d88055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 0x7f1d88055ff0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f1da000b000 tx=0x7f1da0009040).stop 2022-07-20T02:35:16.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 msgr2=0x7f1dac0af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f1da40032f0 tx=0x7f1da4016000).stop 2022-07-20T02:35:16.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 shutdown_connections 2022-07-20T02:35:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1dac0aa340 0x7f1dac0af720 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d88053b30 0x7f1d88055ff0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dac1465b0 0x7f1dac1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 --2- 172.21.15.26:0/2959593840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac05f840 0x7f1dac0ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:16.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 >> 172.21.15.26:0/2959593840 conn(0x7f1dac0b3050 msgr2=0x7f1dac0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:16.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 shutdown_connections 2022-07-20T02:35:16.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.742+0000 7f1dab59e700 1 -- 172.21.15.26:0/2959593840 wait complete. 2022-07-20T02:35:16.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:17.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.946+0000 7eff84544700 1 Processor -- start 2022-07-20T02:35:17.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.946+0000 7eff84544700 1 -- start start 2022-07-20T02:35:17.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 0x7eff7c066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7eff7c061050 0x7eff7c066f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c0b1bc0 con 0x7eff7c0b2920 2022-07-20T02:35:17.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c14b040 con 0x7eff7c061050 2022-07-20T02:35:17.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0a8860 con 0x7eff7c1484a0 2022-07-20T02:35:17.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7eff7c061050 0x7eff7c066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff822e0700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7eff7c0b2920 0x7eff7c147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43010/0 (socket says 172.21.15.26:43010) 2022-07-20T02:35:17.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7eff7c061050 0x7eff7c066f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43732/0 (socket says 172.21.15.26:43732) 2022-07-20T02:35:17.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff822e0700 1 -- 172.21.15.26:0/3322524190 learned_addr learned my addr 172.21.15.26:0/3322524190 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:17.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff81adf700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 0x7eff7c066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 -- 172.21.15.26:0/3322524190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 msgr2=0x7eff7c066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:17.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 0x7eff7c066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 -- 172.21.15.26:0/3322524190 >> v1:172.21.15.26:6789/0 conn(0x7eff7c0b2920 legacy=0x7eff7c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff812de700 1 -- 172.21.15.26:0/3322524190 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801183818 0 0) 0x7eff7c0b1bc0 con 0x7eff7c0b2920 2022-07-20T02:35:17.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 -- 172.21.15.26:0/3322524190 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c147700 con 0x7eff7c061050 2022-07-20T02:35:17.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff81adf700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 0x7eff7c066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:17.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff82ae1700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c061050 0x7eff7c066f70 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7eff74002f70 tx=0x7eff74006df0).ready entity=mon.0 client_cookie=3de61b8ea49ee73 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff812de700 1 -- 172.21.15.26:0/3322524190 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff7400d570 con 0x7eff7c061050 2022-07-20T02:35:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff812de700 1 -- 172.21.15.26:0/3322524190 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74000e90 con 0x7eff7c061050 2022-07-20T02:35:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff812de700 1 -- 172.21.15.26:0/3322524190 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff7400d8c0 con 0x7eff7c061050 2022-07-20T02:35:17.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- 172.21.15.26:0/3322524190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c061050 msgr2=0x7eff7c066f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c061050 0x7eff7c066f70 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7eff74002f70 tx=0x7eff74006df0).stop 2022-07-20T02:35:17.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- 172.21.15.26:0/3322524190 shutdown_connections 2022-07-20T02:35:17.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c061050 0x7eff7c066f70 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 --2- 172.21.15.26:0/3322524190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c1484a0 0x7eff7c066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- 172.21.15.26:0/3322524190 >> 172.21.15.26:0/3322524190 conn(0x7eff7c0b60a0 msgr2=0x7eff7c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:17.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- 172.21.15.26:0/3322524190 shutdown_connections 2022-07-20T02:35:17.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.950+0000 7eff84544700 1 -- 172.21.15.26:0/3322524190 wait complete. 2022-07-20T02:35:17.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 Processor -- start 2022-07-20T02:35:17.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 -- start start 2022-07-20T02:35:17.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c061050 0x7eff7c0ac040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c0b2920 0x7eff7c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0aaa00 con 0x7eff7c1484a0 2022-07-20T02:35:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c066420 con 0x7eff7c061050 2022-07-20T02:35:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff84544700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c157160 con 0x7eff7c0b2920 2022-07-20T02:35:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.954+0000 7eff82ae1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43738/0 (socket says 172.21.15.26:43738) 2022-07-20T02:35:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 -- 172.21.15.26:0/3809814784 learned_addr learned my addr 172.21.15.26:0/3809814784 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff81adf700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c0b2920 0x7eff7c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff822e0700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c061050 0x7eff7c0ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 -- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c0b2920 msgr2=0x7eff7c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c0b2920 0x7eff7c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 -- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c061050 msgr2=0x7eff7c0ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c061050 0x7eff7c0ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 -- 172.21.15.26:0/3809814784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78009470 con 0x7eff7c1484a0 2022-07-20T02:35:17.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff82ae1700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7eff74002f70 tx=0x7eff74014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff74016070 con 0x7eff7c1484a0 2022-07-20T02:35:17.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74013850 con 0x7eff7c1484a0 2022-07-20T02:35:17.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7400b180 con 0x7eff7c1484a0 2022-07-20T02:35:17.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff74013bb0 con 0x7eff7c1484a0 2022-07-20T02:35:17.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c066650 con 0x7eff7c1484a0 2022-07-20T02:35:17.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c14ada0 con 0x7eff7c1484a0 2022-07-20T02:35:17.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.958+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7eff74025030 con 0x7eff7c1484a0 2022-07-20T02:35:17.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.962+0000 7eff6b7fe700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 0x7eff5c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.962+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7eff7406ce40 con 0x7eff7c1484a0 2022-07-20T02:35:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.962+0000 7eff822e0700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 0x7eff5c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.962+0000 7eff822e0700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 0x7eff5c055ff0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7eff6c00d000 tx=0x7eff6c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:16.966+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7eff74043c00 con 0x7eff7c1484a0 2022-07-20T02:35:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.130+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff7c0574e0 con 0x7eff7c1484a0 2022-07-20T02:35:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.130+0000 7eff6b7fe700 1 -- 172.21.15.26:0/3809814784 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2486 (secure 0 0 0) 0x7eff74023020 con 0x7eff7c1484a0 2022-07-20T02:35:17.270 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:17.271 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:16.734420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8505":{"gid":8505,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/853590136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":853590136},{"type":"v1","addr":"172.21.15.26:6837","nonce":853590136}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:17.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 msgr2=0x7eff5c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 0x7eff5c055ff0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7eff6c00d000 tx=0x7eff6c009040).stop 2022-07-20T02:35:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 msgr2=0x7eff7c156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7eff74002f70 tx=0x7eff74014000).stop 2022-07-20T02:35:17.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 shutdown_connections 2022-07-20T02:35:17.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff7c1484a0 0x7eff7c156c70 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7eff5c053b30 0x7eff5c055ff0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff7c0b2920 0x7eff7c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 --2- 172.21.15.26:0/3809814784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c061050 0x7eff7c0ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 >> 172.21.15.26:0/3809814784 conn(0x7eff7c0b60a0 msgr2=0x7eff7c148ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:17.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 shutdown_connections 2022-07-20T02:35:17.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.134+0000 7eff84544700 1 -- 172.21.15.26:0/3809814784 wait complete. 2022-07-20T02:35:17.282 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 80 2022-07-20T02:35:17.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:17.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 Processor -- start 2022-07-20T02:35:17.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 -- start start 2022-07-20T02:35:17.492 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74040b2920 0x7f7404147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7404061050 0x7f7404148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74040b1bc0 con 0x7f7404066a80 2022-07-20T02:35:17.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740414b040 con 0x7f7404061050 2022-07-20T02:35:17.493 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f740956c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74040a8860 con 0x7f74040b2920 2022-07-20T02:35:17.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74037fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7404066a80 0x7f7404066ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43026/0 (socket says 172.21.15.26:43026) 2022-07-20T02:35:17.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74037fe700 1 -- 172.21.15.26:0/1597759360 learned_addr learned my addr 172.21.15.26:0/1597759360 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:17.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 --2- 172.21.15.26:0/1597759360 >> v2:172.21.15.120:3300/0 conn(0x7f7404061050 0x7f7404148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7402ffd700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74040b2920 0x7f7404147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608165977 0 0) 0x7f74040b1bc0 con 0x7f7404066a80 2022-07-20T02:35:17.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8003260 con 0x7f7404066a80 2022-07-20T02:35:17.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 -- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74040b2920 msgr2=0x7f7404147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:17.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74040b2920 0x7f7404147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 -- 172.21.15.26:0/1597759360 >> v1:172.21.15.26:6789/0 conn(0x7f7404066a80 legacy=0x7f7404066ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41914874 0 0) 0x7f73e8003260 con 0x7f7404066a80 2022-07-20T02:35:17.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 -- 172.21.15.26:0/1597759360 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404147700 con 0x7f7404061050 2022-07-20T02:35:17.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f74027fc700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404061050 0x7f7404148280 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f73f4001800 tx=0x7f73fc0091d0).ready entity=mon.1 client_cookie=33d5b495d72fadd4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73f4001660 con 0x7f7404061050 2022-07-20T02:35:17.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73fc00f510 con 0x7f7404061050 2022-07-20T02:35:17.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.418+0000 7f7401ffb700 1 -- 172.21.15.26:0/1597759360 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73fc00f680 con 0x7f7404061050 2022-07-20T02:35:17.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404061050 msgr2=0x7f7404148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404061050 0x7f7404148280 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f73f4001800 tx=0x7f73fc0091d0).stop 2022-07-20T02:35:17.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- 172.21.15.26:0/1597759360 shutdown_connections 2022-07-20T02:35:17.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404061050 0x7f7404148280 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- 172.21.15.26:0/1597759360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74040b2920 0x7f7404147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- 172.21.15.26:0/1597759360 >> 172.21.15.26:0/1597759360 conn(0x7f74040b60a0 msgr2=0x7f74040b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:17.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- 172.21.15.26:0/1597759360 shutdown_connections 2022-07-20T02:35:17.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- 172.21.15.26:0/1597759360 wait complete. 2022-07-20T02:35:17.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 Processor -- start 2022-07-20T02:35:17.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- start start 2022-07-20T02:35:17.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7404061050 0x7f74040b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404066a80 0x7f7404064220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7404065410 con 0x7f74040b2920 2022-07-20T02:35:17.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7404064d40 con 0x7f7404066a80 2022-07-20T02:35:17.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f740956c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7404065cb0 con 0x7f7404061050 2022-07-20T02:35:17.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43752/0 (socket says 172.21.15.26:43752) 2022-07-20T02:35:17.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 -- 172.21.15.26:0/3680127334 learned_addr learned my addr 172.21.15.26:0/3680127334 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:17.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f7402ffd700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7404061050 0x7f74040b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.505 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74027fc700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404066a80 0x7f7404064220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 -- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7404061050 msgr2=0x7f74040b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7404061050 0x7f74040b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 -- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404066a80 msgr2=0x7f7404064220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404066a80 0x7f7404064220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 -- 172.21.15.26:0/3680127334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc009470 con 0x7f74040b2920 2022-07-20T02:35:17.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.422+0000 7f74037fe700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f74040a9c30 tx=0x7f73f4001b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73f4007170 con 0x7f74040b2920 2022-07-20T02:35:17.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f40072d0 con 0x7f74040b2920 2022-07-20T02:35:17.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7404065f50 con 0x7f74040b2920 2022-07-20T02:35:17.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73f4004920 con 0x7f74040b2920 2022-07-20T02:35:17.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74040ab560 con 0x7f74040b2920 2022-07-20T02:35:17.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73f4004a80 con 0x7f74040b2920 2022-07-20T02:35:17.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 0x7f73e0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7404065170 con 0x7f74040b2920 2022-07-20T02:35:17.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f73f406bca0 con 0x7f74040b2920 2022-07-20T02:35:17.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f7402ffd700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 0x7f73e0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.426+0000 7f7402ffd700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 0x7f73e0055fc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f73f800b000 tx=0x7f73f8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.434+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7404065170 con 0x7f74040b2920 2022-07-20T02:35:17.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.558+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7404065170 con 0x7f74040b2920 2022-07-20T02:35:17.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f73f3fff700 1 -- 172.21.15.26:0/3680127334 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f7404065170 con 0x7f74040b2920 2022-07-20T02:35:17.577 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:17.578 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":150,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:15.715598+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:13.545762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:14.132638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:17.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 msgr2=0x7f73e0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 0x7f73e0055fc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f73f800b000 tx=0x7f73f8006df0).stop 2022-07-20T02:35:17.586 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 msgr2=0x7f7404145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f74040a9c30 tx=0x7f73f4001b30).stop 2022-07-20T02:35:17.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 shutdown_connections 2022-07-20T02:35:17.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f74040b2920 0x7f7404145890 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73e0053b00 0x7f73e0055fc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404066a80 0x7f7404064220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 --2- 172.21.15.26:0/3680127334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7404061050 0x7f74040b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 >> 172.21.15.26:0/3680127334 conn(0x7f74040b60a0 msgr2=0x7f740414a220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:17.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 shutdown_connections 2022-07-20T02:35:17.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.562+0000 7f740956c700 1 -- 172.21.15.26:0/3680127334 wait complete. 2022-07-20T02:35:17.626 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:35:17.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 Processor -- start 2022-07-20T02:35:17.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 -- start start 2022-07-20T02:35:17.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59b80add60 0x7f59b80af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59b80a9ef0 0x7f59b8142570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80b12b0 con 0x7f59b80b2d40 2022-07-20T02:35:17.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b8067110 con 0x7f59b80a9ef0 2022-07-20T02:35:17.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59bcd84700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b8066e70 con 0x7f59b80add60 2022-07-20T02:35:17.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59b80a9ef0 0x7f59b8142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f59b80a9ef0 0x7f59b8142570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43760/0 (socket says 172.21.15.26:43760) 2022-07-20T02:35:17.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 -- 172.21.15.26:0/646442676 learned_addr learned my addr 172.21.15.26:0/646442676 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:17.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b659c700 1 --2- 172.21.15.26:0/646442676 >> v2:172.21.15.120:3300/0 conn(0x7f59b80add60 0x7f59b80af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b559a700 1 -- 172.21.15.26:0/646442676 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448642017 0 0) 0x7f59b80b12b0 con 0x7f59b80b2d40 2022-07-20T02:35:17.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 -- 172.21.15.26:0/646442676 >> v2:172.21.15.120:3300/0 conn(0x7f59b80add60 msgr2=0x7f59b80af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:17.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 --2- 172.21.15.26:0/646442676 >> v2:172.21.15.120:3300/0 conn(0x7f59b80add60 0x7f59b80af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 -- 172.21.15.26:0/646442676 >> v1:172.21.15.120:6789/0 conn(0x7f59b80b2d40 legacy=0x7f59b80afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 -- 172.21.15.26:0/646442676 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b80b0400 con 0x7f59b80a9ef0 2022-07-20T02:35:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.750+0000 7f59b6d9d700 1 --2- 172.21.15.26:0/646442676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8142570 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f59b80af750 tx=0x7f59b0006eb0).ready entity=mon.0 client_cookie=2f295ac7cb6ce0a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59b559a700 1 -- 172.21.15.26:0/646442676 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b000d610 con 0x7f59b80a9ef0 2022-07-20T02:35:17.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59b559a700 1 -- 172.21.15.26:0/646442676 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b0000e90 con 0x7f59b80a9ef0 2022-07-20T02:35:17.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59b559a700 1 -- 172.21.15.26:0/646442676 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b000d960 con 0x7f59b80a9ef0 2022-07-20T02:35:17.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- 172.21.15.26:0/646442676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 msgr2=0x7f59b8142570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 --2- 172.21.15.26:0/646442676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8142570 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f59b80af750 tx=0x7f59b0006eb0).stop 2022-07-20T02:35:17.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- 172.21.15.26:0/646442676 shutdown_connections 2022-07-20T02:35:17.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 --2- 172.21.15.26:0/646442676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8142570 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 --2- 172.21.15.26:0/646442676 >> v2:172.21.15.120:3300/0 conn(0x7f59b80add60 0x7f59b80af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- 172.21.15.26:0/646442676 >> 172.21.15.26:0/646442676 conn(0x7f59b80b66e0 msgr2=0x7f59b80b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- 172.21.15.26:0/646442676 shutdown_connections 2022-07-20T02:35:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- 172.21.15.26:0/646442676 wait complete. 2022-07-20T02:35:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 Processor -- start 2022-07-20T02:35:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.754+0000 7f59bcd84700 1 -- start start 2022-07-20T02:35:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b80add60 0x7f59b80aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b80b2d40 0x7f59b80ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b813fc10 con 0x7f59b80a9ef0 2022-07-20T02:35:17.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b8145570 con 0x7f59b80b2d40 2022-07-20T02:35:17.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b80a91d0 con 0x7f59b80add60 2022-07-20T02:35:17.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43762/0 (socket says 172.21.15.26:43762) 2022-07-20T02:35:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 -- 172.21.15.26:0/4082634783 learned_addr learned my addr 172.21.15.26:0/4082634783 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b6d9d700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b80b2d40 0x7f59b80ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b5d9b700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b80add60 0x7f59b80aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 -- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b80add60 msgr2=0x7f59b80aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b80add60 0x7f59b80aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 -- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b80b2d40 msgr2=0x7f59b80ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b80b2d40 0x7f59b80ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 -- 172.21.15.26:0/4082634783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b000b1e0 con 0x7f59b80a9ef0 2022-07-20T02:35:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59b659c700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f59a000c000 tx=0x7f59a00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:17.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a000ee30 con 0x7f59b80a9ef0 2022-07-20T02:35:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a001a470 con 0x7f59b80a9ef0 2022-07-20T02:35:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b80ab5c0 con 0x7f59b80a9ef0 2022-07-20T02:35:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a001a7d0 con 0x7f59b80a9ef0 2022-07-20T02:35:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.758+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b81452d0 con 0x7f59b80a9ef0 2022-07-20T02:35:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f59a001a930 con 0x7f59b80a9ef0 2022-07-20T02:35:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59af7fe700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 0x7f599c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b809f980 con 0x7f59b80a9ef0 2022-07-20T02:35:18.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f59a006ebb0 con 0x7f59b80a9ef0 2022-07-20T02:35:18.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59b5d9b700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 0x7f599c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.762+0000 7f59b5d9b700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 0x7f599c055ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f59a800b000 tx=0x7f59a8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:18.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.766+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59a0045970 con 0x7f59b80a9ef0 2022-07-20T02:35:18.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.934+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f59b8066af0 con 0x7f59b80a9ef0 2022-07-20T02:35:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59af7fe700 1 -- 172.21.15.26:0/4082634783 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7f59a001f070 con 0x7f59b80a9ef0 2022-07-20T02:35:18.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 msgr2=0x7f599c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 0x7f599c055ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f59a800b000 tx=0x7f59a8009040).stop 2022-07-20T02:35:18.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 msgr2=0x7f59b8141430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f59a000c000 tx=0x7f59a00091d0).stop 2022-07-20T02:35:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 shutdown_connections 2022-07-20T02:35:18.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f599c053b30 0x7f599c055ff0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b80b2d40 0x7f59b80ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b80add60 0x7f59b80aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 --2- 172.21.15.26:0/4082634783 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f59b80a9ef0 0x7f59b8141430 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 >> 172.21.15.26:0/4082634783 conn(0x7f59b80b66e0 msgr2=0x7f59b8060950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:18.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 shutdown_connections 2022-07-20T02:35:18.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:17.946+0000 7f59bcd84700 1 -- 172.21.15.26:0/4082634783 wait complete. 2022-07-20T02:35:18.045 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:35:18.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:35:18.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 Processor -- start 2022-07-20T02:35:18.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 -- start start 2022-07-20T02:35:18.257 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1805cfd0 0x7f4f18143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f18060f00 0x7f4f1813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f181474e0 con 0x7f4f18147000 2022-07-20T02:35:18.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180b4e40 con 0x7f4f18060f00 2022-07-20T02:35:18.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1e34c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1805d3b0 con 0x7f4f1805cfd0 2022-07-20T02:35:18.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f18060f00 0x7f4f1813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1cb49700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4f18147000 0x7f4f180b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43052/0 (socket says 172.21.15.26:43052) 2022-07-20T02:35:18.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1cb49700 1 -- 172.21.15.26:0/1706567298 learned_addr learned my addr 172.21.15.26:0/1706567298 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:18.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1d34a700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1805cfd0 0x7f4f18143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 -- 172.21.15.26:0/1706567298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1805cfd0 msgr2=0x7f4f18143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:18.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1805cfd0 0x7f4f18143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 -- 172.21.15.26:0/1706567298 >> v1:172.21.15.26:6789/0 conn(0x7f4f18147000 legacy=0x7f4f180b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 -- 172.21.15.26:0/1706567298 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f181431e0 con 0x7f4f18060f00 2022-07-20T02:35:18.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f1db4b700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f1813c9b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f4f100032f0 tx=0x7f4f100091d0).ready entity=mon.0 client_cookie=846283fc2e066e2e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:18.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f0ffff700 1 -- 172.21.15.26:0/1706567298 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000b6c0 con 0x7f4f18060f00 2022-07-20T02:35:18.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f0ffff700 1 -- 172.21.15.26:0/1706567298 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000b820 con 0x7f4f18060f00 2022-07-20T02:35:18.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.166+0000 7f4f0ffff700 1 -- 172.21.15.26:0/1706567298 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000b990 con 0x7f4f18060f00 2022-07-20T02:35:18.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- 172.21.15.26:0/1706567298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 msgr2=0x7f4f1813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f1813c9b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f4f100032f0 tx=0x7f4f100091d0).stop 2022-07-20T02:35:18.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- 172.21.15.26:0/1706567298 shutdown_connections 2022-07-20T02:35:18.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f1813c9b0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/1706567298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f1805cfd0 0x7f4f18143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- 172.21.15.26:0/1706567298 >> 172.21.15.26:0/1706567298 conn(0x7f4f180620c0 msgr2=0x7f4f1805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:18.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- 172.21.15.26:0/1706567298 shutdown_connections 2022-07-20T02:35:18.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- 172.21.15.26:0/1706567298 wait complete. 2022-07-20T02:35:18.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 Processor -- start 2022-07-20T02:35:18.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- start start 2022-07-20T02:35:18.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147000 0x7f4f181442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 0x7f4f18144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18156430 con 0x7f4f18060f00 2022-07-20T02:35:18.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.170+0000 7f4f1e34c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180b2540 con 0x7f4f18147000 2022-07-20T02:35:18.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1e34c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180b27e0 con 0x7f4f180b3a10 2022-07-20T02:35:18.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43776/0 (socket says 172.21.15.26:43776) 2022-07-20T02:35:18.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 -- 172.21.15.26:0/3214233404 learned_addr learned my addr 172.21.15.26:0/3214233404 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:18.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1db4b700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 0x7f4f18144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1cb49700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147000 0x7f4f181442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 -- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 msgr2=0x7f4f18144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 0x7f4f18144790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 -- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147000 msgr2=0x7f4f181442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147000 0x7f4f181442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 -- 172.21.15.26:0/3214233404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10009470 con 0x7f4f18060f00 2022-07-20T02:35:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1db4b700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 0x7f4f18144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1d34a700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f4f1400c000 tx=0x7f4f14006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f14010780 con 0x7f4f18060f00 2022-07-20T02:35:18.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f140108e0 con 0x7f4f18060f00 2022-07-20T02:35:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18156160 con 0x7f4f18060f00 2022-07-20T02:35:18.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f14010c40 con 0x7f4f18060f00 2022-07-20T02:35:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f18144ea0 con 0x7f4f18060f00 2022-07-20T02:35:18.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f1401c460 con 0x7f4f18060f00 2022-07-20T02:35:18.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.174+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18146900 con 0x7f4f18060f00 2022-07-20T02:35:18.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.178+0000 7f4f0e7fc700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 0x7f4f00055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.178+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4f1406c470 con 0x7f4f18060f00 2022-07-20T02:35:18.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.178+0000 7f4f1cb49700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 0x7f4f00055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.178+0000 7f4f1cb49700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 0x7f4f00055f60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f4f0800b000 tx=0x7f4f08009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:18.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.186+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f18146900 con 0x7f4f18060f00 2022-07-20T02:35:18.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:18.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.346+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4f18146900 con 0x7f4f18060f00 2022-07-20T02:35:18.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.354+0000 7f4f0e7fc700 1 -- 172.21.15.26:0/3214233404 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4f18146900 con 0x7f4f18060f00 2022-07-20T02:35:18.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 msgr2=0x7f4f00055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 0x7f4f00055f60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f4f0800b000 tx=0x7f4f08009040).stop 2022-07-20T02:35:18.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 msgr2=0x7f4f180b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.568 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f4f1400c000 tx=0x7f4f14006df0).stop 2022-07-20T02:35:18.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 shutdown_connections 2022-07-20T02:35:18.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f00053aa0 0x7f4f00055f60 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f180b3a10 0x7f4f18144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f18147000 0x7f4f181442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 --2- 172.21.15.26:0/3214233404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f18060f00 0x7f4f180b3520 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 >> 172.21.15.26:0/3214233404 conn(0x7f4f1805cfd0 msgr2=0x7f4f1813be60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:18.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 shutdown_connections 2022-07-20T02:35:18.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.358+0000 7f4f1e34c700 1 -- 172.21.15.26:0/3214233404 wait complete. 2022-07-20T02:35:18.572 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:35:18.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:18.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:18.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:18.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:18.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 Processor -- start 2022-07-20T02:35:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 -- start start 2022-07-20T02:35:18.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff56005cfd0 0x7ff560143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5601474e0 con 0x7ff560147000 2022-07-20T02:35:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b4e40 con 0x7ff560060f00 2022-07-20T02:35:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff565424700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56005d3b0 con 0x7ff56005cfd0 2022-07-20T02:35:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff55f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff560147000 0x7ff5600b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43064/0 (socket says 172.21.15.26:43064) 2022-07-20T02:35:18.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff55f7fe700 1 -- 172.21.15.26:0/356737449 learned_addr learned my addr 172.21.15.26:0/356737449 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.726+0000 7ff55ffff700 1 --2- 172.21.15.26:0/356737449 >> v2:172.21.15.120:3300/0 conn(0x7ff56005cfd0 0x7ff560143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411161134 0 0) 0x7ff5601474e0 con 0x7ff560147000 2022-07-20T02:35:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540003260 con 0x7ff560147000 2022-07-20T02:35:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 835037861 0 0) 0x7ff540003260 con 0x7ff560147000 2022-07-20T02:35:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 >> v2:172.21.15.120:3300/0 conn(0x7ff56005cfd0 msgr2=0x7ff560143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 --2- 172.21.15.26:0/356737449 >> v2:172.21.15.120:3300/0 conn(0x7ff56005cfd0 0x7ff560143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 >> v1:172.21.15.120:6789/0 conn(0x7ff560060f00 legacy=0x7ff56013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5601431e0 con 0x7ff560147000 2022-07-20T02:35:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7ff5480019e0 con 0x7ff560147000 2022-07-20T02:35:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 >> v1:172.21.15.26:6789/0 conn(0x7ff560147000 legacy=0x7ff5600b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff540003750 0x7ff540003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540004500 0x7ff540004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 0x7ff54000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56005d3b0 con 0x7ff540004e70 2022-07-20T02:35:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54000bd20 con 0x7ff540004500 2022-07-20T02:35:18.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54000bee0 con 0x7ff540003750 2022-07-20T02:35:18.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 0x7ff54000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55f7fe700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540004500 0x7ff540004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:18.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 -- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff540003750 msgr2=0x7ff540003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:18.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff540003750 0x7ff540003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T02:35:18.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 -- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540004500 msgr2=0x7ff540004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:18.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540004500 0x7ff540004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 -- 172.21.15.26:0/356737449 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff540003260 con 0x7ff540004e70 2022-07-20T02:35:18.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 -- 172.21.15.26:0/356737449 reap_dead start 2022-07-20T02:35:18.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff564c23700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 0x7ff54000b750 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7ff560068700 tx=0x7ff550007a00).ready entity=mon.0 client_cookie=704f6b519ae17581 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:18.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550007d10 con 0x7ff540004e70 2022-07-20T02:35:18.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550007030 con 0x7ff540004e70 2022-07-20T02:35:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff55effd700 1 -- 172.21.15.26:0/356737449 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550007390 con 0x7ff540004e70 2022-07-20T02:35:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 -- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 msgr2=0x7ff54000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 0x7ff54000b750 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7ff560068700 tx=0x7ff550007a00).stop 2022-07-20T02:35:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 -- 172.21.15.26:0/356737449 shutdown_connections 2022-07-20T02:35:18.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 --2- 172.21.15.26:0/356737449 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff540004e70 0x7ff54000b750 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:18.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 -- 172.21.15.26:0/356737449 >> 172.21.15.26:0/356737449 conn(0x7ff5600620c0 msgr2=0x7ff56005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:18.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 -- 172.21.15.26:0/356737449 shutdown_connections 2022-07-20T02:35:18.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.730+0000 7ff565424700 1 -- 172.21.15.26:0/356737449 wait complete. 2022-07-20T02:35:18.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 Processor -- start 2022-07-20T02:35:18.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- start start 2022-07-20T02:35:18.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600b3170 0x7ff560155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff560156190 0x7ff560144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:18.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5601452d0 con 0x7ff560144950 2022-07-20T02:35:19.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff560146280 con 0x7ff560156190 2022-07-20T02:35:19.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56013d150 con 0x7ff5600b3170 2022-07-20T02:35:19.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43800/0 (socket says 172.21.15.26:43800) 2022-07-20T02:35:19.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 -- 172.21.15.26:0/2818900057 learned_addr learned my addr 172.21.15.26:0/2818900057 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:19.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff55ffff700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600b3170 0x7ff560155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff55f7fe700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff560156190 0x7ff560144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 -- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600b3170 msgr2=0x7ff560155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600b3170 0x7ff560155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 -- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff560156190 msgr2=0x7ff560144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff560156190 0x7ff560144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 -- 172.21.15.26:0/2818900057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600af150 con 0x7ff560144950 2022-07-20T02:35:19.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff564c23700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7ff560068700 tx=0x7ff550015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:19.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550018030 con 0x7ff560144950 2022-07-20T02:35:19.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550006e90 con 0x7ff560144950 2022-07-20T02:35:19.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550007d10 con 0x7ff560144950 2022-07-20T02:35:19.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5600b3550 con 0x7ff560144950 2022-07-20T02:35:19.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.734+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff560145f30 con 0x7ff560144950 2022-07-20T02:35:19.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5601460c0 con 0x7ff560144950 2022-07-20T02:35:19.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff550026070 con 0x7ff560144950 2022-07-20T02:35:19.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff55d7fa700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 0x7ff544055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff550001e40 con 0x7ff560144950 2022-07-20T02:35:19.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff55ffff700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 0x7ff544055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.738+0000 7ff55ffff700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 0x7ff544055f60 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7ff55800b000 tx=0x7ff558006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:19.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.742+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff5500431b0 con 0x7ff560144950 2022-07-20T02:35:19.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff560143480 con 0x7ff560144950 2022-07-20T02:35:19.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff55d7fa700 1 -- 172.21.15.26:0/2818900057 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2486 (secure 0 0 0) 0x7ff5500467d0 con 0x7ff560144950 2022-07-20T02:35:19.079 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:19.080 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:16.734420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8505":{"gid":8505,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/853590136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":853590136},{"type":"v1","addr":"172.21.15.26:6837","nonce":853590136}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:19.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 msgr2=0x7ff544055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 0x7ff544055f60 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7ff55800b000 tx=0x7ff558006df0).stop 2022-07-20T02:35:19.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 msgr2=0x7ff560144de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7ff560068700 tx=0x7ff550015000).stop 2022-07-20T02:35:19.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 shutdown_connections 2022-07-20T02:35:19.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff560144950 0x7ff560144de0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff544053aa0 0x7ff544055f60 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff560156190 0x7ff560144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 --2- 172.21.15.26:0/2818900057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600b3170 0x7ff560155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 >> 172.21.15.26:0/2818900057 conn(0x7ff540004e70 msgr2=0x7ff5600b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:19.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 shutdown_connections 2022-07-20T02:35:19.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:18.906+0000 7ff565424700 1 -- 172.21.15.26:0/2818900057 wait complete. 2022-07-20T02:35:19.109 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 80 2022-07-20T02:35:19.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:14.691126+0000', 'modified': '2022-07-20T02:35:16.734420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8505': {'gid': 8505, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/853590136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 853590136}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 853590136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:19.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:19.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:19.559 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8505,"want_state":"up:active","state":"up:active","rank_uptime":4.8555293229999998,"mdsmap_epoch":80,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":9.2551677320000003} 2022-07-20T02:35:19.559 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8505, "mdsmap_epoch": 80, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 4.855529323, "state": "up:active", "uptime": 9.255167732, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:19.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 Processor -- start 2022-07-20T02:35:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- start start 2022-07-20T02:35:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b7c0b2920 0x7f9b7c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c0b1bc0 con 0x7f9b7c061050 2022-07-20T02:35:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b7c14b040 con 0x7f9b7c0b2920 2022-07-20T02:35:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0a8860 con 0x7f9b7c067190 2022-07-20T02:35:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b812c7700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9b7c061050 0x7f9b7c066a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43086/0 (socket says 172.21.15.26:43086) 2022-07-20T02:35:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b812c7700 1 -- 172.21.15.26:0/95511006 learned_addr learned my addr 172.21.15.26:0/95511006 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b81ac8700 1 --2- 172.21.15.26:0/95511006 >> v2:172.21.15.120:3300/0 conn(0x7f9b7c0b2920 0x7f9b7c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 --2- 172.21.15.26:0/95511006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b80ac6700 1 -- 172.21.15.26:0/95511006 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996969903 0 0) 0x7f9b7c0b1bc0 con 0x7f9b7c061050 2022-07-20T02:35:19.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b80ac6700 1 -- 172.21.15.26:0/95511006 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003260 con 0x7f9b7c061050 2022-07-20T02:35:19.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 -- 172.21.15.26:0/95511006 >> v2:172.21.15.120:3300/0 conn(0x7f9b7c0b2920 msgr2=0x7f9b7c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 --2- 172.21.15.26:0/95511006 >> v2:172.21.15.120:3300/0 conn(0x7f9b7c0b2920 0x7f9b7c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 -- 172.21.15.26:0/95511006 >> v1:172.21.15.26:6789/0 conn(0x7f9b7c061050 legacy=0x7f9b7c066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 -- 172.21.15.26:0/95511006 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c147700 con 0x7f9b7c067190 2022-07-20T02:35:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b822c9700 1 --2- 172.21.15.26:0/95511006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c148280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9b740032f0 tx=0x7f9b740091d0).ready entity=mon.2 client_cookie=f4e822f214ffbd4d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:19.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b80ac6700 1 -- 172.21.15.26:0/95511006 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b7400b600 con 0x7f9b7c067190 2022-07-20T02:35:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b80ac6700 1 -- 172.21.15.26:0/95511006 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7400b760 con 0x7f9b7c067190 2022-07-20T02:35:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b80ac6700 1 -- 172.21.15.26:0/95511006 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b7400b8d0 con 0x7f9b7c067190 2022-07-20T02:35:19.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- 172.21.15.26:0/95511006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 msgr2=0x7f9b7c148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/95511006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c148280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9b740032f0 tx=0x7f9b740091d0).stop 2022-07-20T02:35:19.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- 172.21.15.26:0/95511006 shutdown_connections 2022-07-20T02:35:19.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/95511006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c148280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/95511006 >> v2:172.21.15.120:3300/0 conn(0x7f9b7c0b2920 0x7f9b7c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- 172.21.15.26:0/95511006 >> 172.21.15.26:0/95511006 conn(0x7f9b7c0b60a0 msgr2=0x7f9b7c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:19.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- 172.21.15.26:0/95511006 shutdown_connections 2022-07-20T02:35:19.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.666+0000 7f9b83d2c700 1 -- 172.21.15.26:0/95511006 wait complete. 2022-07-20T02:35:19.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 Processor -- start 2022-07-20T02:35:19.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- start start 2022-07-20T02:35:19.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 0x7f9b7c065a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0668e0 con 0x7f9b7c0b2920 2022-07-20T02:35:19.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c064c90 con 0x7f9b7c061050 2022-07-20T02:35:19.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0663a0 con 0x7f9b7c067190 2022-07-20T02:35:19.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b812c7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43812/0 (socket says 172.21.15.26:43812) 2022-07-20T02:35:19.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 -- 172.21.15.26:0/3664028947 learned_addr learned my addr 172.21.15.26:0/3664028947 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:19.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b81ac8700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 0x7f9b7c065a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 -- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 msgr2=0x7f9b7c14e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c14e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 -- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 msgr2=0x7f9b7c065a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 0x7f9b7c065a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 -- 172.21.15.26:0/3664028947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b74009470 con 0x7f9b7c0b2920 2022-07-20T02:35:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b81ac8700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 0x7f9b7c065a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b822c9700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f9b740032f0 tx=0x7f9b74016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:19.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b7400b8d0 con 0x7f9b7c0b2920 2022-07-20T02:35:19.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b74021670 con 0x7f9b7c0b2920 2022-07-20T02:35:19.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b7400b600 con 0x7f9b7c0b2920 2022-07-20T02:35:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c066640 con 0x7f9b7c0b2920 2022-07-20T02:35:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.670+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c0aa3d0 con 0x7f9b7c0b2920 2022-07-20T02:35:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c15aa00 con 0x7f9b7c0b2920 2022-07-20T02:35:19.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b74028070 con 0x7f9b7c0b2920 2022-07-20T02:35:19.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b6affd700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 0x7f9b5c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:19.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9b7406d1f0 con 0x7f9b7c0b2920 2022-07-20T02:35:19.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b81ac8700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 0x7f9b5c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:19.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.674+0000 7f9b81ac8700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 0x7f9b5c055f30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f9b6c00b000 tx=0x7f9b6c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:19.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.678+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b7c15aa00 con 0x7f9b7c0b2920 2022-07-20T02:35:19.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.806+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b7c15aa00 con 0x7f9b7c0b2920 2022-07-20T02:35:19.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.810+0000 7f9b6affd700 1 -- 172.21.15.26:0/3664028947 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f9b7c15aa00 con 0x7f9b7c0b2920 2022-07-20T02:35:19.843 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:19.843 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":150,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:15.715598+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:13.545762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:14.132638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.810+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 msgr2=0x7f9b5c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.810+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 0x7f9b5c055f30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f9b6c00b000 tx=0x7f9b6c008040).stop 2022-07-20T02:35:19.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.810+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 msgr2=0x7f9b7c0a9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:19.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.810+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f9b740032f0 tx=0x7f9b74016000).stop 2022-07-20T02:35:19.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 shutdown_connections 2022-07-20T02:35:19.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b7c0b2920 0x7f9b7c0a9cc0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b5c053a70 0x7f9b5c055f30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b7c067190 0x7f9b7c14e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 --2- 172.21.15.26:0/3664028947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b7c061050 0x7f9b7c065a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 >> 172.21.15.26:0/3664028947 conn(0x7f9b7c0b60a0 msgr2=0x7f9b7c149880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:19.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 shutdown_connections 2022-07-20T02:35:19.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:19.814+0000 7f9b83d2c700 1 -- 172.21.15.26:0/3664028947 wait complete. 2022-07-20T02:35:19.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:35:13.545762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:35:14.132638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:35:19.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 Processor -- start 2022-07-20T02:35:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 -- start start 2022-07-20T02:35:20.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfe8143070 0x7fbfe81434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbfe8140320 0x7fbfe81407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe8146750 con 0x7fbfe80b5d50 2022-07-20T02:35:20.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe8067ba0 con 0x7fbfe8143070 2022-07-20T02:35:20.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfec9d4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe81469f0 con 0x7fbfe8140320 2022-07-20T02:35:20.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfe8143070 0x7fbfe81434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbfe8143070 0x7fbfe81434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43822/0 (socket says 172.21.15.26:43822) 2022-07-20T02:35:20.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 -- 172.21.15.26:0/1090509164 learned_addr learned my addr 172.21.15.26:0/1090509164 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe759e700 1 --2- 172.21.15.26:0/1090509164 >> v2:172.21.15.120:3300/0 conn(0x7fbfe8140320 0x7fbfe81407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 -- 172.21.15.26:0/1090509164 >> v2:172.21.15.120:3300/0 conn(0x7fbfe8140320 msgr2=0x7fbfe81407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 --2- 172.21.15.26:0/1090509164 >> v2:172.21.15.120:3300/0 conn(0x7fbfe8140320 0x7fbfe81407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 -- 172.21.15.26:0/1090509164 >> v1:172.21.15.120:6789/0 conn(0x7fbfe80b5d50 legacy=0x7fbfe8067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.042+0000 7fbfe659c700 1 -- 172.21.15.26:0/1090509164 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8146c90 con 0x7fbfe8143070 2022-07-20T02:35:20.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfe659c700 1 --2- 172.21.15.26:0/1090509164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe81434c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fbfd0002f70 tx=0x7fbfd0006eb0).ready entity=mon.0 client_cookie=24c27137a024c98c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfe5d9b700 1 -- 172.21.15.26:0/1090509164 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd000d5c0 con 0x7fbfe8143070 2022-07-20T02:35:20.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfe5d9b700 1 -- 172.21.15.26:0/1090509164 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0000e90 con 0x7fbfe8143070 2022-07-20T02:35:20.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfe5d9b700 1 -- 172.21.15.26:0/1090509164 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd000d950 con 0x7fbfe8143070 2022-07-20T02:35:20.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1090509164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 msgr2=0x7fbfe81434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1090509164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe81434c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fbfd0002f70 tx=0x7fbfd0006eb0).stop 2022-07-20T02:35:20.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1090509164 shutdown_connections 2022-07-20T02:35:20.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1090509164 >> v2:172.21.15.120:3300/0 conn(0x7fbfe8140320 0x7fbfe81407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1090509164 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe81434c0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1090509164 >> 172.21.15.26:0/1090509164 conn(0x7fbfe8060310 msgr2=0x7fbfe80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1090509164 shutdown_connections 2022-07-20T02:35:20.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1090509164 wait complete. 2022-07-20T02:35:20.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 Processor -- start 2022-07-20T02:35:20.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.046+0000 7fbfec9d4700 1 -- start start 2022-07-20T02:35:20.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfe80b5d50 0x7fbfe80b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfe8140320 0x7fbfe8156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe80a98f0 con 0x7fbfe8143070 2022-07-20T02:35:20.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8155df0 con 0x7fbfe80b5d50 2022-07-20T02:35:20.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8156c00 con 0x7fbfe8140320 2022-07-20T02:35:20.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43828/0 (socket says 172.21.15.26:43828) 2022-07-20T02:35:20.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 -- 172.21.15.26:0/1523952518 learned_addr learned my addr 172.21.15.26:0/1523952518 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe6d9d700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfe80b5d50 0x7fbfe80b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe659c700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfe8140320 0x7fbfe8156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 -- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfe8140320 msgr2=0x7fbfe8156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfe8140320 0x7fbfe8156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 -- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfe80b5d50 msgr2=0x7fbfe80b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfe80b5d50 0x7fbfe80b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 -- 172.21.15.26:0/1523952518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd000b1e0 con 0x7fbfe8143070 2022-07-20T02:35:20.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfe759e700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fbfe0006e00 tx=0x7fbfe00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfe000c640 con 0x7fbfe8143070 2022-07-20T02:35:20.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000c7a0 con 0x7fbfe8143070 2022-07-20T02:35:20.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe80a9b20 con 0x7fbfe8143070 2022-07-20T02:35:20.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfe000cb00 con 0x7fbfe8143070 2022-07-20T02:35:20.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe80aa790 con 0x7fbfe8143070 2022-07-20T02:35:20.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbfe000cc60 con 0x7fbfe8143070 2022-07-20T02:35:20.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe8146f30 con 0x7fbfe8143070 2022-07-20T02:35:20.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.050+0000 7fbfdffff700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 0x7fbfcc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.054+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbfe006c2e0 con 0x7fbfe8143070 2022-07-20T02:35:20.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.054+0000 7fbfe6d9d700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 0x7fbfcc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.054+0000 7fbfe6d9d700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 0x7fbfcc055ea0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fbfd800d000 tx=0x7fbfd8009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.058+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfe0043040 con 0x7fbfe8143070 2022-07-20T02:35:20.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fbfe80574e0 con 0x7fbfe8143070 2022-07-20T02:35:20.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfdffff700 1 -- 172.21.15.26:0/1523952518 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v150) v1 ==== 110+0+10 (secure 0 0 0) 0x7fbfe0046660 con 0x7fbfe8143070 2022-07-20T02:35:20.203 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:20.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 msgr2=0x7fbfcc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 0x7fbfcc055ea0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fbfd800d000 tx=0x7fbfd8009040).stop 2022-07-20T02:35:20.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 msgr2=0x7fbfe8156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fbfe0006e00 tx=0x7fbfe00091d0).stop 2022-07-20T02:35:20.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 shutdown_connections 2022-07-20T02:35:20.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfe8143070 0x7fbfe8156710 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbfcc0539e0 0x7fbfcc055ea0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbfe8140320 0x7fbfe8156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 --2- 172.21.15.26:0/1523952518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbfe80b5d50 0x7fbfe80b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 >> 172.21.15.26:0/1523952518 conn(0x7fbfe8060310 msgr2=0x7fbfe80b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 shutdown_connections 2022-07-20T02:35:20.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.190+0000 7fbfec9d4700 1 -- 172.21.15.26:0/1523952518 wait complete. 2022-07-20T02:35:20.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:20.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.374+0000 7f17cf391700 1 Processor -- start 2022-07-20T02:35:20.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.374+0000 7f17cf391700 1 -- start start 2022-07-20T02:35:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17c80b9c80 0x7f17c81431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c81403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80ba060 con 0x7f17c8062710 2022-07-20T02:35:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c812bee0 con 0x7f17c80b9c80 2022-07-20T02:35:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8061f80 con 0x7f17c80b5b70 2022-07-20T02:35:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17c80b9c80 0x7f17c81431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd92e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c81403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f17c80b9c80 0x7f17c81431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43832/0 (socket says 172.21.15.26:43832) 2022-07-20T02:35:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 -- 172.21.15.26:0/92240129 learned_addr learned my addr 172.21.15.26:0/92240129 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd92e700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c81403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37120/0 (socket says 172.21.15.26:37120) 2022-07-20T02:35:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 -- 172.21.15.26:0/92240129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 msgr2=0x7f17c81403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:20.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c81403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 -- 172.21.15.26:0/92240129 >> v1:172.21.15.120:6789/0 conn(0x7f17c8062710 legacy=0x7f17c80b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17bffff700 1 -- 172.21.15.26:0/92240129 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074305466 0 0) 0x7f17c80ba060 con 0x7f17c8062710 2022-07-20T02:35:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 -- 172.21.15.26:0/92240129 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8062220 con 0x7f17c80b9c80 2022-07-20T02:35:20.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cd12d700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c81431a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f17b8002f70 tx=0x7f17b8006eb0).ready entity=mon.0 client_cookie=1bbb062538355570 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17bffff700 1 -- 172.21.15.26:0/92240129 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b800d570 con 0x7f17c80b9c80 2022-07-20T02:35:20.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17bffff700 1 -- 172.21.15.26:0/92240129 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b8000e90 con 0x7f17c80b9c80 2022-07-20T02:35:20.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17bffff700 1 -- 172.21.15.26:0/92240129 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b800d8c0 con 0x7f17c80b9c80 2022-07-20T02:35:20.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- 172.21.15.26:0/92240129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 msgr2=0x7f17c81431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c81431a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f17b8002f70 tx=0x7f17b8006eb0).stop 2022-07-20T02:35:20.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- 172.21.15.26:0/92240129 shutdown_connections 2022-07-20T02:35:20.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c81403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 --2- 172.21.15.26:0/92240129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c81431a0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.378+0000 7f17cf391700 1 -- 172.21.15.26:0/92240129 >> 172.21.15.26:0/92240129 conn(0x7f17c8066f70 msgr2=0x7f17c805faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- 172.21.15.26:0/92240129 shutdown_connections 2022-07-20T02:35:20.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- 172.21.15.26:0/92240129 wait complete. 2022-07-20T02:35:20.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 Processor -- start 2022-07-20T02:35:20.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- start start 2022-07-20T02:35:20.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c8062710 0x7f17c805cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c80b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80b2ce0 con 0x7f17c80b9c80 2022-07-20T02:35:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81559a0 con 0x7f17c8062710 2022-07-20T02:35:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cf391700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8155bd0 con 0x7f17c80b5b70 2022-07-20T02:35:20.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43840/0 (socket says 172.21.15.26:43840) 2022-07-20T02:35:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 -- 172.21.15.26:0/1375114616 learned_addr learned my addr 172.21.15.26:0/1375114616 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.483 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd12d700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c8062710 0x7f17c805cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cc92c700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c80b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 -- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 msgr2=0x7f17c80b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c80b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 -- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c8062710 msgr2=0x7f17c805cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c8062710 0x7f17c805cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.382+0000 7f17cd92e700 1 -- 172.21.15.26:0/1375114616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b800b1e0 con 0x7f17c80b9c80 2022-07-20T02:35:20.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cd92e700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f17c0006e00 tx=0x7f17c00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c000c640 con 0x7f17c80b9c80 2022-07-20T02:35:20.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c000c7a0 con 0x7f17c80b9c80 2022-07-20T02:35:20.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8155e70 con 0x7f17c80b9c80 2022-07-20T02:35:20.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c000cb00 con 0x7f17c80b9c80 2022-07-20T02:35:20.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81565b0 con 0x7f17c80b9c80 2022-07-20T02:35:20.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f17c0022070 con 0x7f17c80b9c80 2022-07-20T02:35:20.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80b6650 con 0x7f17c80b9c80 2022-07-20T02:35:20.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 0x7f17b0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f17c006c6b0 con 0x7f17c80b9c80 2022-07-20T02:35:20.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cd12d700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 0x7f17b0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.386+0000 7f17cd12d700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 0x7f17b0055e40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f17b801c000 tx=0x7f17b8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.394+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17c00433d0 con 0x7f17c80b9c80 2022-07-20T02:35:20.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.518+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f17c80574e0 con 0x7f17c80b9c80 2022-07-20T02:35:20.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.518+0000 7f17be7fc700 1 -- 172.21.15.26:0/1375114616 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v150) v1 ==== 92+0+10 (secure 0 0 0) 0x7f17c00469f0 con 0x7f17c80b9c80 2022-07-20T02:35:20.531 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:20.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 msgr2=0x7f17b0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 0x7f17b0055e40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f17b801c000 tx=0x7f17b8015000).stop 2022-07-20T02:35:20.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 msgr2=0x7f17c80b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f17c0006e00 tx=0x7f17c00091d0).stop 2022-07-20T02:35:20.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 shutdown_connections 2022-07-20T02:35:20.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f17c80b9c80 0x7f17c80b0b70 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f17b0053980 0x7f17b0055e40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17c80b5b70 0x7f17c80b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 --2- 172.21.15.26:0/1375114616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c8062710 0x7f17c805cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 >> 172.21.15.26:0/1375114616 conn(0x7f17c8066f70 msgr2=0x7f17c80b83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 shutdown_connections 2022-07-20T02:35:20.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.522+0000 7f17cf391700 1 -- 172.21.15.26:0/1375114616 wait complete. 2022-07-20T02:35:20.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:35:20.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.662+0000 7fed63e5e700 1 Processor -- start 2022-07-20T02:35:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.662+0000 7fed63e5e700 1 -- start start 2022-07-20T02:35:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 0x7fed5c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5c147a00 con 0x7fed5c146e80 2022-07-20T02:35:20.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5c1435b0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c05d3b0 con 0x7fed5c0b1160 2022-07-20T02:35:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37132/0 (socket says 172.21.15.26:37132) 2022-07-20T02:35:20.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 -- 172.21.15.26:0/2047262467 learned_addr learned my addr 172.21.15.26:0/2047262467 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed61bfa700 1 --2- 172.21.15.26:0/2047262467 >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 0x7fed5c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 -- 172.21.15.26:0/2047262467 >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 msgr2=0x7fed5c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:20.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 --2- 172.21.15.26:0/2047262467 >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 0x7fed5c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 -- 172.21.15.26:0/2047262467 >> v1:172.21.15.120:6789/0 conn(0x7fed5c146e80 legacy=0x7fed5c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed60bf8700 1 -- 172.21.15.26:0/2047262467 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694507913 0 0) 0x7fed5c147a00 con 0x7fed5c146e80 2022-07-20T02:35:20.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 -- 172.21.15.26:0/2047262467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0b4bc0 con 0x7fed5c0b1160 2022-07-20T02:35:20.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed61bfa700 1 --2- 172.21.15.26:0/2047262467 >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 0x7fed5c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:20.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed613f9700 1 --2- 172.21.15.26:0/2047262467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fed58002f70 tx=0x7fed58006eb0).ready entity=mon.2 client_cookie=dedeb3fa627cab02 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed60bf8700 1 -- 172.21.15.26:0/2047262467 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed5800d570 con 0x7fed5c0b1160 2022-07-20T02:35:20.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed60bf8700 1 -- 172.21.15.26:0/2047262467 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed58000e90 con 0x7fed5c0b1160 2022-07-20T02:35:20.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed60bf8700 1 -- 172.21.15.26:0/2047262467 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed5800d8c0 con 0x7fed5c0b1160 2022-07-20T02:35:20.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- 172.21.15.26:0/2047262467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 msgr2=0x7fed5c140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 --2- 172.21.15.26:0/2047262467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fed58002f70 tx=0x7fed58006eb0).stop 2022-07-20T02:35:20.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- 172.21.15.26:0/2047262467 shutdown_connections 2022-07-20T02:35:20.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 --2- 172.21.15.26:0/2047262467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c0b1160 0x7fed5c140cb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 --2- 172.21.15.26:0/2047262467 >> v2:172.21.15.120:3300/0 conn(0x7fed5c05cfd0 0x7fed5c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- 172.21.15.26:0/2047262467 >> 172.21.15.26:0/2047262467 conn(0x7fed5c066f70 msgr2=0x7fed5c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- 172.21.15.26:0/2047262467 shutdown_connections 2022-07-20T02:35:20.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.666+0000 7fed63e5e700 1 -- 172.21.15.26:0/2047262467 wait complete. 2022-07-20T02:35:20.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 Processor -- start 2022-07-20T02:35:20.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 -- start start 2022-07-20T02:35:20.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c0b1160 0x7fed5c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 0x7fed5c1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c1393b0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c138530 con 0x7fed5c0b1160 2022-07-20T02:35:20.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.670+0000 7fed63e5e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c1387d0 con 0x7fed5c146e80 2022-07-20T02:35:20.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed623fb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 0x7fed5c1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43852/0 (socket says 172.21.15.26:43852) 2022-07-20T02:35:20.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed623fb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 0x7fed5c1562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37140/0 (socket says 172.21.15.26:37140) 2022-07-20T02:35:20.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 -- 172.21.15.26:0/3090831043 learned_addr learned my addr 172.21.15.26:0/3090831043 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:20.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed613f9700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c0b1160 0x7fed5c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 -- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 msgr2=0x7fed5c1562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 0x7fed5c1562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 -- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c0b1160 msgr2=0x7fed5c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c0b1160 0x7fed5c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 -- 172.21.15.26:0/3090831043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c00b180 con 0x7fed5c05cfd0 2022-07-20T02:35:20.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed61bfa700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fed4c00c000 tx=0x7fed4c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed4c00c7e0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c00c940 con 0x7fed5c05cfd0 2022-07-20T02:35:20.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5800b1e0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed4c00cca0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5c138a70 con 0x7fed5c05cfd0 2022-07-20T02:35:20.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c0574e0 con 0x7fed5c05cfd0 2022-07-20T02:35:20.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fed4c01c460 con 0x7fed5c05cfd0 2022-07-20T02:35:20.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.674+0000 7fed4affd700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 0x7fed3c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:20.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.678+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fed4c06c600 con 0x7fed5c05cfd0 2022-07-20T02:35:20.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.678+0000 7fed613f9700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 0x7fed3c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.678+0000 7fed613f9700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 0x7fed3c055ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fed5801c000 tx=0x7fed58015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.682+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed4c043360 con 0x7fed5c05cfd0 2022-07-20T02:35:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.806+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed5c05db80 con 0x7fed5c05cfd0 2022-07-20T02:35:20.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.806+0000 7fed4affd700 1 -- 172.21.15.26:0/3090831043 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+10 (secure 0 0 0) 0x7fed4c00ce00 con 0x7fed5c05cfd0 2022-07-20T02:35:20.861 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:20.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 msgr2=0x7fed3c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 0x7fed3c055ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fed5801c000 tx=0x7fed58015000).stop 2022-07-20T02:35:20.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 msgr2=0x7fed5c13c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:20.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fed4c00c000 tx=0x7fed4c006df0).stop 2022-07-20T02:35:20.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 shutdown_connections 2022-07-20T02:35:20.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fed3c0539e0 0x7fed3c055ea0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fed5c146e80 0x7fed5c1562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c0b1160 0x7fed5c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 --2- 172.21.15.26:0/3090831043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed5c05cfd0 0x7fed5c13c0f0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:20.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 >> 172.21.15.26:0/3090831043 conn(0x7fed5c066f70 msgr2=0x7fed5c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:20.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 shutdown_connections 2022-07-20T02:35:20.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:20.810+0000 7fed63e5e700 1 -- 172.21.15.26:0/3090831043 wait complete. 2022-07-20T02:35:20.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:35:21.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.062+0000 7f66b08e5700 1 Processor -- start 2022-07-20T02:35:21.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.062+0000 7f66b08e5700 1 -- start start 2022-07-20T02:35:21.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66ac0afe40 0x7f66ac0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66ac0b1200 0x7f66ac0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac05d4d0 con 0x7f66ac05c720 2022-07-20T02:35:21.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0a9ff0 con 0x7f66ac0b1200 2022-07-20T02:35:21.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac067e60 con 0x7f66ac0afe40 2022-07-20T02:35:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66ac0b1200 0x7f66ac0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66ab7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f66ac05c720 0x7f66ac147550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43140/0 (socket says 172.21.15.26:43140) 2022-07-20T02:35:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66ac0b1200 0x7f66ac0ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43864/0 (socket says 172.21.15.26:43864) 2022-07-20T02:35:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66ab7fe700 1 -- 172.21.15.26:0/3799034148 learned_addr learned my addr 172.21.15.26:0/3799034148 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66aaffd700 1 --2- 172.21.15.26:0/3799034148 >> v2:172.21.15.120:3300/0 conn(0x7f66ac0afe40 0x7f66ac0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 -- 172.21.15.26:0/3799034148 >> v2:172.21.15.120:3300/0 conn(0x7f66ac0afe40 msgr2=0x7f66ac0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:21.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 --2- 172.21.15.26:0/3799034148 >> v2:172.21.15.120:3300/0 conn(0x7f66ac0afe40 0x7f66ac0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 -- 172.21.15.26:0/3799034148 >> v1:172.21.15.26:6789/0 conn(0x7f66ac05c720 legacy=0x7f66ac147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 -- 172.21.15.26:0/3799034148 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac067bc0 con 0x7f66ac0b1200 2022-07-20T02:35:21.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66abfff700 1 --2- 172.21.15.26:0/3799034148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac0b1200 0x7f66ac0ac0e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f66a4002f70 tx=0x7f66a4006df0).ready entity=mon.0 client_cookie=87f911a642096300 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66aa7fc700 1 -- 172.21.15.26:0/3799034148 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a400d570 con 0x7f66ac0b1200 2022-07-20T02:35:21.457 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66aa7fc700 1 -- 172.21.15.26:0/3799034148 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a4000e90 con 0x7f66ac0b1200 2022-07-20T02:35:21.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66aa7fc700 1 -- 172.21.15.26:0/3799034148 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a400d8c0 con 0x7f66ac0b1200 2022-07-20T02:35:21.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- 172.21.15.26:0/3799034148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac0b1200 msgr2=0x7f66ac0ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 --2- 172.21.15.26:0/3799034148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac0b1200 0x7f66ac0ac0e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f66a4002f70 tx=0x7f66a4006df0).stop 2022-07-20T02:35:21.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- 172.21.15.26:0/3799034148 shutdown_connections 2022-07-20T02:35:21.459 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 --2- 172.21.15.26:0/3799034148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac0b1200 0x7f66ac0ac0e0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 --2- 172.21.15.26:0/3799034148 >> v2:172.21.15.120:3300/0 conn(0x7f66ac0afe40 0x7f66ac0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.460 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- 172.21.15.26:0/3799034148 >> 172.21.15.26:0/3799034148 conn(0x7f66ac060310 msgr2=0x7f66ac0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:21.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- 172.21.15.26:0/3799034148 shutdown_connections 2022-07-20T02:35:21.506 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.066+0000 7f66b08e5700 1 -- 172.21.15.26:0/3799034148 wait complete. 2022-07-20T02:35:21.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 Processor -- start 2022-07-20T02:35:21.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 -- start start 2022-07-20T02:35:21.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66ac0afe40 0x7f66ac062910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 0x7f66ac066dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac065a40 con 0x7f66ac05c720 2022-07-20T02:35:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac065ff0 con 0x7f66ac0b1200 2022-07-20T02:35:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66b08e5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac1571c0 con 0x7f66ac0afe40 2022-07-20T02:35:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66ab7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66ab7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43866/0 (socket says 172.21.15.26:43866) 2022-07-20T02:35:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66aaffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66ac0afe40 0x7f66ac062910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66ab7fe700 1 -- 172.21.15.26:0/349304502 learned_addr learned my addr 172.21.15.26:0/349304502 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66abfff700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 0x7f66ac066dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.070+0000 7f66ab7fe700 1 -- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66ac0afe40 msgr2=0x7f66ac062910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66ab7fe700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66ac0afe40 0x7f66ac062910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66ab7fe700 1 -- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 msgr2=0x7f66ac066dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66ab7fe700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 0x7f66ac066dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66ab7fe700 1 -- 172.21.15.26:0/349304502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a400b180 con 0x7f66ac05c720 2022-07-20T02:35:21.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66abfff700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 0x7f66ac066dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:21.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66ab7fe700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f66ac05d220 tx=0x7f66a0007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a00070d0 con 0x7f66ac05c720 2022-07-20T02:35:21.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0007230 con 0x7f66ac05c720 2022-07-20T02:35:21.626 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac157460 con 0x7f66ac05c720 2022-07-20T02:35:21.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a0005610 con 0x7f66ac05c720 2022-07-20T02:35:21.627 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac062e00 con 0x7f66ac05c720 2022-07-20T02:35:21.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac0aa250 con 0x7f66ac05c720 2022-07-20T02:35:21.628 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f66a0010070 con 0x7f66ac05c720 2022-07-20T02:35:21.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.074+0000 7f66a8ff9700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 0x7f668c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.629 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.078+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f66a006b730 con 0x7f66ac05c720 2022-07-20T02:35:21.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.078+0000 7f66aaffd700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 0x7f668c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.078+0000 7f66aaffd700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 0x7f668c055ea0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f669c00b000 tx=0x7f669c0091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.082+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66a00423e0 con 0x7f66ac05c720 2022-07-20T02:35:21.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.210+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66ac0574e0 con 0x7f66ac05c720 2022-07-20T02:35:21.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.210+0000 7f66a8ff9700 1 -- 172.21.15.26:0/349304502 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+10 (secure 0 0 0) 0x7f66a0045a00 con 0x7f66ac05c720 2022-07-20T02:35:21.644 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 msgr2=0x7f668c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 0x7f668c055ea0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f669c00b000 tx=0x7f669c0091d0).stop 2022-07-20T02:35:21.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 msgr2=0x7f66ac065550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f66ac05d220 tx=0x7f66a0007c40).stop 2022-07-20T02:35:21.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 shutdown_connections 2022-07-20T02:35:21.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f668c0539e0 0x7f668c055ea0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66ac0b1200 0x7f66ac066dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66ac0afe40 0x7f66ac062910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 --2- 172.21.15.26:0/349304502 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66ac05c720 0x7f66ac065550 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 >> 172.21.15.26:0/349304502 conn(0x7f66ac060310 msgr2=0x7f66ac149630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:21.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 shutdown_connections 2022-07-20T02:35:21.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.214+0000 7f66b08e5700 1 -- 172.21.15.26:0/349304502 wait complete. 2022-07-20T02:35:21.669 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:35:21.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:21.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 Processor -- start 2022-07-20T02:35:21.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- start start 2022-07-20T02:35:21.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68b40670c0 0x7f68b40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68b40b3380 0x7f68b414b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4064880 con 0x7f68b4064450 2022-07-20T02:35:21.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b40b85f0 con 0x7f68b40670c0 2022-07-20T02:35:21.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b40b80b0 con 0x7f68b40b3380 2022-07-20T02:35:21.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68b40670c0 0x7f68b40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f68b40670c0 0x7f68b40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43874/0 (socket says 172.21.15.26:43874) 2022-07-20T02:35:21.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 -- 172.21.15.26:0/2130638123 learned_addr learned my addr 172.21.15.26:0/2130638123 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:21.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b8d42700 1 --2- 172.21.15.26:0/2130638123 >> v2:172.21.15.120:3300/0 conn(0x7f68b40b3380 0x7f68b414b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 -- 172.21.15.26:0/2130638123 >> v2:172.21.15.120:3300/0 conn(0x7f68b40b3380 msgr2=0x7f68b414b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:21.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 --2- 172.21.15.26:0/2130638123 >> v2:172.21.15.120:3300/0 conn(0x7f68b40b3380 0x7f68b414b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 -- 172.21.15.26:0/2130638123 >> v1:172.21.15.120:6789/0 conn(0x7f68b4064450 legacy=0x7f68b4064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 -- 172.21.15.26:0/2130638123 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40b8350 con 0x7f68b40670c0 2022-07-20T02:35:21.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b3fff700 1 --2- 172.21.15.26:0/2130638123 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40670c0 0x7f68b40b2e90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f68a4002f70 tx=0x7f68a4006eb0).ready entity=mon.0 client_cookie=491c2aa58bf855ee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b37fe700 1 -- 172.21.15.26:0/2130638123 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a400d5c0 con 0x7f68b40670c0 2022-07-20T02:35:21.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b37fe700 1 -- 172.21.15.26:0/2130638123 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a4000e90 con 0x7f68b40670c0 2022-07-20T02:35:21.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b37fe700 1 -- 172.21.15.26:0/2130638123 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a400d950 con 0x7f68b40670c0 2022-07-20T02:35:21.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- 172.21.15.26:0/2130638123 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40670c0 msgr2=0x7f68b40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 --2- 172.21.15.26:0/2130638123 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40670c0 0x7f68b40b2e90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f68a4002f70 tx=0x7f68a4006eb0).stop 2022-07-20T02:35:21.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- 172.21.15.26:0/2130638123 shutdown_connections 2022-07-20T02:35:21.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 --2- 172.21.15.26:0/2130638123 >> v2:172.21.15.120:3300/0 conn(0x7f68b40b3380 0x7f68b414b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 --2- 172.21.15.26:0/2130638123 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40670c0 0x7f68b40b2e90 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- 172.21.15.26:0/2130638123 >> 172.21.15.26:0/2130638123 conn(0x7f68b405f840 msgr2=0x7f68b405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:21.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- 172.21.15.26:0/2130638123 shutdown_connections 2022-07-20T02:35:21.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.778+0000 7f68b9543700 1 -- 172.21.15.26:0/2130638123 wait complete. 2022-07-20T02:35:21.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 Processor -- start 2022-07-20T02:35:21.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- start start 2022-07-20T02:35:21.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b40670c0 0x7f68b4061d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 0x7f68b40b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b5300 con 0x7f68b40ab470 2022-07-20T02:35:21.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40ab1a0 con 0x7f68b40670c0 2022-07-20T02:35:21.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b5530 con 0x7f68b40b6b20 2022-07-20T02:35:21.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43884/0 (socket says 172.21.15.26:43884) 2022-07-20T02:35:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 -- 172.21.15.26:0/1882922056 learned_addr learned my addr 172.21.15.26:0/1882922056 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:21.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b3fff700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b40670c0 0x7f68b4061d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68abfff700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 0x7f68b40b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 -- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 msgr2=0x7f68b40b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 0x7f68b40b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 -- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b40670c0 msgr2=0x7f68b4061d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b40670c0 0x7f68b4061d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 -- 172.21.15.26:0/1882922056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a400b1e0 con 0x7f68b40ab470 2022-07-20T02:35:21.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68abfff700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 0x7f68b40b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:21.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b8d42700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f68ac006e00 tx=0x7f68ac0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68ac00c670 con 0x7f68b40ab470 2022-07-20T02:35:21.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00c7d0 con 0x7f68b40ab470 2022-07-20T02:35:21.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b40b6790 con 0x7f68b40ab470 2022-07-20T02:35:21.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68ac00cb30 con 0x7f68b40ab470 2022-07-20T02:35:21.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4152690 con 0x7f68b40ab470 2022-07-20T02:35:21.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.782+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4064ab0 con 0x7f68b40ab470 2022-07-20T02:35:21.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.786+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f68ac00cc90 con 0x7f68b40ab470 2022-07-20T02:35:21.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.786+0000 7f68b1ffb700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 0x7f6898055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:21.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.786+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f68ac06c2f0 con 0x7f68b40ab470 2022-07-20T02:35:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.786+0000 7f68b3fff700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 0x7f6898055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.786+0000 7f68b3fff700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 0x7f6898055f30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f68a401c000 tx=0x7f68a4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.794+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68ac043050 con 0x7f68b40ab470 2022-07-20T02:35:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68b40574e0 con 0x7f68b40ab470 2022-07-20T02:35:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b1ffb700 1 -- 172.21.15.26:0/1882922056 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2486 (secure 0 0 0) 0x7f68ac046670 con 0x7f68b40ab470 2022-07-20T02:35:21.975 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:21.976 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:16.734420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8505":{"gid":8505,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/853590136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":853590136},{"type":"v1","addr":"172.21.15.26:6837","nonce":853590136}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 msgr2=0x7f6898055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 0x7f6898055f30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f68a401c000 tx=0x7f68a4015000).stop 2022-07-20T02:35:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 msgr2=0x7f68b413d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f68ac006e00 tx=0x7f68ac0091d0).stop 2022-07-20T02:35:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 shutdown_connections 2022-07-20T02:35:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f68b40ab470 0x7f68b413d880 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6898053a70 0x7f6898055f30 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b40b6b20 0x7f68b40b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 --2- 172.21.15.26:0/1882922056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b40670c0 0x7f68b4061d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 >> 172.21.15.26:0/1882922056 conn(0x7f68b405f840 msgr2=0x7f68b405ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 shutdown_connections 2022-07-20T02:35:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:21.958+0000 7f68b9543700 1 -- 172.21.15.26:0/1882922056 wait complete. 2022-07-20T02:35:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 80 2022-07-20T02:35:22.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:14.691126+0000', 'modified': '2022-07-20T02:35:16.734420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8505': {'gid': 8505, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/853590136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 853590136}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 853590136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:22.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:22.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:22.443 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8505,"want_state":"up:active","state":"up:active","rank_uptime":7.7317534209999996,"mdsmap_epoch":80,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":12.131394197000001} 2022-07-20T02:35:22.445 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8505, "mdsmap_epoch": 80, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 7.731753421, "state": "up:active", "uptime": 12.131394197, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:22.446 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:35:22.446 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:35:22.448 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:35:22.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:22.450 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:22.482 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:35:22.515 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:22.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:22.607 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:22.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:22.608 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:35:22.658 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:35:22.660 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:22.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:35:22.818 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:22.819 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:22.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:35:22.788+0000 7f82eb679080 -1 init, newargv = 0x558b81055850 newargc=9 2022-07-20T02:35:22.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23466]: starting ceph client 2022-07-20T02:35:22.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23466]: starting fuse 2022-07-20T02:35:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:22.964 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:22.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:22.965 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:35:22.985 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:35:22.986 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:35:22.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:22.989 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:35:22.991 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:35:22.991 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:35:22.992 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:35:22.993 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:35:23.010 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:23.011 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:35:23.012 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:35:23.013 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:35:23.019 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:23.022 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:35:23.023 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:35:23.024 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:35:23.042 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:35:23.044 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:23.045 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:35:23.046 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:35:23.047 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:35:23.047 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:35:23.048 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:35:23.049 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:23.050 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:35:23.051 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:35:23.102 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23466.asok 2022-07-20T02:35:23.117 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23466.asok 2022-07-20T02:35:23.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:23.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23466.asok status 2022-07-20T02:35:23.250 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:35:23.251 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:35:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:35:23.252 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:35:23.253 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:35:23.253 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:35:23.254 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:35:23.265 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23466", 2022-07-20T02:35:23.265 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:35:23.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:23.266 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:35:23.267 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:35:23.267 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 7295, 2022-07-20T02:35:23.267 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:35:23.268 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:35:23.268 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:35:23.269 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 7295 2022-07-20T02:35:23.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:23.271 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:35:23.271 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:35:23.272 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:35:23.272 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2054357578 2022-07-20T02:35:23.272 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:35:23.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:35:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:35:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:35:23.304 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 2054357578 2022-07-20T02:35:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.7295 172.21.15.159:0/2054357578", 2022-07-20T02:35:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/2054357578", 2022-07-20T02:35:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:35:23.305 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 80, 2022-07-20T02:35:23.316 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 150, 2022-07-20T02:35:23.326 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:35:23.326 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:35:23.327 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:35:23.327 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:35:23.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:23.330 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:23.330 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:23.364 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:35:23.365 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:35:23.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:23.446 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:23.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:23.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.634+0000 7fdd606f2700 1 Processor -- start 2022-07-20T02:35:23.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.634+0000 7fdd606f2700 1 -- start start 2022-07-20T02:35:23.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd581484a0 0x7fdd58066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:23.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd580b1bc0 con 0x7fdd58061050 2022-07-20T02:35:23.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd58147700 con 0x7fdd580b2920 2022-07-20T02:35:23.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5814b040 con 0x7fdd581484a0 2022-07-20T02:35:23.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5ec8f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdd58061050 0x7fdd58066f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43172/0 (socket says 172.21.15.26:43172) 2022-07-20T02:35:23.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5ec8f700 1 -- 172.21.15.26:0/2276286154 learned_addr learned my addr 172.21.15.26:0/2276286154 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:23.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5dc8d700 1 --2- 172.21.15.26:0/2276286154 >> v2:172.21.15.120:3300/0 conn(0x7fdd581484a0 0x7fdd58066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:23.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5d48c700 1 -- 172.21.15.26:0/2276286154 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271135172 0 0) 0x7fdd580b1bc0 con 0x7fdd58061050 2022-07-20T02:35:23.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5d48c700 1 -- 172.21.15.26:0/2276286154 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40003260 con 0x7fdd58061050 2022-07-20T02:35:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2276286154 >> v1:172.21.15.120:6789/0 conn(0x7fdd580b2920 legacy=0x7fdd58147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2276286154 >> v1:172.21.15.26:6789/0 conn(0x7fdd58061050 legacy=0x7fdd58066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2276286154 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58147930 con 0x7fdd581484a0 2022-07-20T02:35:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5dc8d700 1 --2- 172.21.15.26:0/2276286154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd581484a0 0x7fdd58066a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fdd54002f70 tx=0x7fdd54006df0).ready entity=mon.1 client_cookie=498ad8e74f25d0f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:23.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5d48c700 1 -- 172.21.15.26:0/2276286154 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd5400d610 con 0x7fdd581484a0 2022-07-20T02:35:23.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5d48c700 1 -- 172.21.15.26:0/2276286154 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd54000e90 con 0x7fdd581484a0 2022-07-20T02:35:23.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd5d48c700 1 -- 172.21.15.26:0/2276286154 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd5400d960 con 0x7fdd581484a0 2022-07-20T02:35:23.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- 172.21.15.26:0/2276286154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd581484a0 msgr2=0x7fdd58066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2276286154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd581484a0 0x7fdd58066a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fdd54002f70 tx=0x7fdd54006df0).stop 2022-07-20T02:35:23.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- 172.21.15.26:0/2276286154 shutdown_connections 2022-07-20T02:35:23.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2276286154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd581484a0 0x7fdd58066a80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- 172.21.15.26:0/2276286154 >> 172.21.15.26:0/2276286154 conn(0x7fdd580b60a0 msgr2=0x7fdd580b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:23.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- 172.21.15.26:0/2276286154 shutdown_connections 2022-07-20T02:35:23.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.638+0000 7fdd606f2700 1 -- 172.21.15.26:0/2276286154 wait complete. 2022-07-20T02:35:23.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 Processor -- start 2022-07-20T02:35:23.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- start start 2022-07-20T02:35:23.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd58061050 0x7fdd5814a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:23.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:23.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd581484a0 0x7fdd5815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:23.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5815b0a0 con 0x7fdd580b2920 2022-07-20T02:35:23.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580ae010 con 0x7fdd58061050 2022-07-20T02:35:23.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580aa210 con 0x7fdd581484a0 2022-07-20T02:35:23.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:23.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43896/0 (socket says 172.21.15.26:43896) 2022-07-20T02:35:23.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2722217042 learned_addr learned my addr 172.21.15.26:0/2722217042 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:23.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5ec8f700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd581484a0 0x7fdd5815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:23.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5e48e700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd58061050 0x7fdd5814a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:23.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd581484a0 msgr2=0x7fdd5815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd581484a0 0x7fdd5815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd58061050 msgr2=0x7fdd5814a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd58061050 0x7fdd5814a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 -- 172.21.15.26:0/2722217042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5400b180 con 0x7fdd580b2920 2022-07-20T02:35:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd5dc8d700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fdd54002f70 tx=0x7fdd54014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd54013de0 con 0x7fdd580b2920 2022-07-20T02:35:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5400d290 con 0x7fdd580b2920 2022-07-20T02:35:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd580aa4b0 con 0x7fdd580b2920 2022-07-20T02:35:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd5401e5f0 con 0x7fdd580b2920 2022-07-20T02:35:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.642+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5815a220 con 0x7fdd580b2920 2022-07-20T02:35:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5815a3e0 con 0x7fdd580b2920 2022-07-20T02:35:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdd54025030 con 0x7fdd580b2920 2022-07-20T02:35:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd477fe700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 0x7fdd38055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdd5406cbb0 con 0x7fdd580b2920 2022-07-20T02:35:23.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd5e48e700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 0x7fdd38055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:23.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.646+0000 7fdd5e48e700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 0x7fdd38055e40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fdd4800d000 tx=0x7fdd48007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:23.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.654+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd54073080 con 0x7fdd580b2920 2022-07-20T02:35:23.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd5814ada0 con 0x7fdd580b2920 2022-07-20T02:35:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd477fe700 1 -- 172.21.15.26:0/2722217042 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16333 (secure 0 0 0) 0x7fdd54023070 con 0x7fdd580b2920 2022-07-20T02:35:23.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:23.842 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":150,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:15.715598+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:13.545762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:14.132638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:23.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 msgr2=0x7fdd38055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 0x7fdd38055e40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fdd4800d000 tx=0x7fdd48007b90).stop 2022-07-20T02:35:23.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 msgr2=0x7fdd5815a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:23.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fdd54002f70 tx=0x7fdd54014000).stop 2022-07-20T02:35:23.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 shutdown_connections 2022-07-20T02:35:23.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdd580b2920 0x7fdd5815a6c0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdd38053980 0x7fdd38055e40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd581484a0 0x7fdd5815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 --2- 172.21.15.26:0/2722217042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd58061050 0x7fdd5814a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:23.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.786+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 >> 172.21.15.26:0/2722217042 conn(0x7fdd580b60a0 msgr2=0x7fdd58145da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:23.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.790+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 shutdown_connections 2022-07-20T02:35:23.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.790+0000 7fdd606f2700 1 -- 172.21.15.26:0/2722217042 wait complete. 2022-07-20T02:35:23.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:35:13.545762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:35:14.132638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:35:23.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:24.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.990+0000 7f02073fa700 1 Processor -- start 2022-07-20T02:35:24.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.990+0000 7f02073fa700 1 -- start start 2022-07-20T02:35:24.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.990+0000 7f02073fa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f02001465b0 0x7f0200066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.990+0000 7f02073fa700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02000b1ae0 con 0x7f02000672a0 2022-07-20T02:35:24.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02000b1d80 con 0x7f02000b24c0 2022-07-20T02:35:24.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000b2e20 con 0x7f02001465b0 2022-07-20T02:35:24.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f0206bf9700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f02000672a0 0x7f0200146990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43186/0 (socket says 172.21.15.26:43186) 2022-07-20T02:35:24.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f0206bf9700 1 -- 172.21.15.26:0/2987596939 learned_addr learned my addr 172.21.15.26:0/2987596939 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:24.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02063f8700 1 --2- 172.21.15.26:0/2987596939 >> v2:172.21.15.26:3300/0 conn(0x7f02001465b0 0x7f0200066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881914258 0 0) 0x7f02000b1ae0 con 0x7f02000672a0 2022-07-20T02:35:24.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e8003260 con 0x7f02000672a0 2022-07-20T02:35:24.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3125539432 0 0) 0x7f01e8003260 con 0x7f02000672a0 2022-07-20T02:35:24.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02063f8700 1 -- 172.21.15.26:0/2987596939 >> v1:172.21.15.120:6789/0 conn(0x7f02000b24c0 legacy=0x7f02000604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02063f8700 1 -- 172.21.15.26:0/2987596939 >> v1:172.21.15.26:6789/0 conn(0x7f02000672a0 legacy=0x7f0200146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02063f8700 1 -- 172.21.15.26:0/2987596939 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000b2020 con 0x7f02001465b0 2022-07-20T02:35:24.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02063f8700 1 --2- 172.21.15.26:0/2987596939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02001465b0 0x7f0200066db0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f01f8001b80 tx=0x7f01fc006df0).ready entity=mon.0 client_cookie=35bd35fa5b518455 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01f80019e0 con 0x7f02001465b0 2022-07-20T02:35:24.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc000e90 con 0x7f02001465b0 2022-07-20T02:35:24.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02053f6700 1 -- 172.21.15.26:0/2987596939 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc011400 con 0x7f02001465b0 2022-07-20T02:35:24.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- 172.21.15.26:0/2987596939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02001465b0 msgr2=0x7f0200066db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 --2- 172.21.15.26:0/2987596939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02001465b0 0x7f0200066db0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f01f8001b80 tx=0x7f01fc006df0).stop 2022-07-20T02:35:24.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- 172.21.15.26:0/2987596939 shutdown_connections 2022-07-20T02:35:24.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 --2- 172.21.15.26:0/2987596939 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f02001465b0 0x7f0200066db0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- 172.21.15.26:0/2987596939 >> 172.21.15.26:0/2987596939 conn(0x7f02000b3050 msgr2=0x7f02000b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- 172.21.15.26:0/2987596939 shutdown_connections 2022-07-20T02:35:24.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 -- 172.21.15.26:0/2987596939 wait complete. 2022-07-20T02:35:24.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.994+0000 7f02073fa700 1 Processor -- start 2022-07-20T02:35:24.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- start start 2022-07-20T02:35:24.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02000672a0 0x7f02001458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02001465b0 0x7f0200155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200150160 con 0x7f0200066260 2022-07-20T02:35:24.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200156890 con 0x7f02000672a0 2022-07-20T02:35:24.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02000aa3e0 con 0x7f02001465b0 2022-07-20T02:35:24.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43912/0 (socket says 172.21.15.26:43912) 2022-07-20T02:35:24.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 -- 172.21.15.26:0/2894615110 learned_addr learned my addr 172.21.15.26:0/2894615110 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02063f8700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02000672a0 0x7f02001458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0205bf7700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02001465b0 0x7f0200155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 -- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02001465b0 msgr2=0x7f0200155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02001465b0 0x7f0200155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 -- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02000672a0 msgr2=0x7f02001458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02000672a0 0x7f02001458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 -- 172.21.15.26:0/2894615110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc00b180 con 0x7f0200066260 2022-07-20T02:35:24.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f0206bf9700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f01f8001bc0 tx=0x7f01f80021e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01f8004ab0 con 0x7f0200066260 2022-07-20T02:35:24.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8004c10 con 0x7f0200066260 2022-07-20T02:35:24.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02000aa680 con 0x7f0200066260 2022-07-20T02:35:24.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01f8007170 con 0x7f0200066260 2022-07-20T02:35:24.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0200155a10 con 0x7f0200066260 2022-07-20T02:35:24.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:23.998+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0200155bd0 con 0x7f0200066260 2022-07-20T02:35:24.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.002+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f01f80072d0 con 0x7f0200066260 2022-07-20T02:35:24.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.002+0000 7f01ef7fe700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 0x7f01e0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.002+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f01f806c4b0 con 0x7f0200066260 2022-07-20T02:35:24.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.002+0000 7f02063f8700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 0x7f01e0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.002+0000 7f02063f8700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 0x7f01e0055f60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f01fc019000 tx=0x7f02000b1c70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.006+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01f8043100 con 0x7f0200066260 2022-07-20T02:35:24.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.130+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0200060bf0 con 0x7f0200066260 2022-07-20T02:35:24.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.130+0000 7f01ef7fe700 1 -- 172.21.15.26:0/2894615110 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v150) v1 ==== 110+0+10 (secure 0 0 0) 0x7f01f8046720 con 0x7f0200066260 2022-07-20T02:35:24.141 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:24.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 msgr2=0x7f01e0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 0x7f01e0055f60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f01fc019000 tx=0x7f02000b1c70).stop 2022-07-20T02:35:24.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 msgr2=0x7f02001563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f01f8001bc0 tx=0x7f01f80021e0).stop 2022-07-20T02:35:24.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 shutdown_connections 2022-07-20T02:35:24.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0200066260 0x7f02001563a0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f01e0053aa0 0x7f01e0055f60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02001465b0 0x7f0200155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 --2- 172.21.15.26:0/2894615110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02000672a0 0x7f02001458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 >> 172.21.15.26:0/2894615110 conn(0x7f02000b3050 msgr2=0x7f02000b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 shutdown_connections 2022-07-20T02:35:24.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.134+0000 7f02073fa700 1 -- 172.21.15.26:0/2894615110 wait complete. 2022-07-20T02:35:24.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:24.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 Processor -- start 2022-07-20T02:35:24.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 -- start start 2022-07-20T02:35:24.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4ec0afe40 0x7fd4ec0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd4ec0b1200 0x7fd4ec0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec05d4d0 con 0x7fd4ec05c720 2022-07-20T02:35:24.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec0a9ff0 con 0x7fd4ec0b1200 2022-07-20T02:35:24.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f1571700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec067e60 con 0x7fd4ec0afe40 2022-07-20T02:35:24.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd4ec0b1200 0x7fd4ec0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4ebfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd4ec05c720 0x7fd4ec147550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43196/0 (socket says 172.21.15.26:43196) 2022-07-20T02:35:24.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4ebfff700 1 -- 172.21.15.26:0/3993039510 learned_addr learned my addr 172.21.15.26:0/3993039510 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:24.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4eb7fe700 1 --2- 172.21.15.26:0/3993039510 >> v2:172.21.15.120:3300/0 conn(0x7fd4ec0afe40 0x7fd4ec0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 -- 172.21.15.26:0/3993039510 >> v2:172.21.15.120:3300/0 conn(0x7fd4ec0afe40 msgr2=0x7fd4ec0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:24.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 --2- 172.21.15.26:0/3993039510 >> v2:172.21.15.120:3300/0 conn(0x7fd4ec0afe40 0x7fd4ec0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 -- 172.21.15.26:0/3993039510 >> v1:172.21.15.26:6789/0 conn(0x7fd4ec05c720 legacy=0x7fd4ec147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 -- 172.21.15.26:0/3993039510 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec067bc0 con 0x7fd4ec0b1200 2022-07-20T02:35:24.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4eaffd700 1 -- 172.21.15.26:0/3993039510 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251189966 0 0) 0x7fd4ec05d4d0 con 0x7fd4ec05c720 2022-07-20T02:35:24.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.270+0000 7fd4f0d70700 1 --2- 172.21.15.26:0/3993039510 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0b1200 0x7fd4ec0ac0e0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7fd4e4002f70 tx=0x7fd4e4006df0).ready entity=mon.0 client_cookie=69cf4e672b0d138d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4eaffd700 1 -- 172.21.15.26:0/3993039510 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4e400d570 con 0x7fd4ec0b1200 2022-07-20T02:35:24.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4eaffd700 1 -- 172.21.15.26:0/3993039510 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e4000e90 con 0x7fd4ec0b1200 2022-07-20T02:35:24.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4eaffd700 1 -- 172.21.15.26:0/3993039510 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4e400d8c0 con 0x7fd4ec0b1200 2022-07-20T02:35:24.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- 172.21.15.26:0/3993039510 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0b1200 msgr2=0x7fd4ec0ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3993039510 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0b1200 0x7fd4ec0ac0e0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7fd4e4002f70 tx=0x7fd4e4006df0).stop 2022-07-20T02:35:24.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- 172.21.15.26:0/3993039510 shutdown_connections 2022-07-20T02:35:24.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3993039510 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0b1200 0x7fd4ec0ac0e0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3993039510 >> v2:172.21.15.120:3300/0 conn(0x7fd4ec0afe40 0x7fd4ec0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- 172.21.15.26:0/3993039510 >> 172.21.15.26:0/3993039510 conn(0x7fd4ec060310 msgr2=0x7fd4ec0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- 172.21.15.26:0/3993039510 shutdown_connections 2022-07-20T02:35:24.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- 172.21.15.26:0/3993039510 wait complete. 2022-07-20T02:35:24.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 Processor -- start 2022-07-20T02:35:24.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.274+0000 7fd4f1571700 1 -- start start 2022-07-20T02:35:24.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ec05c720 0x7fd4ec0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 0x7fd4ec157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec14d830 con 0x7fd4ec0afe40 2022-07-20T02:35:24.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec159ac0 con 0x7fd4ec05c720 2022-07-20T02:35:24.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec142b80 con 0x7fd4ec0b1200 2022-07-20T02:35:24.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43924/0 (socket says 172.21.15.26:43924) 2022-07-20T02:35:24.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 -- 172.21.15.26:0/3153851002 learned_addr learned my addr 172.21.15.26:0/3153851002 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4ebfff700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ec05c720 0x7fd4ec0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f0d70700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 0x7fd4ec157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 -- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 msgr2=0x7fd4ec157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 0x7fd4ec157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 -- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ec05c720 msgr2=0x7fd4ec0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ec05c720 0x7fd4ec0b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 -- 172.21.15.26:0/3153851002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e400b180 con 0x7fd4ec0afe40 2022-07-20T02:35:24.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f0d70700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 0x7fd4ec157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:24.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4eb7fe700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7fd4ec0ab080 tx=0x7fd4dc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4dc00c920 con 0x7fd4ec0afe40 2022-07-20T02:35:24.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00ca80 con 0x7fd4ec0afe40 2022-07-20T02:35:24.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec142e20 con 0x7fd4ec0afe40 2022-07-20T02:35:24.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec0b3950 con 0x7fd4ec0afe40 2022-07-20T02:35:24.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4dc00cde0 con 0x7fd4ec0afe40 2022-07-20T02:35:24.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec0aa250 con 0x7fd4ec0afe40 2022-07-20T02:35:24.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.278+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd4dc022070 con 0x7fd4ec0afe40 2022-07-20T02:35:24.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.282+0000 7fd4e97fa700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 0x7fd4cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.282+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd4dc06c9f0 con 0x7fd4ec0afe40 2022-07-20T02:35:24.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.282+0000 7fd4ebfff700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 0x7fd4cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.282+0000 7fd4ebfff700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 0x7fd4cc055fc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fd4e000b000 tx=0x7fd4e0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.286+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4dc06ee60 con 0x7fd4ec0afe40 2022-07-20T02:35:24.427 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd4ec0574e0 con 0x7fd4ec0afe40 2022-07-20T02:35:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4e97fa700 1 -- 172.21.15.26:0/3153851002 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v150) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd4dc043710 con 0x7fd4ec0afe40 2022-07-20T02:35:24.438 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:24.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 msgr2=0x7fd4cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 0x7fd4cc055fc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fd4e000b000 tx=0x7fd4e0009040).stop 2022-07-20T02:35:24.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 msgr2=0x7fd4ec156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7fd4ec0ab080 tx=0x7fd4dc0091d0).stop 2022-07-20T02:35:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 shutdown_connections 2022-07-20T02:35:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd4ec0afe40 0x7fd4ec156c10 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd4cc053b00 0x7fd4cc055fc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ec0b1200 0x7fd4ec157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 --2- 172.21.15.26:0/3153851002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ec05c720 0x7fd4ec0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 >> 172.21.15.26:0/3153851002 conn(0x7fd4ec060310 msgr2=0x7fd4ec147930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 shutdown_connections 2022-07-20T02:35:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.414+0000 7fd4f1571700 1 -- 172.21.15.26:0/3153851002 wait complete. 2022-07-20T02:35:24.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:35:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 Processor -- start 2022-07-20T02:35:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 -- start start 2022-07-20T02:35:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ce0060f10 0x7f0ce00b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00b2f10 con 0x7f0ce00b2b30 2022-07-20T02:35:24.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00a8e30 con 0x7f0ce0060f10 2022-07-20T02:35:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce6aff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00a7310 con 0x7f0ce0147d70 2022-07-20T02:35:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0ce509c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ce0060f10 0x7f0ce00b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37214/0 (socket says 172.21.15.26:37214) 2022-07-20T02:35:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 -- 172.21.15.26:0/3949377350 learned_addr learned my addr 172.21.15.26:0/3949377350 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 -- 172.21.15.26:0/3949377350 >> v2:172.21.15.120:3300/0 conn(0x7f0ce0060f10 msgr2=0x7f0ce00b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 --2- 172.21.15.26:0/3949377350 >> v2:172.21.15.120:3300/0 conn(0x7f0ce0060f10 0x7f0ce00b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 -- 172.21.15.26:0/3949377350 >> v1:172.21.15.120:6789/0 conn(0x7f0ce00b2b30 legacy=0x7f0ce0147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 -- 172.21.15.26:0/3949377350 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00a7500 con 0x7f0ce0147d70 2022-07-20T02:35:24.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdffff700 1 --2- 172.21.15.26:0/3949377350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0cd0002f70 tx=0x7f0cd0006eb0).ready entity=mon.2 client_cookie=b5f5c1c35b33d4d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdf7fe700 1 -- 172.21.15.26:0/3949377350 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd000d570 con 0x7f0ce0147d70 2022-07-20T02:35:24.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdf7fe700 1 -- 172.21.15.26:0/3949377350 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0000e90 con 0x7f0ce0147d70 2022-07-20T02:35:24.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.634+0000 7f0cdf7fe700 1 -- 172.21.15.26:0/3949377350 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd000d900 con 0x7f0ce0147d70 2022-07-20T02:35:24.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- 172.21.15.26:0/3949377350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 msgr2=0x7f0ce00a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/3949377350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0cd0002f70 tx=0x7f0cd0006eb0).stop 2022-07-20T02:35:24.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- 172.21.15.26:0/3949377350 shutdown_connections 2022-07-20T02:35:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/3949377350 >> v2:172.21.15.120:3300/0 conn(0x7f0ce0060f10 0x7f0ce00b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/3949377350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0147d70 0x7f0ce00a7ee0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- 172.21.15.26:0/3949377350 >> 172.21.15.26:0/3949377350 conn(0x7f0ce00b6110 msgr2=0x7f0ce00b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- 172.21.15.26:0/3949377350 shutdown_connections 2022-07-20T02:35:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- 172.21.15.26:0/3949377350 wait complete. 2022-07-20T02:35:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 Processor -- start 2022-07-20T02:35:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- start start 2022-07-20T02:35:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0060f10 0x7f0ce014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 0x7f0ce014dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 0x7f0ce0156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0149350 con 0x7f0ce0147d70 2022-07-20T02:35:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce014d2c0 con 0x7f0ce00b2b30 2022-07-20T02:35:24.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce6aff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce014d560 con 0x7f0ce0060f10 2022-07-20T02:35:24.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 0x7f0ce0156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0cdffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 0x7f0ce014dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce489b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0060f10 0x7f0ce014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0cdffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 0x7f0ce014dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35396/0 (socket says 172.21.15.26:35396) 2022-07-20T02:35:24.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0cdffff700 1 -- 172.21.15.26:0/4282286885 learned_addr learned my addr 172.21.15.26:0/4282286885 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 -- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0060f10 msgr2=0x7f0ce014c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0060f10 0x7f0ce014c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 -- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 msgr2=0x7f0ce014dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 0x7f0ce014dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.638+0000 7f0ce509c700 1 -- 172.21.15.26:0/4282286885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd000b1e0 con 0x7f0ce0147d70 2022-07-20T02:35:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0ce509c700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 0x7f0ce0156880 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f0cd800c000 tx=0x7f0cd80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd8012600 con 0x7f0ce0147d70 2022-07-20T02:35:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8012760 con 0x7f0ce0147d70 2022-07-20T02:35:24.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce014d800 con 0x7f0ce0147d70 2022-07-20T02:35:24.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd8012ac0 con 0x7f0ce0147d70 2022-07-20T02:35:24.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce0063660 con 0x7f0ce0147d70 2022-07-20T02:35:24.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0cd8022070 con 0x7f0ce0147d70 2022-07-20T02:35:24.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce00a77a0 con 0x7f0ce0147d70 2022-07-20T02:35:24.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 0x7f0cc0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0cd806c710 con 0x7f0ce0147d70 2022-07-20T02:35:24.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.642+0000 7f0ce489b700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 0x7f0cc0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.646+0000 7f0ce489b700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 0x7f0cc0055e10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f0cd400d000 tx=0x7f0cd4007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.654+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cd80433c0 con 0x7f0ce0147d70 2022-07-20T02:35:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.778+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ce00574e0 con 0x7f0ce0147d70 2022-07-20T02:35:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.782+0000 7f0cddffb700 1 -- 172.21.15.26:0/4282286885 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0cd8016020 con 0x7f0ce0147d70 2022-07-20T02:35:24.810 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 msgr2=0x7f0cc0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 0x7f0cc0055e10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f0cd400d000 tx=0x7f0cd4007b90).stop 2022-07-20T02:35:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 msgr2=0x7f0ce0156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 0x7f0ce0156880 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f0cd800c000 tx=0x7f0cd80091d0).stop 2022-07-20T02:35:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 shutdown_connections 2022-07-20T02:35:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ce0147d70 0x7f0ce0156880 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0cc0053950 0x7f0cc0055e10 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce00b2b30 0x7f0ce014dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 --2- 172.21.15.26:0/4282286885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ce0060f10 0x7f0ce014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 >> 172.21.15.26:0/4282286885 conn(0x7f0ce00b6110 msgr2=0x7f0ce00b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 shutdown_connections 2022-07-20T02:35:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.786+0000 7f0ce6aff700 1 -- 172.21.15.26:0/4282286885 wait complete. 2022-07-20T02:35:24.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:35:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 Processor -- start 2022-07-20T02:35:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- start start 2022-07-20T02:35:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faf38140320 0x7faf38140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faf38147020 0x7faf3805c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf380ab530 con 0x7faf3805ca70 2022-07-20T02:35:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf380b1ce0 con 0x7faf38140320 2022-07-20T02:35:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf380aab50 con 0x7faf38147020 2022-07-20T02:35:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faf38140320 0x7faf38140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7faf38140320 0x7faf38140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43944/0 (socket says 172.21.15.26:43944) 2022-07-20T02:35:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 -- 172.21.15.26:0/1385893248 learned_addr learned my addr 172.21.15.26:0/1385893248 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf2effd700 1 --2- 172.21.15.26:0/1385893248 >> v2:172.21.15.120:3300/0 conn(0x7faf38147020 0x7faf3805c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 -- 172.21.15.26:0/1385893248 >> v2:172.21.15.120:3300/0 conn(0x7faf38147020 msgr2=0x7faf3805c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 --2- 172.21.15.26:0/1385893248 >> v2:172.21.15.120:3300/0 conn(0x7faf38147020 0x7faf3805c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 -- 172.21.15.26:0/1385893248 >> v1:172.21.15.120:6789/0 conn(0x7faf3805ca70 legacy=0x7faf38143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 -- 172.21.15.26:0/1385893248 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf380aadf0 con 0x7faf38140320 2022-07-20T02:35:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf377fe700 1 --2- 172.21.15.26:0/1385893248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf38140700 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7faf28002f70 tx=0x7faf28006eb0).ready entity=mon.0 client_cookie=e1a6a235ff303f35 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf36ffd700 1 -- 172.21.15.26:0/1385893248 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf2800d610 con 0x7faf38140320 2022-07-20T02:35:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf36ffd700 1 -- 172.21.15.26:0/1385893248 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28000e90 con 0x7faf38140320 2022-07-20T02:35:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf36ffd700 1 -- 172.21.15.26:0/1385893248 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf2800d9a0 con 0x7faf38140320 2022-07-20T02:35:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- 172.21.15.26:0/1385893248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 msgr2=0x7faf38140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1385893248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf38140700 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7faf28002f70 tx=0x7faf28006eb0).stop 2022-07-20T02:35:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- 172.21.15.26:0/1385893248 shutdown_connections 2022-07-20T02:35:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1385893248 >> v2:172.21.15.120:3300/0 conn(0x7faf38147020 0x7faf3805c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1385893248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf38140700 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- 172.21.15.26:0/1385893248 >> 172.21.15.26:0/1385893248 conn(0x7faf38060310 msgr2=0x7faf380b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- 172.21.15.26:0/1385893248 shutdown_connections 2022-07-20T02:35:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.938+0000 7faf3cb30700 1 -- 172.21.15.26:0/1385893248 wait complete. 2022-07-20T02:35:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 Processor -- start 2022-07-20T02:35:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 -- start start 2022-07-20T02:35:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf38147020 0x7faf38155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 0x7faf38152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf380aed00 con 0x7faf38140320 2022-07-20T02:35:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf380aefa0 con 0x7faf38147020 2022-07-20T02:35:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf381525b0 con 0x7faf380ae580 2022-07-20T02:35:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43950/0 (socket says 172.21.15.26:43950) 2022-07-20T02:35:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 -- 172.21.15.26:0/1298125056 learned_addr learned my addr 172.21.15.26:0/1298125056 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf2effd700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf38147020 0x7faf38155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf37fff700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 0x7faf38152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 -- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 msgr2=0x7faf38152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 0x7faf38152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 -- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf38147020 msgr2=0x7faf38155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf38147020 0x7faf38155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 -- 172.21.15.26:0/1298125056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2800b1e0 con 0x7faf38140320 2022-07-20T02:35:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf37fff700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 0x7faf38152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:35:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf377fe700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7faf28002f70 tx=0x7faf28015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf2800d990 con 0x7faf38140320 2022-07-20T02:35:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28011ca0 con 0x7faf38140320 2022-07-20T02:35:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.942+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38152850 con 0x7faf38140320 2022-07-20T02:35:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf2801e460 con 0x7faf38140320 2022-07-20T02:35:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf380ae1a0 con 0x7faf38140320 2022-07-20T02:35:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf38153380 con 0x7faf38140320 2022-07-20T02:35:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faf28025030 con 0x7faf38140320 2022-07-20T02:35:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf357fa700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 0x7faf1c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faf2806d320 con 0x7faf38140320 2022-07-20T02:35:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.946+0000 7faf2effd700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 0x7faf1c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.950+0000 7faf2effd700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 0x7faf1c055e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7faf2000c000 tx=0x7faf20008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:24.954+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf280440e0 con 0x7faf38140320 2022-07-20T02:35:25.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.078+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf381529e0 con 0x7faf38140320 2022-07-20T02:35:25.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf357fa700 1 -- 172.21.15.26:0/1298125056 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+10 (secure 0 0 0) 0x7faf28023020 con 0x7faf38140320 2022-07-20T02:35:25.086 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:25.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 msgr2=0x7faf1c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 0x7faf1c055e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7faf2000c000 tx=0x7faf20008040).stop 2022-07-20T02:35:25.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 msgr2=0x7faf380b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7faf28002f70 tx=0x7faf28015000).stop 2022-07-20T02:35:25.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 shutdown_connections 2022-07-20T02:35:25.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7faf1c053950 0x7faf1c055e10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faf380ae580 0x7faf38152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf38147020 0x7faf38155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 --2- 172.21.15.26:0/1298125056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf38140320 0x7faf380b4370 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 >> 172.21.15.26:0/1298125056 conn(0x7faf38060310 msgr2=0x7faf380ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:25.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 shutdown_connections 2022-07-20T02:35:25.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.082+0000 7faf3cb30700 1 -- 172.21.15.26:0/1298125056 wait complete. 2022-07-20T02:35:25.127 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:35:25.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.246+0000 7fb502cac700 1 Processor -- start 2022-07-20T02:35:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.246+0000 7fb502cac700 1 -- start start 2022-07-20T02:35:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 0x7fb4fc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 0x7fb4fc060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0aa1e0 con 0x7fb4fc061280 2022-07-20T02:35:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc0ae110 con 0x7fb4fc0add30 2022-07-20T02:35:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0a9f40 con 0x7fb4fc060950 2022-07-20T02:35:25.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb501caa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 0x7fb4fc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 0x7fb4fc060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5024ab700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb4fc061280 0x7fb4fc143450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:57454/0 (socket says 172.21.15.26:57454) 2022-07-20T02:35:25.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5024ab700 1 -- 172.21.15.26:0/1881491879 learned_addr learned my addr 172.21.15.26:0/1881491879 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:25.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb501caa700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 0x7fb4fc0b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43960/0 (socket says 172.21.15.26:43960) 2022-07-20T02:35:25.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb500ca8700 1 -- 172.21.15.26:0/1881491879 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207573058 0 0) 0x7fb4fc0aa1e0 con 0x7fb4fc061280 2022-07-20T02:35:25.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 -- 172.21.15.26:0/1881491879 >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 msgr2=0x7fb4fc0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:25.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 --2- 172.21.15.26:0/1881491879 >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 0x7fb4fc0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 -- 172.21.15.26:0/1881491879 >> v1:172.21.15.120:6789/0 conn(0x7fb4fc061280 legacy=0x7fb4fc143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 -- 172.21.15.26:0/1881491879 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0b1110 con 0x7fb4fc060950 2022-07-20T02:35:25.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb5014a9700 1 --2- 172.21.15.26:0/1881491879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 0x7fb4fc060d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8006df0).ready entity=mon.2 client_cookie=46a0fcc9cbe57dca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:25.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb500ca8700 1 -- 172.21.15.26:0/1881491879 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f800d570 con 0x7fb4fc060950 2022-07-20T02:35:25.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb500ca8700 1 -- 172.21.15.26:0/1881491879 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f8000e90 con 0x7fb4fc060950 2022-07-20T02:35:25.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb500ca8700 1 -- 172.21.15.26:0/1881491879 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f800d8c0 con 0x7fb4fc060950 2022-07-20T02:35:25.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- 172.21.15.26:0/1881491879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 msgr2=0x7fb4fc060d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 --2- 172.21.15.26:0/1881491879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 0x7fb4fc060d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8006df0).stop 2022-07-20T02:35:25.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- 172.21.15.26:0/1881491879 shutdown_connections 2022-07-20T02:35:25.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 --2- 172.21.15.26:0/1881491879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc060950 0x7fb4fc060d90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 --2- 172.21.15.26:0/1881491879 >> v2:172.21.15.26:3300/0 conn(0x7fb4fc0add30 0x7fb4fc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- 172.21.15.26:0/1881491879 >> 172.21.15.26:0/1881491879 conn(0x7fb4fc0b3050 msgr2=0x7fb4fc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:25.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- 172.21.15.26:0/1881491879 shutdown_connections 2022-07-20T02:35:25.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.250+0000 7fb502cac700 1 -- 172.21.15.26:0/1881491879 wait complete. 2022-07-20T02:35:25.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 Processor -- start 2022-07-20T02:35:25.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- start start 2022-07-20T02:35:25.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc060950 0x7fb4fc140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc0add30 0x7fb4fc0661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc1404a0 con 0x7fb4fc061280 2022-07-20T02:35:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc1406d0 con 0x7fb4fc060950 2022-07-20T02:35:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc13f110 con 0x7fb4fc0add30 2022-07-20T02:35:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43964/0 (socket says 172.21.15.26:43964) 2022-07-20T02:35:25.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 -- 172.21.15.26:0/2373084619 learned_addr learned my addr 172.21.15.26:0/2373084619 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:25.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5024ab700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc0add30 0x7fb4fc0661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb501caa700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc060950 0x7fb4fc140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 -- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc0add30 msgr2=0x7fb4fc0661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc0add30 0x7fb4fc0661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 -- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc060950 msgr2=0x7fb4fc140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc060950 0x7fb4fc140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 -- 172.21.15.26:0/2373084619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f800b180 con 0x7fb4fc061280 2022-07-20T02:35:25.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb5014a9700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:25.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f8016070 con 0x7fb4fc061280 2022-07-20T02:35:25.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80139c0 con 0x7fb4fc061280 2022-07-20T02:35:25.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc13f3b0 con 0x7fb4fc061280 2022-07-20T02:35:25.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f8013d20 con 0x7fb4fc061280 2022-07-20T02:35:25.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.254+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc140050 con 0x7fb4fc061280 2022-07-20T02:35:25.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc145770 con 0x7fb4fc061280 2022-07-20T02:35:25.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb4f8025030 con 0x7fb4fc061280 2022-07-20T02:35:25.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb4eeffd700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:25.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb4f806cf80 con 0x7fb4fc061280 2022-07-20T02:35:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb501caa700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.258+0000 7fb501caa700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb4f000b000 tx=0x7fb4f0006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.262+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4f8043c30 con 0x7fb4fc061280 2022-07-20T02:35:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4fc0b13b0 con 0x7fb4fc061280 2022-07-20T02:35:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb4eeffd700 1 -- 172.21.15.26:0/2373084619 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2486 (secure 0 0 0) 0x7fb4f8023070 con 0x7fb4fc061280 2022-07-20T02:35:25.447 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:25.448 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:16.734420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8505":{"gid":8505,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/853590136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":853590136},{"type":"v1","addr":"172.21.15.26:6837","nonce":853590136}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:25.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 msgr2=0x7fb4dc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb4f000b000 tx=0x7fb4f0006df0).stop 2022-07-20T02:35:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 msgr2=0x7fb4fc065cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8014000).stop 2022-07-20T02:35:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 shutdown_connections 2022-07-20T02:35:25.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4fc061280 0x7fb4fc065cf0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb4dc053950 0x7fb4dc055e10 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.454 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb4fc0add30 0x7fb4fc0661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 --2- 172.21.15.26:0/2373084619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4fc060950 0x7fb4fc140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:25.455 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 >> 172.21.15.26:0/2373084619 conn(0x7fb4fc0b3050 msgr2=0x7fb4fc0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:25.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 shutdown_connections 2022-07-20T02:35:25.456 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:25.426+0000 7fb502cac700 1 -- 172.21.15.26:0/2373084619 wait complete. 2022-07-20T02:35:25.457 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 80 2022-07-20T02:35:25.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:14.691126+0000', 'modified': '2022-07-20T02:35:16.734420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8505': {'gid': 8505, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/853590136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6836', 'nonce': 853590136}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 853590136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:25.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:25.793 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:25.942 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8505,"want_state":"up:active","state":"up:active","rank_uptime":11.214407638000001,"mdsmap_epoch":80,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":15.614040922999999} 2022-07-20T02:35:25.943 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8505, "mdsmap_epoch": 80, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 11.214407638, "state": "up:active", "uptime": 15.614040923, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:25.944 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:35:25.944 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:35:25.945 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:35:25.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:25.963 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:25.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:35:26.001 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:26.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:26.070 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:26.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.071 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:35:26.131 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:35:26.132 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:26.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:35:26.199 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:26.199 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:26.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:35:26.207+0000 7f720dd8e080 -1 init, newargv = 0x5619dd4f6830 newargc=9ceph-fuse[ 2022-07-20T02:35:26.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:23119]: starting ceph client 2022-07-20T02:35:26.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23119]: starting fuse 2022-07-20T02:35:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:26.231 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:26.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.232 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:35:26.296 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:35:26.297 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:35:26.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:35:26.298 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:35:26.299 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:35:26.299 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:35:26.300 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:35:26.301 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:26.302 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:35:26.302 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:35:26.303 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:35:26.303 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:26.304 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:35:26.304 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:35:26.305 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:35:26.305 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:35:26.315 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:26.335 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:35:26.336 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:35:26.338 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:35:26.339 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:35:26.340 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:35:26.341 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:26.343 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:35:26.343 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:35:26.390 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.23119.asok 2022-07-20T02:35:26.408 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23119.asok 2022-07-20T02:35:26.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23119.asok status 2022-07-20T02:35:26.556 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:35:26.556 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:35:26.557 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:35:26.557 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:35:26.562 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:35:26.563 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:35:26.563 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:35:26.563 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "23119", 2022-07-20T02:35:26.564 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:35:26.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:26.564 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:35:26.564 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:35:26.565 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 8446, 2022-07-20T02:35:26.565 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:35:26.565 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:35:26.565 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:35:26.565 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 8446 2022-07-20T02:35:26.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:26.566 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:35:26.566 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:35:26.566 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:35:26.567 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 1449789372 2022-07-20T02:35:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:35:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:35:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:35:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:35:26.569 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 1449789372 2022-07-20T02:35:26.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:26.569 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.8446 172.21.15.176:0/1449789372", 2022-07-20T02:35:26.569 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/1449789372", 2022-07-20T02:35:26.570 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:35:26.570 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 80, 2022-07-20T02:35:26.570 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 150, 2022-07-20T02:35:26.570 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:35:26.571 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:35:26.571 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:35:26.571 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:35:26.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.572 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:26.572 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:26.631 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:35:26.631 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:26.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.640 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:26.671 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T02:35:26.672 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.rctime . 2022-07-20T02:35:26.725 INFO:teuthology.orchestra.run.smithi159.stdout:1658284526.665885451 2022-07-20T02:35:26.738 INFO:tasks.cephfs.test_misc:rctime = 1658284526.665885451 2022-07-20T02:35:26.739 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:35:26.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:26.790 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:26.820 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:26.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:26.960 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:35:26.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:26.961 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:26.962 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:26.975 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:35:26.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:35:26.976 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi159.front.sepia.ceph.com... 2022-07-20T02:35:26.976 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:35:26.977 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:27.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23466]: fuse finished with error 0 and tester_r 0 2022-07-20T02:35:27.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:27.035 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:27.036 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:27.055 INFO:teuthology.orchestra.run.smithi159.stdout:ext2/ext3 2022-07-20T02:35:27.055 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:35:27.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:35:27.176 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:27.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:27.216 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:27.217 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:27.251 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:35:27.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:27.252 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:35:27.253 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:35:27.253 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:27.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23119]: fuse finished with error 0 and tester_r 0 2022-07-20T02:35:27.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:27.321 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:27.385 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:27.403 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:35:27.405 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:35:27.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:35:27.434 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:27.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:35:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 Processor -- start 2022-07-20T02:35:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- start start 2022-07-20T02:35:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f499c0add30 0x7f499c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f499c0af6d0 0x7f499c0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0aa1e0 con 0x7f499c060950 2022-07-20T02:35:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499c0ae110 con 0x7f499c0add30 2022-07-20T02:35:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499c0a9f40 con 0x7f499c0af6d0 2022-07-20T02:35:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f499c0add30 0x7f499c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1614700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f499c060950 0x7f499c060dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43252/0 (socket says 172.21.15.26:43252) 2022-07-20T02:35:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1614700 1 -- 172.21.15.26:0/97161666 learned_addr learned my addr 172.21.15.26:0/97161666 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f499c0add30 0x7f499c0b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43974/0 (socket says 172.21.15.26:43974) 2022-07-20T02:35:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2616700 1 --2- 172.21.15.26:0/97161666 >> v2:172.21.15.120:3300/0 conn(0x7f499c0af6d0 0x7f499c0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a0e13700 1 -- 172.21.15.26:0/97161666 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673386198 0 0) 0x7f499c0aa1e0 con 0x7f499c060950 2022-07-20T02:35:27.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 -- 172.21.15.26:0/97161666 >> v2:172.21.15.120:3300/0 conn(0x7f499c0af6d0 msgr2=0x7f499c0afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:27.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 --2- 172.21.15.26:0/97161666 >> v2:172.21.15.120:3300/0 conn(0x7f499c0af6d0 0x7f499c0afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:27.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 -- 172.21.15.26:0/97161666 >> v1:172.21.15.26:6789/0 conn(0x7f499c060950 legacy=0x7f499c060dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:27.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 -- 172.21.15.26:0/97161666 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c0b1110 con 0x7f499c0add30 2022-07-20T02:35:27.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a1e15700 1 --2- 172.21.15.26:0/97161666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c0b2ab0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998006df0).ready entity=mon.0 client_cookie=538327bd4140d18a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:27.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a0e13700 1 -- 172.21.15.26:0/97161666 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499800d570 con 0x7f499c0add30 2022-07-20T02:35:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a0e13700 1 -- 172.21.15.26:0/97161666 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4998000e90 con 0x7f499c0add30 2022-07-20T02:35:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a0e13700 1 -- 172.21.15.26:0/97161666 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f499800d8c0 con 0x7f499c0add30 2022-07-20T02:35:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- 172.21.15.26:0/97161666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 msgr2=0x7f499c0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:27.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 --2- 172.21.15.26:0/97161666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c0b2ab0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f4998002f70 tx=0x7f4998006df0).stop 2022-07-20T02:35:27.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- 172.21.15.26:0/97161666 shutdown_connections 2022-07-20T02:35:27.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 --2- 172.21.15.26:0/97161666 >> v2:172.21.15.120:3300/0 conn(0x7f499c0af6d0 0x7f499c0afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:27.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 --2- 172.21.15.26:0/97161666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c0b2ab0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:27.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- 172.21.15.26:0/97161666 >> 172.21.15.26:0/97161666 conn(0x7f499c0b3050 msgr2=0x7f499c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- 172.21.15.26:0/97161666 shutdown_connections 2022-07-20T02:35:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.590+0000 7f49a2e17700 1 -- 172.21.15.26:0/97161666 wait complete. 2022-07-20T02:35:27.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 Processor -- start 2022-07-20T02:35:27.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 -- start start 2022-07-20T02:35:27.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 0x7f499c05c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0af6d0 0x7f499c1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0aac80 con 0x7f499c0add30 2022-07-20T02:35:27.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0639f0 con 0x7f499c0af6d0 2022-07-20T02:35:27.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2e17700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c154170 con 0x7f499c060950 2022-07-20T02:35:27.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43980/0 (socket says 172.21.15.26:43980) 2022-07-20T02:35:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 -- 172.21.15.26:0/2405325604 learned_addr learned my addr 172.21.15.26:0/2405325604 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a2616700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0af6d0 0x7f499c1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1e15700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 0x7f499c05c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 -- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 msgr2=0x7f499c05c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:27.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 0x7f499c05c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:27.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 -- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0af6d0 msgr2=0x7f499c1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:27.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0af6d0 0x7f499c1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:27.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1614700 1 -- 172.21.15.26:0/2405325604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499800b180 con 0x7f499c0add30 2022-07-20T02:35:27.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.594+0000 7f49a1e15700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 0x7f499c05c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f49a1614700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f498c000fe0 tx=0x7f498c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f498c00ce40 con 0x7f499c0add30 2022-07-20T02:35:27.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c0072e0 con 0x7f499c0add30 2022-07-20T02:35:27.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c0aa540 con 0x7f499c0add30 2022-07-20T02:35:27.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f498c005800 con 0x7f499c0add30 2022-07-20T02:35:27.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c153e50 con 0x7f499c0add30 2022-07-20T02:35:27.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499c0b13b0 con 0x7f499c0add30 2022-07-20T02:35:27.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f498c01a060 con 0x7f499c0add30 2022-07-20T02:35:27.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 0x7f497c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:27.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f49a1e15700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 0x7f497c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:27.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.598+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f498c06b7f0 con 0x7f499c0add30 2022-07-20T02:35:27.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.602+0000 7f49a1e15700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 0x7f497c055f60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f4998018000 tx=0x7f4998014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:27.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.606+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f498c0425b0 con 0x7f499c0add30 2022-07-20T02:35:27.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.734+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f499c150a40 con 0x7f499c0add30 2022-07-20T02:35:27.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f498affd700 1 -- 172.21.15.26:0/2405325604 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v150) v1 ==== 81+0+34099 (secure 0 0 0) 0x7f498c018070 con 0x7f499c0add30 2022-07-20T02:35:27.967 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:27.970 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:35:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 150, 2022-07-20T02:35:27.971 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:35:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:35:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:35:15.715598+0000", 2022-07-20T02:35:27.972 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:27.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:35:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:35:27.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:35:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:35:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:35:27.984 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:35:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:35:27.985 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:27.986 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:35:27.993 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:35:27.994 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:35:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:35:27.995 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:35:27.996 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 28, 2022-07-20T02:35:27.996 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:35:27.997 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:35:27.999 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:35:27.999 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:35:28.000 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:35:28.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:35:28.013 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:35:28.014 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:35:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:28.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:28.016 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:28.017 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:28.030 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:35:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:35:28.031 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:35:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:35:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:35:28.039 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:28.040 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:28.048 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:28.049 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:28.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:35:28.051 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:28.052 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:28.053 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:28.054 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:28.055 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:28.056 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:28.057 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:28.058 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:28.060 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:28.061 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:28.063 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:28.067 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:28.068 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:28.069 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:28.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.071 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:28.079 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:28.080 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:28.081 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:28.105 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:28.107 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:28.107 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:28.123 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:28.123 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:35:28.124 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:35:28.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.124 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:28.125 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:35:28.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.128 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:35:28.128 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:35:28.129 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:35:28.129 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:28.130 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:28.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:28.131 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:28.131 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:28.132 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:28.132 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:28.133 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:28.133 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:35:28.134 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:35:28.134 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:35:28.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:35:28.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:35:28.136 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:28.136 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:28.137 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:28.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:28.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:28.138 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:28.138 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:28.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.139 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:35:28.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:28.152 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:28.152 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:28.152 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:28.153 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:28.153 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:28.153 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:28.154 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:28.154 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:28.155 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:28.156 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:28.165 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:28.166 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:28.167 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:28.172 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:28.172 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:28.173 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:28.173 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:28.174 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:28.174 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:28.175 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:28.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:28.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.177 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:28.178 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:28.178 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:28.178 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:28.179 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:28.179 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:28.180 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:28.180 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:28.181 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:28.181 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:28.182 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:28.182 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:35:28.183 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:28.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:35:28.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 27, 2022-07-20T02:35:28.186 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:35:28.186 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:35:13.545762+0000", 2022-07-20T02:35:28.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:28.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:28.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:28.188 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:28.189 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:28.189 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:28.197 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:28.198 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:28.198 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:35:28.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:35:28.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:35:28.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:35:28.201 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:35:28.201 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:28.216 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:28.217 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:28.225 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:28.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.227 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "149", 2022-07-20T02:35:28.227 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:28.228 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:28.228 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:28.229 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:28.265 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:28.280 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:28.284 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:28.285 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:28.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:28.304 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:28.306 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:28.312 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:28.313 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:28.313 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:28.314 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:28.334 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:28.335 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:28.335 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:28.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:28.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.337 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:28.337 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:28.345 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:28.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:28.347 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:28.347 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:35:28.384 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:35:28.384 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:35:28.385 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:35:28.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.386 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:28.386 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:35:28.386 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:35:28.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.389 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 28, 2022-07-20T02:35:28.389 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:35:28.389 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:35:14.132638+0000", 2022-07-20T02:35:28.442 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:28.442 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:28.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:28.443 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:28.462 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:28.463 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:28.464 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:28.465 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:28.466 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:35:28.466 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:35:28.466 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:35:28.467 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:35:28.467 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:35:28.468 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:28.468 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:28.469 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:28.469 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:28.469 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:28.482 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:28.483 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:28.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.484 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "150", 2022-07-20T02:35:28.484 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:28.485 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:28.486 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:28.487 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:28.507 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:28.507 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:28.507 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:28.507 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:28.508 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:28.508 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:28.508 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:28.508 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:28.509 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:28.509 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:28.509 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:28.509 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:28.569 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:28.571 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:28.571 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:28.571 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:28.572 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:28.572 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:28.573 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:28.573 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:28.574 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:28.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:28.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.575 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:28.577 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:28.610 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:28.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:28.612 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:28.613 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:28.629 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:28.639 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:35:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:28.640 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:35:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:35:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:35:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:35:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:28.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:28.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:35:28.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:35:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:35:28.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:35:28.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.682 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:28.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:35:28.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:35:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:35:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:35:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:35:28.700 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:35:28.700 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:35:28.701 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:35:28.701 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:28.701 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:28.702 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:28.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:28.703 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:35:28.703 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:28.704 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:28.704 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:28.704 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:28.704 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:28.705 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:28.705 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:28.705 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:28.705 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:28.705 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:28.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:28.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:35:28.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:35:28.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.710 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:28.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:35:28.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:35:28.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.714 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:28.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:35:28.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:35:28.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.744 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:28.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.745 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:35:28.745 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:35:28.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:28.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.748 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:35:28.748 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:35:28.748 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:35:28.748 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:35:28.748 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:28.852 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:35:28.852 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 147, 2022-07-20T02:35:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:35:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:35:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:35:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:35:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:35:28.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:35:28.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.933 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:28.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:28.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:28.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:35:28.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:28.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:35:28.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:28.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:28.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:35:28.950 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:35:28.950 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:35:28.951 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:35:28.951 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:28.952 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:28.952 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:35:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:28.956 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:28.957 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:35:29.046 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:29.046 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:29.047 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:29.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:35:29.049 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:35:29.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.076 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:29.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:35:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:35:29.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.104 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:29.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:35:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:35:29.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.141 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:29.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:35:29.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:35:29.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:35:29.176 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:35:29.176 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:35:29.177 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:35:29.177 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:29.177 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:29.178 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:29.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.180 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:29.180 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:35:29.180 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:29.181 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:29.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:29.182 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:29.182 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:29.183 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:29.183 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:29.184 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 147, 2022-07-20T02:35:29.184 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:29.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:35:29.200 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:35:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.213 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:29.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:35:29.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:35:29.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.253 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:29.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:35:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:35:29.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.267 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:35:29.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:35:29.290 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.306 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:35:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:35:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:35:29.307 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:35:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:29.308 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:29.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.331 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:35:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:29.332 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:29.333 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:29.334 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:29.336 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:29.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:35:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:35:29.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:29.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:35:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:35:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.351 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:35:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:35:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:35:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:35:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:35:29.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:35:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:35:29.371 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:35:29.372 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:29.378 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:29.383 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:29.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.385 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:29.385 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:35:29.385 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:29.386 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:29.386 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:29.386 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:29.386 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:29.387 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:29.387 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:29.387 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:29.388 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:29.388 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:29.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:29.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:35:29.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:35:29.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.392 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.392 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:29.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:35:29.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.395 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:35:29.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.396 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:29.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:35:29.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:35:29.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.408 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:29.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:35:29.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:35:29.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:29.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:35:29.412 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:35:29.413 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:35:29.413 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:35:29.413 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:29.414 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:29.414 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:29.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.417 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:29.417 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:35:29.417 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:29.417 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:29.418 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:29.418 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:29.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:29.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:29.419 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:29.419 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:29.419 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:29.419 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:29.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:29.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:35:29.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:35:29.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.424 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:29.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:35:29.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:35:29.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:35:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:35:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.434 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:29.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:35:29.434 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:35:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:35:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:35:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:35:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:35:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:35:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:35:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.443 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:29.444 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.444 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.445 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.445 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.446 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.446 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:35:29.446 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.447 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:29.447 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.447 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.449 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.449 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.461 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.462 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:35:29.462 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.464 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:29.464 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.465 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.465 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.467 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.467 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.468 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:35:29.468 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.471 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:29.472 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.472 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.473 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.474 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.474 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:35:29.475 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.477 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:29.477 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.478 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.478 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.479 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.488 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:35:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.526 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:35:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:29.528 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:35:29.531 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.532 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:35:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:35:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:35:29.533 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:35:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:35:29.534 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:35:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:35:29.543 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:35:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:35:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:35:29.544 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:35:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:35:29.545 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:35:29.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:35:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:29.554 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:35:29.566 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:35:29.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:29.568 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:35:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:35:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:35:29.569 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:35:29.574 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:35:29.574 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:29.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 msgr2=0x7f497c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 0x7f497c055f60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f4998018000 tx=0x7f4998014000).stop 2022-07-20T02:35:29.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 msgr2=0x7f499c14fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f498c000fe0 tx=0x7f498c007b50).stop 2022-07-20T02:35:29.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 shutdown_connections 2022-07-20T02:35:29.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f499c0add30 0x7f499c14fec0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f497c053aa0 0x7f497c055f60 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f499c0af6d0 0x7f499c1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 --2- 172.21.15.26:0/2405325604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f499c060950 0x7f499c05c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 >> 172.21.15.26:0/2405325604 conn(0x7f499c0b3050 msgr2=0x7f499c139770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:29.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 shutdown_connections 2022-07-20T02:35:29.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:27.738+0000 7f49a2e17700 1 -- 172.21.15.26:0/2405325604 wait complete. 2022-07-20T02:35:29.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.750+0000 7f60bc89d700 1 Processor -- start 2022-07-20T02:35:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.750+0000 7f60bc89d700 1 -- start start 2022-07-20T02:35:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60b8060950 0x7f60b8096220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b8067dc0 con 0x7f60b80a95a0 2022-07-20T02:35:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b80b3640 con 0x7f60b80b3260 2022-07-20T02:35:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b80615c0 con 0x7f60b8060950 2022-07-20T02:35:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b6ffd700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60b80a95a0 0x7f60b8095b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43268/0 (socket says 172.21.15.26:43268) 2022-07-20T02:35:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b6ffd700 1 -- 172.21.15.26:0/3909353584 learned_addr learned my addr 172.21.15.26:0/3909353584 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b7fff700 1 --2- 172.21.15.26:0/3909353584 >> v2:172.21.15.120:3300/0 conn(0x7f60b8060950 0x7f60b8096220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b67fc700 1 -- 172.21.15.26:0/3909353584 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487862790 0 0) 0x7f60b8067dc0 con 0x7f60b80a95a0 2022-07-20T02:35:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b67fc700 1 -- 172.21.15.26:0/3909353584 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a4003260 con 0x7f60b80a95a0 2022-07-20T02:35:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b7fff700 1 -- 172.21.15.26:0/3909353584 >> v1:172.21.15.120:6789/0 conn(0x7f60b80b3260 legacy=0x7f60b80676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b7fff700 1 -- 172.21.15.26:0/3909353584 >> v1:172.21.15.26:6789/0 conn(0x7f60b80a95a0 legacy=0x7f60b8095b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b7fff700 1 -- 172.21.15.26:0/3909353584 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b8061860 con 0x7f60b8060950 2022-07-20T02:35:29.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b7fff700 1 --2- 172.21.15.26:0/3909353584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8096220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f60b00032f0 tx=0x7f60b00091d0).ready entity=mon.1 client_cookie=c33f468b5166ddf1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:29.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b67fc700 1 -- 172.21.15.26:0/3909353584 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60b000b6c0 con 0x7f60b8060950 2022-07-20T02:35:29.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b67fc700 1 -- 172.21.15.26:0/3909353584 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b000b820 con 0x7f60b8060950 2022-07-20T02:35:29.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60b67fc700 1 -- 172.21.15.26:0/3909353584 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60b000b990 con 0x7f60b8060950 2022-07-20T02:35:29.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- 172.21.15.26:0/3909353584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 msgr2=0x7f60b8096220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 --2- 172.21.15.26:0/3909353584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8096220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f60b00032f0 tx=0x7f60b00091d0).stop 2022-07-20T02:35:29.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- 172.21.15.26:0/3909353584 shutdown_connections 2022-07-20T02:35:29.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 --2- 172.21.15.26:0/3909353584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8096220 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- 172.21.15.26:0/3909353584 >> 172.21.15.26:0/3909353584 conn(0x7f60b80b67f0 msgr2=0x7f60b80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:29.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- 172.21.15.26:0/3909353584 shutdown_connections 2022-07-20T02:35:29.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.754+0000 7f60bc89d700 1 -- 172.21.15.26:0/3909353584 wait complete. 2022-07-20T02:35:29.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 Processor -- start 2022-07-20T02:35:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- start start 2022-07-20T02:35:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8066820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b80b3260 0x7f60b81510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8150520 con 0x7f60b80a95a0 2022-07-20T02:35:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80ae9c0 con 0x7f60b8060950 2022-07-20T02:35:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80afe40 con 0x7f60b80b3260 2022-07-20T02:35:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43994/0 (socket says 172.21.15.26:43994) 2022-07-20T02:35:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 -- 172.21.15.26:0/894944304 learned_addr learned my addr 172.21.15.26:0/894944304 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b77fe700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8066820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b7fff700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b80b3260 0x7f60b81510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 -- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b80b3260 msgr2=0x7f60b81510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b80b3260 0x7f60b81510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 -- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 msgr2=0x7f60b8066820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8066820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 -- 172.21.15.26:0/894944304 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0009470 con 0x7f60b80a95a0 2022-07-20T02:35:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b6ffd700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f60b8064dc0 tx=0x7f60ac007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ac0070d0 con 0x7f60b80a95a0 2022-07-20T02:35:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac007230 con 0x7f60b80a95a0 2022-07-20T02:35:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b8150710 con 0x7f60b80a95a0 2022-07-20T02:35:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ac005610 con 0x7f60b80a95a0 2022-07-20T02:35:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.758+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b80661f0 con 0x7f60b80a95a0 2022-07-20T02:35:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b80b0040 con 0x7f60b80a95a0 2022-07-20T02:35:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60ac01a060 con 0x7f60b80a95a0 2022-07-20T02:35:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60b4ff9700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 0x7f6098055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f60ac06b500 con 0x7f60b80a95a0 2022-07-20T02:35:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60b77fe700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 0x7f6098055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.762+0000 7f60b77fe700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 0x7f6098055f30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f60a800d000 tx=0x7f60a8007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.770+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60ac0421b0 con 0x7f60b80a95a0 2022-07-20T02:35:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.934+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60b81508a0 con 0x7f60b80a95a0 2022-07-20T02:35:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60b4ff9700 1 -- 172.21.15.26:0/894944304 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2486 (secure 0 0 0) 0x7f60ac018020 con 0x7f60b80a95a0 2022-07-20T02:35:29.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:29.981 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:16.734420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8505":{"gid":8505,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/853590136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":853590136},{"type":"v1","addr":"172.21.15.26:6837","nonce":853590136}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:30.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 msgr2=0x7f6098055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 0x7f6098055f30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f60a800d000 tx=0x7f60a8007b90).stop 2022-07-20T02:35:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 msgr2=0x7f60b8150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f60b8064dc0 tx=0x7f60ac007b50).stop 2022-07-20T02:35:30.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 shutdown_connections 2022-07-20T02:35:30.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f60b80a95a0 0x7f60b8150bb0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6098053a70 0x7f6098055f30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b80b3260 0x7f60b81510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 --2- 172.21.15.26:0/894944304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b8060950 0x7f60b8066820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 >> 172.21.15.26:0/894944304 conn(0x7f60b80b67f0 msgr2=0x7f60b80abe70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:30.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 shutdown_connections 2022-07-20T02:35:30.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:29.938+0000 7f60bc89d700 1 -- 172.21.15.26:0/894944304 wait complete. 2022-07-20T02:35:30.038 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 80 2022-07-20T02:35:30.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:35:30.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 Processor -- start 2022-07-20T02:35:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 -- start start 2022-07-20T02:35:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb100b2b30 0x7fcb100a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb10060f10 0x7fcb100b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100a8e30 con 0x7fcb10147660 2022-07-20T02:35:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100a7310 con 0x7fcb10060f10 2022-07-20T02:35:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb15245700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10147fe0 con 0x7fcb100b2b30 2022-07-20T02:35:30.191 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb0f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb0e59c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb10060f10 0x7fcb100b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb0f59e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44004/0 (socket says 172.21.15.26:44004) 2022-07-20T02:35:30.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb0f59e700 1 -- 172.21.15.26:0/1790366280 learned_addr learned my addr 172.21.15.26:0/1790366280 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:30.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.158+0000 7fcb0ed9d700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb100b2b30 0x7fcb100a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 -- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb100b2b30 msgr2=0x7fcb100a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb100b2b30 0x7fcb100a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 -- 172.21.15.26:0/1790366280 >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 msgr2=0x7fcb10147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/1790366280 >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 -- 172.21.15.26:0/1790366280 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100b2f10 con 0x7fcb10060f10 2022-07-20T02:35:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0f59e700 1 --2- 172.21.15.26:0/1790366280 >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10060f10 0x7fcb100b1990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fcb000032f0 tx=0x7fcb000091d0).ready entity=mon.1 client_cookie=feeb162cfc5147a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0dd9b700 1 -- 172.21.15.26:0/1790366280 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0000b600 con 0x7fcb10060f10 2022-07-20T02:35:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0dd9b700 1 -- 172.21.15.26:0/1790366280 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0000b760 con 0x7fcb10060f10 2022-07-20T02:35:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb0dd9b700 1 -- 172.21.15.26:0/1790366280 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0000b8d0 con 0x7fcb10060f10 2022-07-20T02:35:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10060f10 msgr2=0x7fcb100b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10060f10 0x7fcb100b1990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fcb000032f0 tx=0x7fcb000091d0).stop 2022-07-20T02:35:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- 172.21.15.26:0/1790366280 shutdown_connections 2022-07-20T02:35:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 --2- 172.21.15.26:0/1790366280 >> v2:172.21.15.26:3300/0 conn(0x7fcb10147660 0x7fcb10147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10060f10 0x7fcb100b1990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 --2- 172.21.15.26:0/1790366280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb100b2b30 0x7fcb100a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- 172.21.15.26:0/1790366280 >> 172.21.15.26:0/1790366280 conn(0x7fcb100b6110 msgr2=0x7fcb100b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- 172.21.15.26:0/1790366280 shutdown_connections 2022-07-20T02:35:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- 172.21.15.26:0/1790366280 wait complete. 2022-07-20T02:35:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 Processor -- start 2022-07-20T02:35:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.162+0000 7fcb15245700 1 -- start start 2022-07-20T02:35:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 0x7fcb100ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10147660 0x7fcb10156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1015ad40 con 0x7fcb100b2b30 2022-07-20T02:35:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10064060 con 0x7fcb10147660 2022-07-20T02:35:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100636c0 con 0x7fcb10060f10 2022-07-20T02:35:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0ed9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 0x7fcb100ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44008/0 (socket says 172.21.15.26:44008) 2022-07-20T02:35:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0ed9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 0x7fcb100ad6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37290/0 (socket says 172.21.15.26:37290) 2022-07-20T02:35:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 -- 172.21.15.26:0/2771262910 learned_addr learned my addr 172.21.15.26:0/2771262910 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0f59e700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10147660 0x7fcb10156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 -- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 msgr2=0x7fcb100ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 0x7fcb100ad6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 -- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10147660 msgr2=0x7fcb10156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10147660 0x7fcb10156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 -- 172.21.15.26:0/2771262910 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0800b1e0 con 0x7fcb100b2b30 2022-07-20T02:35:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb0e59c700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7fcb100641a0 tx=0x7fcb00016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0000b600 con 0x7fcb100b2b30 2022-07-20T02:35:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0001f3f0 con 0x7fcb100b2b30 2022-07-20T02:35:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb00009470 con 0x7fcb100b2b30 2022-07-20T02:35:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0001f750 con 0x7fcb100b2b30 2022-07-20T02:35:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb10063920 con 0x7fcb100b2b30 2022-07-20T02:35:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.166+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb10157450 con 0x7fcb100b2b30 2022-07-20T02:35:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.170+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb00026030 con 0x7fcb100b2b30 2022-07-20T02:35:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.170+0000 7fcafffff700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 0x7fcaec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.170+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcb0006cb00 con 0x7fcb100b2b30 2022-07-20T02:35:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.170+0000 7fcb0ed9d700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 0x7fcaec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.170+0000 7fcb0ed9d700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 0x7fcaec055e10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fcb0400b000 tx=0x7fcb04006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.174+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb000438c0 con 0x7fcb100b2b30 2022-07-20T02:35:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.338+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcb100a7500 con 0x7fcb100b2b30 2022-07-20T02:35:30.629 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:35:30.614+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:35:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.622+0000 7fcafffff700 1 -- 172.21.15.26:0/2771262910 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcb00024090 con 0x7fcb100b2b30 2022-07-20T02:35:30.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 msgr2=0x7fcaec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 0x7fcaec055e10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fcb0400b000 tx=0x7fcb04006df0).stop 2022-07-20T02:35:30.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 msgr2=0x7fcb101568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7fcb100641a0 tx=0x7fcb00016000).stop 2022-07-20T02:35:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 shutdown_connections 2022-07-20T02:35:30.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb100b2b30 0x7fcb101568d0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fcaec053950 0x7fcaec055e10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb10147660 0x7fcb10156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 --2- 172.21.15.26:0/2771262910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb10060f10 0x7fcb100ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 >> 172.21.15.26:0/2771262910 conn(0x7fcb100b6110 msgr2=0x7fcb100aadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:30.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 shutdown_connections 2022-07-20T02:35:30.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.626+0000 7fcb15245700 1 -- 172.21.15.26:0/2771262910 wait complete. 2022-07-20T02:35:30.655 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:35:30.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:30.667 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:35:30.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.806+0000 7fac0f34e700 1 Processor -- start 2022-07-20T02:35:30.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- start start 2022-07-20T02:35:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fac080b2920 0x7fac08147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 0x7fac08148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080b1bc0 con 0x7fac08066a80 2022-07-20T02:35:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0814b040 con 0x7fac080b2920 2022-07-20T02:35:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac080a8860 con 0x7fac08061050 2022-07-20T02:35:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fac080b2920 0x7fac08147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0c8e9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 0x7fac08148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fac080b2920 0x7fac08147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44028/0 (socket says 172.21.15.26:44028) 2022-07-20T02:35:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 -- 172.21.15.26:0/307228774 learned_addr learned my addr 172.21.15.26:0/307228774 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fabfffff700 1 -- 172.21.15.26:0/307228774 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018350313 0 0) 0x7fac080b1bc0 con 0x7fac08066a80 2022-07-20T02:35:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fabfffff700 1 -- 172.21.15.26:0/307228774 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec003260 con 0x7fac08066a80 2022-07-20T02:35:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 -- 172.21.15.26:0/307228774 >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 msgr2=0x7fac08148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 --2- 172.21.15.26:0/307228774 >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 0x7fac08148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 -- 172.21.15.26:0/307228774 >> v1:172.21.15.120:6789/0 conn(0x7fac08066a80 legacy=0x7fac08066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 -- 172.21.15.26:0/307228774 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08147700 con 0x7fac080b2920 2022-07-20T02:35:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0c8e9700 1 --2- 172.21.15.26:0/307228774 >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 0x7fac08148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0d0ea700 1 --2- 172.21.15.26:0/307228774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac080b2920 0x7fac08147d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fabf8002f70 tx=0x7fabf8006eb0).ready entity=mon.0 client_cookie=3b5c8e5705986a56 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fabfffff700 1 -- 172.21.15.26:0/307228774 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf800d610 con 0x7fac080b2920 2022-07-20T02:35:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fabfffff700 1 -- 172.21.15.26:0/307228774 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8000e90 con 0x7fac080b2920 2022-07-20T02:35:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fabfffff700 1 -- 172.21.15.26:0/307228774 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf800d9a0 con 0x7fac080b2920 2022-07-20T02:35:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- 172.21.15.26:0/307228774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac080b2920 msgr2=0x7fac08147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 --2- 172.21.15.26:0/307228774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac080b2920 0x7fac08147d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fabf8002f70 tx=0x7fabf8006eb0).stop 2022-07-20T02:35:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- 172.21.15.26:0/307228774 shutdown_connections 2022-07-20T02:35:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 --2- 172.21.15.26:0/307228774 >> v2:172.21.15.120:3300/0 conn(0x7fac08061050 0x7fac08148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 --2- 172.21.15.26:0/307228774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac080b2920 0x7fac08147d90 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- 172.21.15.26:0/307228774 >> 172.21.15.26:0/307228774 conn(0x7fac080b60a0 msgr2=0x7fac080b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- 172.21.15.26:0/307228774 shutdown_connections 2022-07-20T02:35:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.810+0000 7fac0f34e700 1 -- 172.21.15.26:0/307228774 wait complete. 2022-07-20T02:35:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 Processor -- start 2022-07-20T02:35:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- start start 2022-07-20T02:35:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac08061050 0x7fac080647b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac080b2920 0x7fac081454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:30.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814eaf0 con 0x7fac08066a80 2022-07-20T02:35:30.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814e310 con 0x7fac08061050 2022-07-20T02:35:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814e5b0 con 0x7fac080b2920 2022-07-20T02:35:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44034/0 (socket says 172.21.15.26:44034) 2022-07-20T02:35:30.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 -- 172.21.15.26:0/3532053219 learned_addr learned my addr 172.21.15.26:0/3532053219 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:30.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0d0ea700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac08061050 0x7fac080647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0d8eb700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac080b2920 0x7fac081454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:30.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 -- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac080b2920 msgr2=0x7fac081454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac080b2920 0x7fac081454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 -- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac08061050 msgr2=0x7fac080647b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac08061050 0x7fac080647b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 -- 172.21.15.26:0/3532053219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0400b180 con 0x7fac08066a80 2022-07-20T02:35:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0c8e9700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fac0400c000 tx=0x7fac04006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac04010780 con 0x7fac08066a80 2022-07-20T02:35:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac040108e0 con 0x7fac08066a80 2022-07-20T02:35:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf800b1e0 con 0x7fac08066a80 2022-07-20T02:35:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac04010c40 con 0x7fac08066a80 2022-07-20T02:35:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.814+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac08065c10 con 0x7fac08066a80 2022-07-20T02:35:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac080aaba0 con 0x7fac08066a80 2022-07-20T02:35:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fac0401c460 con 0x7fac08066a80 2022-07-20T02:35:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fabfe7fc700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 0x7fabf0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fac0406c680 con 0x7fac08066a80 2022-07-20T02:35:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fac0d0ea700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 0x7fabf0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.818+0000 7fac0d0ea700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 0x7fabf0055f60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fabf801c000 tx=0x7fabf8015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.826+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fac04069e90 con 0x7fac08066a80 2022-07-20T02:35:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0814ada0 con 0x7fac08066a80 2022-07-20T02:35:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fabfe7fc700 1 -- 172.21.15.26:0/3532053219 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2129 (secure 0 0 0) 0x7fac04043280 con 0x7fac08066a80 2022-07-20T02:35:31.007 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":81,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:14.691126+0000","modified":"2022-07-20T02:35:30.616956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T02:35:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 msgr2=0x7fabf0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 0x7fabf0055f60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fabf801c000 tx=0x7fabf8015000).stop 2022-07-20T02:35:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 msgr2=0x7fac080aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.986+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fac0400c000 tx=0x7fac04006df0).stop 2022-07-20T02:35:31.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 shutdown_connections 2022-07-20T02:35:31.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac08066a80 0x7fac080aaeb0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fabf0053aa0 0x7fabf0055f60 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac080b2920 0x7fac081454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 --2- 172.21.15.26:0/3532053219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac08061050 0x7fac080647b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 >> 172.21.15.26:0/3532053219 conn(0x7fac080b60a0 msgr2=0x7fac08149730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:31.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 shutdown_connections 2022-07-20T02:35:31.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:30.990+0000 7fac0f34e700 1 -- 172.21.15.26:0/3532053219 wait complete. 2022-07-20T02:35:31.068 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 81 2022-07-20T02:35:31.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:35:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 Processor -- start 2022-07-20T02:35:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 -- start start 2022-07-20T02:35:31.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b3c147000 0x7f9b3c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 0x7f9b3c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c1474e0 con 0x7f9b3c05cfd0 2022-07-20T02:35:31.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3c0b4e40 con 0x7f9b3c147000 2022-07-20T02:35:31.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42d03700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c05d3b0 con 0x7f9b3c060f00 2022-07-20T02:35:31.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 0x7f9b3c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b41d01700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9b3c05cfd0 0x7f9b3c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:57532/0 (socket says 172.21.15.26:57532) 2022-07-20T02:35:31.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b41d01700 1 -- 172.21.15.26:0/1618214402 learned_addr learned my addr 172.21.15.26:0/1618214402 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:31.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b41500700 1 --2- 172.21.15.26:0/1618214402 >> v2:172.21.15.120:3300/0 conn(0x7f9b3c147000 0x7f9b3c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 -- 172.21.15.26:0/1618214402 >> v2:172.21.15.120:3300/0 conn(0x7f9b3c147000 msgr2=0x7f9b3c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:31.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 --2- 172.21.15.26:0/1618214402 >> v2:172.21.15.120:3300/0 conn(0x7f9b3c147000 0x7f9b3c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 -- 172.21.15.26:0/1618214402 >> v1:172.21.15.120:6789/0 conn(0x7f9b3c05cfd0 legacy=0x7f9b3c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 -- 172.21.15.26:0/1618214402 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c1431e0 con 0x7f9b3c060f00 2022-07-20T02:35:31.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b42502700 1 --2- 172.21.15.26:0/1618214402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 0x7f9b3c13c9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9b340032f0 tx=0x7f9b340091d0).ready entity=mon.2 client_cookie=5bec65390a22d25d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b40cff700 1 -- 172.21.15.26:0/1618214402 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b3400b6c0 con 0x7f9b3c060f00 2022-07-20T02:35:31.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b40cff700 1 -- 172.21.15.26:0/1618214402 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3400b820 con 0x7f9b3c060f00 2022-07-20T02:35:31.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.194+0000 7f9b40cff700 1 -- 172.21.15.26:0/1618214402 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b3400b990 con 0x7f9b3c060f00 2022-07-20T02:35:31.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- 172.21.15.26:0/1618214402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 msgr2=0x7f9b3c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- 172.21.15.26:0/1618214402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 0x7f9b3c13c9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9b340032f0 tx=0x7f9b340091d0).stop 2022-07-20T02:35:31.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- 172.21.15.26:0/1618214402 shutdown_connections 2022-07-20T02:35:31.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- 172.21.15.26:0/1618214402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c060f00 0x7f9b3c13c9b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- 172.21.15.26:0/1618214402 >> v2:172.21.15.120:3300/0 conn(0x7f9b3c147000 0x7f9b3c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- 172.21.15.26:0/1618214402 >> 172.21.15.26:0/1618214402 conn(0x7f9b3c0620c0 msgr2=0x7f9b3c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:31.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- 172.21.15.26:0/1618214402 shutdown_connections 2022-07-20T02:35:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- 172.21.15.26:0/1618214402 wait complete. 2022-07-20T02:35:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 Processor -- start 2022-07-20T02:35:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- start start 2022-07-20T02:35:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 0x7f9b3c0b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 0x7f9b3c1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ad690 con 0x7f9b3c1460c0 2022-07-20T02:35:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c156ac0 con 0x7f9b3c060f00 2022-07-20T02:35:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42d03700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ad8c0 con 0x7f9b3c147000 2022-07-20T02:35:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42502700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42502700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44050/0 (socket says 172.21.15.26:44050) 2022-07-20T02:35:31.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b42502700 1 -- 172.21.15.26:0/3920932370 learned_addr learned my addr 172.21.15.26:0/3920932370 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:31.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b41500700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 0x7f9b3c1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.198+0000 7f9b41d01700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 0x7f9b3c0b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 -- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 msgr2=0x7f9b3c1560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 0x7f9b3c1560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 -- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 msgr2=0x7f9b3c0b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 0x7f9b3c0b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 -- 172.21.15.26:0/3920932370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34009470 con 0x7f9b3c1460c0 2022-07-20T02:35:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b41500700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 0x7f9b3c1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b41d01700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 0x7f9b3c0b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42502700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f9b340032f0 tx=0x7f9b34016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b3400b6c0 con 0x7f9b3c1460c0 2022-07-20T02:35:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b340107a0 con 0x7f9b3c1460c0 2022-07-20T02:35:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c0adb60 con 0x7f9b3c1460c0 2022-07-20T02:35:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b34010b00 con 0x7f9b3c1460c0 2022-07-20T02:35:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c0b8130 con 0x7f9b3c1460c0 2022-07-20T02:35:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c155e40 con 0x7f9b3c1460c0 2022-07-20T02:35:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b34029070 con 0x7f9b3c1460c0 2022-07-20T02:35:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 0x7f9b1c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f9b3406d170 con 0x7f9b3c1460c0 2022-07-20T02:35:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b41d01700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 0x7f9b1c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.202+0000 7f9b41d01700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 0x7f9b1c055ff0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f9b3800b000 tx=0x7f9b38009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.206+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b3c155e40 con 0x7f9b3c1460c0 2022-07-20T02:35:31.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.370+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9b3c155e40 con 0x7f9b3c1460c0 2022-07-20T02:35:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.638+0000 7f9b2affd700 1 -- 172.21.15.26:0/3920932370 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9b3c155e40 con 0x7f9b3c1460c0 2022-07-20T02:35:31.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 msgr2=0x7f9b1c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 0x7f9b1c055ff0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f9b3800b000 tx=0x7f9b38009040).stop 2022-07-20T02:35:31.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 msgr2=0x7f9b3c1565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f9b340032f0 tx=0x7f9b34016000).stop 2022-07-20T02:35:31.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 shutdown_connections 2022-07-20T02:35:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b3c1460c0 0x7f9b3c1565d0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9b1c053b30 0x7f9b1c055ff0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b3c147000 0x7f9b3c1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 --2- 172.21.15.26:0/3920932370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3c060f00 0x7f9b3c0b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 >> 172.21.15.26:0/3920932370 conn(0x7f9b3c05cfd0 msgr2=0x7f9b3c138f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:31.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 shutdown_connections 2022-07-20T02:35:31.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.642+0000 7f9b42d03700 1 -- 172.21.15.26:0/3920932370 wait complete. 2022-07-20T02:35:31.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:35:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 Processor -- start 2022-07-20T02:35:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 -- start start 2022-07-20T02:35:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 0x7fb7580b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb7580b3060 0x7fb7580b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7580b2e40 con 0x7fb758064220 2022-07-20T02:35:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580617a0 con 0x7fb7580b3060 2022-07-20T02:35:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.826+0000 7fb75d4b5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb758060b20 con 0x7fb7580b7d80 2022-07-20T02:35:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75ccb4700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb758064220 0x7fb758146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43336/0 (socket says 172.21.15.26:43336) 2022-07-20T02:35:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb7580b3060 0x7fb7580b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75ccb4700 1 -- 172.21.15.26:0/583382711 learned_addr learned my addr 172.21.15.26:0/583382711 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb757fff700 1 --2- 172.21.15.26:0/583382711 >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 0x7fb7580b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb756ffd700 1 -- 172.21.15.26:0/583382711 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929903551 0 0) 0x7fb7580b2e40 con 0x7fb758064220 2022-07-20T02:35:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 -- 172.21.15.26:0/583382711 >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 msgr2=0x7fb7580b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 --2- 172.21.15.26:0/583382711 >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 0x7fb7580b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 -- 172.21.15.26:0/583382711 >> v1:172.21.15.26:6789/0 conn(0x7fb758064220 legacy=0x7fb758146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 -- 172.21.15.26:0/583382711 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758060dc0 con 0x7fb7580b3060 2022-07-20T02:35:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb757fff700 1 --2- 172.21.15.26:0/583382711 >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 0x7fb7580b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb7577fe700 1 --2- 172.21.15.26:0/583382711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b3060 0x7fb7580b4510 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fb740002f70 tx=0x7fb740006df0).ready entity=mon.0 client_cookie=e9a738c81c68839e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb756ffd700 1 -- 172.21.15.26:0/583382711 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb74000d5c0 con 0x7fb7580b3060 2022-07-20T02:35:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb756ffd700 1 -- 172.21.15.26:0/583382711 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740000e90 con 0x7fb7580b3060 2022-07-20T02:35:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb756ffd700 1 -- 172.21.15.26:0/583382711 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb74000d910 con 0x7fb7580b3060 2022-07-20T02:35:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 -- 172.21.15.26:0/583382711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b3060 msgr2=0x7fb7580b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/583382711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b3060 0x7fb7580b4510 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fb740002f70 tx=0x7fb740006df0).stop 2022-07-20T02:35:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 -- 172.21.15.26:0/583382711 shutdown_connections 2022-07-20T02:35:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/583382711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b3060 0x7fb7580b4510 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/583382711 >> v2:172.21.15.120:3300/0 conn(0x7fb7580b7d80 0x7fb7580b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 -- 172.21.15.26:0/583382711 >> 172.21.15.26:0/583382711 conn(0x7fb75805f840 msgr2=0x7fb75805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 -- 172.21.15.26:0/583382711 shutdown_connections 2022-07-20T02:35:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.830+0000 7fb75d4b5700 1 -- 172.21.15.26:0/583382711 wait complete. 2022-07-20T02:35:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 Processor -- start 2022-07-20T02:35:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- start start 2022-07-20T02:35:31.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb758064220 0x7fb758145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7580b3060 0x7fb75813c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75813d2c0 con 0x7fb7580b7d80 2022-07-20T02:35:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758148460 con 0x7fb7580b3060 2022-07-20T02:35:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb758148700 con 0x7fb758064220 2022-07-20T02:35:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb757fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb758064220 0x7fb758145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44064/0 (socket says 172.21.15.26:44064) 2022-07-20T02:35:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 -- 172.21.15.26:0/3762635599 learned_addr learned my addr 172.21.15.26:0/3762635599 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb7577fe700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7580b3060 0x7fb75813c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 -- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb758064220 msgr2=0x7fb758145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb758064220 0x7fb758145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 -- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7580b3060 msgr2=0x7fb75813c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7580b3060 0x7fb75813c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 -- 172.21.15.26:0/3762635599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74000b180 con 0x7fb7580b7d80 2022-07-20T02:35:31.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75ccb4700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7fb750001360 tx=0x7fb750007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:31.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb750005ba0 con 0x7fb7580b7d80 2022-07-20T02:35:31.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750005d00 con 0x7fb7580b7d80 2022-07-20T02:35:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7581489a0 con 0x7fb7580b7d80 2022-07-20T02:35:32.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7500072c0 con 0x7fb7580b7d80 2022-07-20T02:35:32.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.834+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb758148ee0 con 0x7fb7580b7d80 2022-07-20T02:35:32.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb758061060 con 0x7fb7580b7d80 2022-07-20T02:35:32.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb750010070 con 0x7fb7580b7d80 2022-07-20T02:35:32.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb7557fa700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 0x7fb73c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:32.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb75006b990 con 0x7fb7580b7d80 2022-07-20T02:35:32.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb757fff700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 0x7fb73c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:32.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.838+0000 7fb757fff700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 0x7fb73c055fc0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7fb74800c000 tx=0x7fb748008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:32.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.846+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb750042700 con 0x7fb7580b7d80 2022-07-20T02:35:32.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:31.970+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7580574e0 con 0x7fb7580b7d80 2022-07-20T02:35:32.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.642+0000 7fb7557fa700 1 -- 172.21.15.26:0/3762635599 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v152) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb750045d20 con 0x7fb7580b7d80 2022-07-20T02:35:32.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 msgr2=0x7fb73c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:32.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 0x7fb73c055fc0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7fb74800c000 tx=0x7fb748008040).stop 2022-07-20T02:35:32.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 msgr2=0x7fb75813cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:32.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7fb750001360 tx=0x7fb750007e50).stop 2022-07-20T02:35:32.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 shutdown_connections 2022-07-20T02:35:32.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7580b7d80 0x7fb75813cdd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:32.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb73c053b00 0x7fb73c055fc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:32.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7580b3060 0x7fb75813c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:32.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 --2- 172.21.15.26:0/3762635599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb758064220 0x7fb758145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:32.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 >> 172.21.15.26:0/3762635599 conn(0x7fb75805f840 msgr2=0x7fb758066450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:32.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 shutdown_connections 2022-07-20T02:35:32.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.646+0000 7fb75d4b5700 1 -- 172.21.15.26:0/3762635599 wait complete. 2022-07-20T02:35:32.658 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:35:32.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:35:32.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.802+0000 7f2ffbfff700 1 Processor -- start 2022-07-20T02:35:32.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.802+0000 7f2ffbfff700 1 -- start start 2022-07-20T02:35:32.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2ffc0b30c0 0x7f2ffc060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:32.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2ffc05f840 0x7f2ffc066f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:32.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc1485b0 con 0x7f2ffc0952f0 2022-07-20T02:35:32.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc095a50 con 0x7f2ffc05f840 2022-07-20T02:35:32.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc14b730 con 0x7f2ffc0b30c0 2022-07-20T02:35:32.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2ffc05f840 0x7f2ffc066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:32.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2ffc05f840 0x7f2ffc066f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44072/0 (socket says 172.21.15.26:44072) 2022-07-20T02:35:32.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffa7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ffc0952f0 0x7f2ffc147ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43350/0 (socket says 172.21.15.26:43350) 2022-07-20T02:35:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/695031540 learned_addr learned my addr 172.21.15.26:0/695031540 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffaffd700 1 --2- 172.21.15.26:0/695031540 >> v2:172.21.15.120:3300/0 conn(0x7f2ffc0b30c0 0x7f2ffc060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:32.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/695031540 >> v2:172.21.15.120:3300/0 conn(0x7f2ffc0b30c0 msgr2=0x7f2ffc060570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:32.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 --2- 172.21.15.26:0/695031540 >> v2:172.21.15.120:3300/0 conn(0x7f2ffc0b30c0 0x7f2ffc060570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:32.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/695031540 >> v1:172.21.15.26:6789/0 conn(0x7f2ffc0952f0 legacy=0x7f2ffc147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:32.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/695031540 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc14b9d0 con 0x7f2ffc05f840 2022-07-20T02:35:32.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ff9ffb700 1 -- 172.21.15.26:0/695031540 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131138164 0 0) 0x7f2ffc1485b0 con 0x7f2ffc0952f0 2022-07-20T02:35:33.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffb7fe700 1 --2- 172.21.15.26:0/695031540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc05f840 0x7f2ffc066f90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f2ffc147680 tx=0x7f2ff4009290).ready entity=mon.0 client_cookie=86be1e623c848c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ff9ffb700 1 -- 172.21.15.26:0/695031540 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ff400b860 con 0x7f2ffc05f840 2022-07-20T02:35:33.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ff9ffb700 1 -- 172.21.15.26:0/695031540 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff400b9c0 con 0x7f2ffc05f840 2022-07-20T02:35:33.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ff9ffb700 1 -- 172.21.15.26:0/695031540 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ff400bb30 con 0x7f2ffc05f840 2022-07-20T02:35:33.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- 172.21.15.26:0/695031540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc05f840 msgr2=0x7f2ffc066f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/695031540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc05f840 0x7f2ffc066f90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f2ffc147680 tx=0x7f2ff4009290).stop 2022-07-20T02:35:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- 172.21.15.26:0/695031540 shutdown_connections 2022-07-20T02:35:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/695031540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc05f840 0x7f2ffc066f90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/695031540 >> v2:172.21.15.120:3300/0 conn(0x7f2ffc0b30c0 0x7f2ffc060570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.806+0000 7f2ffbfff700 1 -- 172.21.15.26:0/695031540 >> 172.21.15.26:0/695031540 conn(0x7f2ffc0b67f0 msgr2=0x7f2ffc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- 172.21.15.26:0/695031540 shutdown_connections 2022-07-20T02:35:33.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- 172.21.15.26:0/695031540 wait complete. 2022-07-20T02:35:33.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 Processor -- start 2022-07-20T02:35:33.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- start start 2022-07-20T02:35:33.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ffc05f840 0x7f2ffc09b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 0x7f2ffc063650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc09af80 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc062f50 con 0x7f2ffc05f840 2022-07-20T02:35:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc09b1b0 con 0x7f2ffc0952f0 2022-07-20T02:35:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44078/0 (socket says 172.21.15.26:44078) 2022-07-20T02:35:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/2493994971 learned_addr learned my addr 172.21.15.26:0/2493994971 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffaffd700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ffc05f840 0x7f2ffc09b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffa7fc700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 0x7f2ffc063650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 msgr2=0x7f2ffc063650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 0x7f2ffc063650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ffc05f840 msgr2=0x7f2ffc09b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ffc05f840 0x7f2ffc09b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 -- 172.21.15.26:0/2493994971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4009530 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffa7fc700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 0x7f2ffc063650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:33.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffb7fe700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f2ffc147680 tx=0x7f2ff4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ff4021630 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff4021790 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ff4021af0 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc0631b0 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.810+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc156c40 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc063340 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2ff400f070 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2febfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 0x7f2fd8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f2ff406d540 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2ffaffd700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 0x7f2fd8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.814+0000 7f2ffaffd700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 0x7f2fd8055e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f2fec00b000 tx=0x7f2fec006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.818+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ff4044460 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:32.942+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ffc147760 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.654+0000 7f2febfff700 1 -- 172.21.15.26:0/2493994971 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v153) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2ff4015070 con 0x7f2ffc0b30c0 2022-07-20T02:35:33.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 msgr2=0x7f2fd8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 0x7f2fd8055e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f2fec00b000 tx=0x7f2fec006df0).stop 2022-07-20T02:35:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 msgr2=0x7f2ffc156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f2ffc147680 tx=0x7f2ff4016000).stop 2022-07-20T02:35:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 shutdown_connections 2022-07-20T02:35:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2ffc0b30c0 0x7f2ffc156750 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2fd8053950 0x7f2fd8055e10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2ffc0952f0 0x7f2ffc063650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 --2- 172.21.15.26:0/2493994971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ffc05f840 0x7f2ffc09b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 >> 172.21.15.26:0/2493994971 conn(0x7f2ffc0b67f0 msgr2=0x7f2ffc0b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 shutdown_connections 2022-07-20T02:35:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.658+0000 7f2ffbfff700 1 -- 172.21.15.26:0/2493994971 wait complete. 2022-07-20T02:35:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:35:33.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T02:35:33.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 Processor -- start 2022-07-20T02:35:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 -- start start 2022-07-20T02:35:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4c24147be0 0x7f4c240aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b62d0 con 0x7f4c240a7fc0 2022-07-20T02:35:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2414aac0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c288bf700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2414ad60 con 0x7f4c24147be0 2022-07-20T02:35:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4c24147be0 0x7f4c240aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4c24147be0 0x7f4c240aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44084/0 (socket says 172.21.15.26:44084) 2022-07-20T02:35:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 -- 172.21.15.26:0/2885059158 learned_addr learned my addr 172.21.15.26:0/2885059158 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2359e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c240a7fc0 0x7f4c240af330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43366/0 (socket says 172.21.15.26:43366) 2022-07-20T02:35:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 -- 172.21.15.26:0/2885059158 >> v1:172.21.15.120:6789/0 conn(0x7f4c240b5ef0 legacy=0x7f4c241474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 -- 172.21.15.26:0/2885059158 >> v1:172.21.15.26:6789/0 conn(0x7f4c240a7fc0 legacy=0x7f4c240af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 -- 172.21.15.26:0/2885059158 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2414b000 con 0x7f4c24147be0 2022-07-20T02:35:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c2259c700 1 --2- 172.21.15.26:0/2885059158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c24147be0 0x7f4c240aee40 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f4c140032f0 tx=0x7f4c14009290).ready entity=mon.0 client_cookie=429296c64188321e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c21d9b700 1 -- 172.21.15.26:0/2885059158 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c1400b8a0 con 0x7f4c24147be0 2022-07-20T02:35:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c21d9b700 1 -- 172.21.15.26:0/2885059158 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1400ba00 con 0x7f4c24147be0 2022-07-20T02:35:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.834+0000 7f4c21d9b700 1 -- 172.21.15.26:0/2885059158 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c1400bb70 con 0x7f4c24147be0 2022-07-20T02:35:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- 172.21.15.26:0/2885059158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c24147be0 msgr2=0x7f4c240aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 --2- 172.21.15.26:0/2885059158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c24147be0 0x7f4c240aee40 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f4c140032f0 tx=0x7f4c14009290).stop 2022-07-20T02:35:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- 172.21.15.26:0/2885059158 shutdown_connections 2022-07-20T02:35:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 --2- 172.21.15.26:0/2885059158 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c24147be0 0x7f4c240aee40 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- 172.21.15.26:0/2885059158 >> 172.21.15.26:0/2885059158 conn(0x7f4c24060310 msgr2=0x7f4c240b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- 172.21.15.26:0/2885059158 shutdown_connections 2022-07-20T02:35:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- 172.21.15.26:0/2885059158 wait complete. 2022-07-20T02:35:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 Processor -- start 2022-07-20T02:35:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- start start 2022-07-20T02:35:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c240a7fc0 0x7f4c241472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c24147be0 0x7f4c24156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c24065dd0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c24064a10 con 0x7f4c24147be0 2022-07-20T02:35:33.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c288bf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c24064c40 con 0x7f4c240a7fc0 2022-07-20T02:35:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44090/0 (socket says 172.21.15.26:44090) 2022-07-20T02:35:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 -- 172.21.15.26:0/115635046 learned_addr learned my addr 172.21.15.26:0/115635046 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:33.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c22d9d700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c240a7fc0 0x7f4c241472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2359e700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c24147be0 0x7f4c24156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 -- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c240a7fc0 msgr2=0x7f4c241472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c240a7fc0 0x7f4c241472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 -- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c24147be0 msgr2=0x7f4c24156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c24147be0 0x7f4c24156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.838+0000 7f4c2259c700 1 -- 172.21.15.26:0/115635046 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14009530 con 0x7f4c240b5ef0 2022-07-20T02:35:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c2259c700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f4c240678a0 tx=0x7f4c14016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c1400bde0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1401f3f0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c241570c0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c1401f750 con 0x7f4c240b5ef0 2022-07-20T02:35:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2415aa00 con 0x7f4c240b5ef0 2022-07-20T02:35:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2414b2a0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4c14026030 con 0x7f4c240b5ef0 2022-07-20T02:35:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.842+0000 7f4c1bfff700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 0x7f4c08055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.846+0000 7f4c22d9d700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 0x7f4c08055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.846+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f4c1406ca40 con 0x7f4c240b5ef0 2022-07-20T02:35:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.846+0000 7f4c22d9d700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 0x7f4c08055fc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f4c0c00d000 tx=0x7f4c0c009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.850+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c14043ae0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.978+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4c240574e0 con 0x7f4c240b5ef0 2022-07-20T02:35:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c1bfff700 1 -- 172.21.15.26:0/115635046 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4c140240a0 con 0x7f4c240b5ef0 2022-07-20T02:35:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 msgr2=0x7f4c08055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 0x7f4c08055fc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f4c0c00d000 tx=0x7f4c0c009040).stop 2022-07-20T02:35:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 msgr2=0x7f4c241566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f4c240678a0 tx=0x7f4c14016000).stop 2022-07-20T02:35:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 shutdown_connections 2022-07-20T02:35:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.986+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4c240b5ef0 0x7f4c241566e0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4c08053b00 0x7f4c08055fc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c24147be0 0x7f4c24156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 --2- 172.21.15.26:0/115635046 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c240a7fc0 0x7f4c241472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 >> 172.21.15.26:0/115635046 conn(0x7f4c24060310 msgr2=0x7f4c240b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 shutdown_connections 2022-07-20T02:35:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:33.990+0000 7f4c288bf700 1 -- 172.21.15.26:0/115635046 wait complete. 2022-07-20T02:35:34.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-20T02:35:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.130+0000 7f0ff1481700 1 Processor -- start 2022-07-20T02:35:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.130+0000 7f0ff1481700 1 -- start start 2022-07-20T02:35:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.130+0000 7f0ff1481700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fec0b3260 0x7f0fec067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.130+0000 7f0ff1481700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec0979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.130+0000 7f0ff1481700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0fec097eb0 0x7f0fec098320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fec05ff70 con 0x7f0fec097eb0 2022-07-20T02:35:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fec0606e0 con 0x7f0fec0b3260 2022-07-20T02:35:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec0b3640 con 0x7f0fec061860 2022-07-20T02:35:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0fec097eb0 0x7f0fec098320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f0fec097eb0 0x7f0fec098320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44100/0 (socket says 172.21.15.26:44100) 2022-07-20T02:35:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 -- 172.21.15.26:0/268247418 learned_addr learned my addr 172.21.15.26:0/268247418 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0feb7fe700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec0979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0febfff700 1 --2- 172.21.15.26:0/268247418 >> v2:172.21.15.120:3300/0 conn(0x7f0fec0b3260 0x7f0fec067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 -- 172.21.15.26:0/268247418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 msgr2=0x7f0fec0979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec0979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 -- 172.21.15.26:0/268247418 >> v2:172.21.15.120:3300/0 conn(0x7f0fec0b3260 msgr2=0x7f0fec067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 --2- 172.21.15.26:0/268247418 >> v2:172.21.15.120:3300/0 conn(0x7f0fec0b3260 0x7f0fec067900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 -- 172.21.15.26:0/268247418 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0601a0 con 0x7f0fec097eb0 2022-07-20T02:35:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff0c80700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec098320 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f0fec097600 tx=0x7f0fe4009290).ready entity=mon.0 client_cookie=67305c2a932615e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0feaffd700 1 -- 172.21.15.26:0/268247418 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b680 con 0x7f0fec097eb0 2022-07-20T02:35:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0feaffd700 1 -- 172.21.15.26:0/268247418 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400b7e0 con 0x7f0fec097eb0 2022-07-20T02:35:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0feaffd700 1 -- 172.21.15.26:0/268247418 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe400b950 con 0x7f0fec097eb0 2022-07-20T02:35:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- 172.21.15.26:0/268247418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 msgr2=0x7f0fec098320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec098320 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f0fec097600 tx=0x7f0fe4009290).stop 2022-07-20T02:35:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- 172.21.15.26:0/268247418 shutdown_connections 2022-07-20T02:35:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec098320 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 --2- 172.21.15.26:0/268247418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec0979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 --2- 172.21.15.26:0/268247418 >> v2:172.21.15.120:3300/0 conn(0x7f0fec0b3260 0x7f0fec067900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- 172.21.15.26:0/268247418 >> 172.21.15.26:0/268247418 conn(0x7f0fec0b67f0 msgr2=0x7f0fec0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:34.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- 172.21.15.26:0/268247418 shutdown_connections 2022-07-20T02:35:34.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.134+0000 7f0ff1481700 1 -- 172.21.15.26:0/268247418 wait complete. 2022-07-20T02:35:34.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 Processor -- start 2022-07-20T02:35:34.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- start start 2022-07-20T02:35:34.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec09b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fec0b3260 0x7f0fec0b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec065120 con 0x7f0fec097eb0 2022-07-20T02:35:34.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1422d0 con 0x7f0fec0b3260 2022-07-20T02:35:34.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec09b860 con 0x7f0fec061860 2022-07-20T02:35:34.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0febfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec09b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44104/0 (socket says 172.21.15.26:44104) 2022-07-20T02:35:34.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 -- 172.21.15.26:0/4091875173 learned_addr learned my addr 172.21.15.26:0/4091875173 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:34.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff0c80700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fec0b3260 0x7f0fec0b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 -- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 msgr2=0x7f0fec09b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec09b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.255 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 -- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fec0b3260 msgr2=0x7f0fec0b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fec0b3260 0x7f0fec0b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 -- 172.21.15.26:0/4091875173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4009530 con 0x7f0fec097eb0 2022-07-20T02:35:34.256 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0feb7fe700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f0fe000c000 tx=0x7f0fe0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:34.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe0010780 con 0x7f0fec097eb0 2022-07-20T02:35:34.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe000b3b0 con 0x7f0fec097eb0 2022-07-20T02:35:34.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe0010ad0 con 0x7f0fec097eb0 2022-07-20T02:35:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec09b6a0 con 0x7f0fec097eb0 2022-07-20T02:35:34.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.138+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec0b3af0 con 0x7f0fec097eb0 2022-07-20T02:35:34.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec0574e0 con 0x7f0fec097eb0 2022-07-20T02:35:34.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0fe0010c30 con 0x7f0fec097eb0 2022-07-20T02:35:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0fe97fa700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 0x7f0fcc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0fe006bf10 con 0x7f0fec097eb0 2022-07-20T02:35:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0febfff700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 0x7f0fcc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.142+0000 7f0febfff700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 0x7f0fcc055fc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f0fdc008000 tx=0x7f0fdc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.146+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0fe0042f10 con 0x7f0fec097eb0 2022-07-20T02:35:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.306+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f0fec09b6a0 con 0x7f0fec097eb0 2022-07-20T02:35:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.658+0000 7f0fe97fa700 1 -- 172.21.15.26:0/4091875173 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v470) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0fe0046530 con 0x7f0fec097eb0 2022-07-20T02:35:34.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 msgr2=0x7f0fcc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 0x7f0fcc055fc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f0fdc008000 tx=0x7f0fdc008040).stop 2022-07-20T02:35:34.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 msgr2=0x7f0fec150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f0fe000c000 tx=0x7f0fe0006df0).stop 2022-07-20T02:35:34.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 shutdown_connections 2022-07-20T02:35:34.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0fec097eb0 0x7f0fec150330 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0fcc053b00 0x7f0fcc055fc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fec0b3260 0x7f0fec0b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 --2- 172.21.15.26:0/4091875173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fec061860 0x7f0fec09b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 >> 172.21.15.26:0/4091875173 conn(0x7f0fec0b67f0 msgr2=0x7f0fec0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:34.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 shutdown_connections 2022-07-20T02:35:34.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.662+0000 7f0ff1481700 1 -- 172.21.15.26:0/4091875173 wait complete. 2022-07-20T02:35:34.688 INFO:tasks.cephfs_test_runner:test_root_rctime (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T02:35:34.689 INFO:tasks.cephfs_test_runner:Starting test: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T02:35:34.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs' 2022-07-20T02:35:34.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 Processor -- start 2022-07-20T02:35:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 -- start start 2022-07-20T02:35:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f86a814c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a805d370 con 0x7f86a80ac150 2022-07-20T02:35:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80ab770 con 0x7f86a80b5d50 2022-07-20T02:35:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80aba10 con 0x7f86a80ab230 2022-07-20T02:35:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f869dd9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86a80ac150 0x7f86a80aab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43390/0 (socket says 172.21.15.26:43390) 2022-07-20T02:35:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f869dd9b700 1 -- 172.21.15.26:0/824230230 learned_addr learned my addr 172.21.15.26:0/824230230 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a6d9d700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f86a814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.925 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564924784 0 0) 0x7f86a805d370 con 0x7f86a80ac150 2022-07-20T02:35:34.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8688003260 con 0x7f86a80ac150 2022-07-20T02:35:34.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126828875 0 0) 0x7f86a80ab770 con 0x7f86a80b5d50 2022-07-20T02:35:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a805d370 con 0x7f86a80b5d50 2022-07-20T02:35:34.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226338522 0 0) 0x7f8688003260 con 0x7f86a80ac150 2022-07-20T02:35:34.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 msgr2=0x7f86a814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f86a814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.806+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 >> v1:172.21.15.120:6789/0 conn(0x7f86a80b5d50 legacy=0x7f86a8144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80abcb0 con 0x7f86a80ac150 2022-07-20T02:35:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f8690001660 con 0x7f86a80ac150 2022-07-20T02:35:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 >> v1:172.21.15.26:6789/0 conn(0x7f86a80ac150 legacy=0x7f86a80aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8688003910 0x7f8688003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f8688004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f8688004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:34.950 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80aba10 con 0x7f8688004500 2022-07-20T02:35:34.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868800b800 con 0x7f8688003910 2022-07-20T02:35:34.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868800b990 con 0x7f86a80ab230 2022-07-20T02:35:34.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f8688004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a659c700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8688003910 0x7f8688003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a6d9d700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f8688004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:34.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 -- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 msgr2=0x7f8688004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:34.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a80ab230 0x7f8688004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 -- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8688003910 msgr2=0x7f8688003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:34.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8688003910 0x7f8688003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:34.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 -- 172.21.15.26:0/824230230 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688003260 con 0x7f8688004500 2022-07-20T02:35:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f869dd9b700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f8688004980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f86880054d0 tx=0x7f86a805c860).ready entity=mon.0 client_cookie=488759fcba2d897b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86900070d0 con 0x7f8688004500 2022-07-20T02:35:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8690007230 con 0x7f8688004500 2022-07-20T02:35:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a5d9b700 1 -- 172.21.15.26:0/824230230 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869000c8b0 con 0x7f8688004500 2022-07-20T02:35:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- 172.21.15.26:0/824230230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 msgr2=0x7f8688004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 --2- 172.21.15.26:0/824230230 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f8688004980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f86880054d0 tx=0x7f86a805c860).stop 2022-07-20T02:35:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a6d9d700 1 -- 172.21.15.26:0/824230230 reap_dead start 2022-07-20T02:35:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- 172.21.15.26:0/824230230 shutdown_connections 2022-07-20T02:35:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- 172.21.15.26:0/824230230 >> 172.21.15.26:0/824230230 conn(0x7f86a8060310 msgr2=0x7f86a80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- 172.21.15.26:0/824230230 shutdown_connections 2022-07-20T02:35:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- 172.21.15.26:0/824230230 wait complete. 2022-07-20T02:35:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 Processor -- start 2022-07-20T02:35:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.810+0000 7f86a759e700 1 -- start start 2022-07-20T02:35:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a8150c60 0x7f86a80626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8153740 0x7f86a8153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8150280 con 0x7f8688004500 2022-07-20T02:35:35.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8150520 con 0x7f86a8150c60 2022-07-20T02:35:35.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a81507c0 con 0x7f86a8153740 2022-07-20T02:35:35.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44122/0 (socket says 172.21.15.26:44122) 2022-07-20T02:35:35.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a6d9d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8153740 0x7f86a8153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 -- 172.21.15.26:0/4078771922 learned_addr learned my addr 172.21.15.26:0/4078771922 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:35.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f869dd9b700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a8150c60 0x7f86a80626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 -- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8153740 msgr2=0x7f86a8153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8153740 0x7f86a8153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 -- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a8150c60 msgr2=0x7f86a80626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a8150c60 0x7f86a80626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 -- 172.21.15.26:0/4078771922 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a81510c0 con 0x7f8688004500 2022-07-20T02:35:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a659c700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f869800d000 tx=0x7f8698007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:35.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8698011780 con 0x7f8688004500 2022-07-20T02:35:35.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86980118e0 con 0x7f8688004500 2022-07-20T02:35:35.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a8063000 con 0x7f8688004500 2022-07-20T02:35:35.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8698011c40 con 0x7f8688004500 2022-07-20T02:35:35.107 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a8062be0 con 0x7f8688004500 2022-07-20T02:35:35.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.814+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a805cff0 con 0x7f8688004500 2022-07-20T02:35:35.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.818+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f869801b030 con 0x7f8688004500 2022-07-20T02:35:35.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.818+0000 7f869ffff700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 0x7f868c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.818+0000 7f869dd9b700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 0x7f868c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.818+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f869806be60 con 0x7f8688004500 2022-07-20T02:35:35.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.818+0000 7f869dd9b700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 0x7f868c055e10 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f8690019000 tx=0x7f8690013000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:35.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.822+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8698042f00 con 0x7f8688004500 2022-07-20T02:35:35.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:34.998+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]} v 0) v1 -- 0x7f86a80574e0 con 0x7f8688004500 2022-07-20T02:35:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.658+0000 7f869ffff700 1 -- 172.21.15.26:0/4078771922 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]}]=0 v471) v1 ==== 139+0+0 (secure 0 0 0) 0x7f8698046520 con 0x7f8688004500 2022-07-20T02:35:35.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 msgr2=0x7f868c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 0x7f868c055e10 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f8690019000 tx=0x7f8690013000).stop 2022-07-20T02:35:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 msgr2=0x7f86a813f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f869800d000 tx=0x7f8698007a00).stop 2022-07-20T02:35:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 shutdown_connections 2022-07-20T02:35:35.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f868c053950 0x7f868c055e10 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a8153740 0x7f86a8153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a8150c60 0x7f86a80626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 --2- 172.21.15.26:0/4078771922 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8688004500 0x7f86a813f4f0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 >> 172.21.15.26:0/4078771922 conn(0x7f86a8060310 msgr2=0x7f86a813a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:35.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 shutdown_connections 2022-07-20T02:35:35.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.662+0000 7f86a759e700 1 -- 172.21.15.26:0/4078771922 wait complete. 2022-07-20T02:35:35.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T02:35:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 Processor -- start 2022-07-20T02:35:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 -- start start 2022-07-20T02:35:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 0x7efe04066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efe040ac6f0 0x7efe040acb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 0x7efe04061110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe04146bb0 con 0x7efe040ac6f0 2022-07-20T02:35:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe04146e50 con 0x7efe0405ce30 2022-07-20T02:35:35.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe0a11a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe041470f0 con 0x7efe04060c80 2022-07-20T02:35:35.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efe040ac6f0 0x7efe040acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7efe040ac6f0 0x7efe040acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44134/0 (socket says 172.21.15.26:44134) 2022-07-20T02:35:35.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 -- 172.21.15.26:0/2641343570 learned_addr learned my addr 172.21.15.26:0/2641343570 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:35.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe09919700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 0x7efe04061110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe09118700 1 --2- 172.21.15.26:0/2641343570 >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 0x7efe04066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 -- 172.21.15.26:0/2641343570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 msgr2=0x7efe04061110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 0x7efe04061110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 -- 172.21.15.26:0/2641343570 >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 msgr2=0x7efe04066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 --2- 172.21.15.26:0/2641343570 >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 0x7efe04066c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 -- 172.21.15.26:0/2641343570 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04147390 con 0x7efe040ac6f0 2022-07-20T02:35:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe09919700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 0x7efe04061110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe09118700 1 --2- 172.21.15.26:0/2641343570 >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 0x7efe04066c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.814+0000 7efe08917700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe040ac6f0 0x7efe040acb10 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7efdf4002f70 tx=0x7efdf4006eb0).ready entity=mon.0 client_cookie=bf8178081bdbdd96 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efdf3fff700 1 -- 172.21.15.26:0/2641343570 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdf400d570 con 0x7efe040ac6f0 2022-07-20T02:35:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efdf3fff700 1 -- 172.21.15.26:0/2641343570 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4000e90 con 0x7efe040ac6f0 2022-07-20T02:35:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efdf3fff700 1 -- 172.21.15.26:0/2641343570 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdf400d900 con 0x7efe040ac6f0 2022-07-20T02:35:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- 172.21.15.26:0/2641343570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe040ac6f0 msgr2=0x7efe040acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe040ac6f0 0x7efe040acb10 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7efdf4002f70 tx=0x7efdf4006eb0).stop 2022-07-20T02:35:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- 172.21.15.26:0/2641343570 shutdown_connections 2022-07-20T02:35:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe04060c80 0x7efe04061110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 --2- 172.21.15.26:0/2641343570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe040ac6f0 0x7efe040acb10 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 --2- 172.21.15.26:0/2641343570 >> v2:172.21.15.120:3300/0 conn(0x7efe0405ce30 0x7efe04066c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- 172.21.15.26:0/2641343570 >> 172.21.15.26:0/2641343570 conn(0x7efe040b7090 msgr2=0x7efe0405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- 172.21.15.26:0/2641343570 shutdown_connections 2022-07-20T02:35:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- 172.21.15.26:0/2641343570 wait complete. 2022-07-20T02:35:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 Processor -- start 2022-07-20T02:35:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.818+0000 7efe0a11a700 1 -- start start 2022-07-20T02:35:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe04060c80 0x7efe040b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe040ac6f0 0x7efe040b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040afa40 con 0x7efe0405ce30 2022-07-20T02:35:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040aee10 con 0x7efe04060c80 2022-07-20T02:35:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe040aeb70 con 0x7efe040ac6f0 2022-07-20T02:35:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44138/0 (socket says 172.21.15.26:44138) 2022-07-20T02:35:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 -- 172.21.15.26:0/530417515 learned_addr learned my addr 172.21.15.26:0/530417515 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe08917700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe04060c80 0x7efe040b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09919700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe040ac6f0 0x7efe040b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 -- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe040ac6f0 msgr2=0x7efe040b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe040ac6f0 0x7efe040b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 -- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe04060c80 msgr2=0x7efe040b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe04060c80 0x7efe040b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 -- 172.21.15.26:0/530417515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0000b180 con 0x7efe0405ce30 2022-07-20T02:35:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe09118700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7efe0000c000 tx=0x7efe00006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe0000c890 con 0x7efe0405ce30 2022-07-20T02:35:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0000c9f0 con 0x7efe0405ce30 2022-07-20T02:35:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc009470 con 0x7efe0405ce30 2022-07-20T02:35:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe0000cd50 con 0x7efe0405ce30 2022-07-20T02:35:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.822+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf400b1e0 con 0x7efe0405ce30 2022-07-20T02:35:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efe0001c460 con 0x7efe0405ce30 2022-07-20T02:35:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe04065f00 con 0x7efe0405ce30 2022-07-20T02:35:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efdf27fc700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 0x7efde4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7efe0006c330 con 0x7efe0405ce30 2022-07-20T02:35:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efe08917700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 0x7efde4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.826+0000 7efe08917700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 0x7efde4055e10 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7efdf401c000 tx=0x7efdf4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.834+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe04065f00 con 0x7efe0405ce30 2022-07-20T02:35:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.962+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efe04065f00 con 0x7efe0405ce30 2022-07-20T02:35:36.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efdf27fc700 1 -- 172.21.15.26:0/530417515 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7efe04065f00 con 0x7efe0405ce30 2022-07-20T02:35:36.017 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 msgr2=0x7efde4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 0x7efde4055e10 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7efdf401c000 tx=0x7efdf4015000).stop 2022-07-20T02:35:36.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 msgr2=0x7efe04148ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7efe0000c000 tx=0x7efe00006df0).stop 2022-07-20T02:35:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 shutdown_connections 2022-07-20T02:35:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efde4053950 0x7efde4055e10 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe040ac6f0 0x7efe040b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe04060c80 0x7efe040b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 --2- 172.21.15.26:0/530417515 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efe0405ce30 0x7efe04148ad0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 >> 172.21.15.26:0/530417515 conn(0x7efe040b7090 msgr2=0x7efe04144af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:36.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 shutdown_connections 2022-07-20T02:35:36.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:35.970+0000 7efe0a11a700 1 -- 172.21.15.26:0/530417515 wait complete. 2022-07-20T02:35:36.028 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:35:36.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:36.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:36.088 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:36.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:35:36.301 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:35:36.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:36.314 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:36.314 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:35:36.338 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T02:35:36.339 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T02:35:36.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:36.341 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:36.355 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:35:36.411 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T02:35:36.413 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:36.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:35:36.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 Processor -- start 2022-07-20T02:35:36.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 -- start start 2022-07-20T02:35:36.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdfdc0b2920 0x7fdfdc147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfdc061050 0x7fdfdc148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc0b1bc0 con 0x7fdfdc066a80 2022-07-20T02:35:36.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc14b040 con 0x7fdfdc061050 2022-07-20T02:35:36.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe2cb8700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc0a8860 con 0x7fdfdc0b2920 2022-07-20T02:35:36.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfdc061050 0x7fdfdc148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfdc061050 0x7fdfdc148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44146/0 (socket says 172.21.15.26:44146) 2022-07-20T02:35:36.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 -- 172.21.15.26:0/1388786092 learned_addr learned my addr 172.21.15.26:0/1388786092 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:36.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfe0a54700 1 --2- 172.21.15.26:0/1388786092 >> v2:172.21.15.120:3300/0 conn(0x7fdfdc0b2920 0x7fdfdc147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 -- 172.21.15.26:0/1388786092 >> v2:172.21.15.120:3300/0 conn(0x7fdfdc0b2920 msgr2=0x7fdfdc147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:36.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 --2- 172.21.15.26:0/1388786092 >> v2:172.21.15.120:3300/0 conn(0x7fdfdc0b2920 0x7fdfdc147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 -- 172.21.15.26:0/1388786092 >> v1:172.21.15.120:6789/0 conn(0x7fdfdc066a80 legacy=0x7fdfdc066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 -- 172.21.15.26:0/1388786092 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc147700 con 0x7fdfdc061050 2022-07-20T02:35:36.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdbfff700 1 --2- 172.21.15.26:0/1388786092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc061050 0x7fdfdc148280 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fdfcc0032f0 tx=0x7fdfcc009290).ready entity=mon.0 client_cookie=ed0c394346d99ba7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:36.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdb7fe700 1 -- 172.21.15.26:0/1388786092 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfcc00b870 con 0x7fdfdc061050 2022-07-20T02:35:36.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdb7fe700 1 -- 172.21.15.26:0/1388786092 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00b9d0 con 0x7fdfdc061050 2022-07-20T02:35:36.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.534+0000 7fdfdb7fe700 1 -- 172.21.15.26:0/1388786092 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfcc00bb40 con 0x7fdfdc061050 2022-07-20T02:35:36.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/1388786092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc061050 msgr2=0x7fdfdc148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/1388786092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc061050 0x7fdfdc148280 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fdfcc0032f0 tx=0x7fdfcc009290).stop 2022-07-20T02:35:36.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/1388786092 shutdown_connections 2022-07-20T02:35:36.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/1388786092 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc061050 0x7fdfdc148280 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/1388786092 >> v2:172.21.15.120:3300/0 conn(0x7fdfdc0b2920 0x7fdfdc147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/1388786092 >> 172.21.15.26:0/1388786092 conn(0x7fdfdc0b60a0 msgr2=0x7fdfdc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:36.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/1388786092 shutdown_connections 2022-07-20T02:35:36.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/1388786092 wait complete. 2022-07-20T02:35:36.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 Processor -- start 2022-07-20T02:35:36.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- start start 2022-07-20T02:35:36.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfdc061050 0x7fdfdc0a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc066a80 0x7fdfdc0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.553 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0a98f0 con 0x7fdfdc0b2920 2022-07-20T02:35:36.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc062990 con 0x7fdfdc066a80 2022-07-20T02:35:36.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe2cb8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc062c30 con 0x7fdfdc061050 2022-07-20T02:35:36.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44154/0 (socket says 172.21.15.26:44154) 2022-07-20T02:35:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 -- 172.21.15.26:0/2612207280 learned_addr learned my addr 172.21.15.26:0/2612207280 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfdbfff700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc066a80 0x7fdfdc0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe0a54700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfdc061050 0x7fdfdc0a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 -- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfdc061050 msgr2=0x7fdfdc0a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfdc061050 0x7fdfdc0a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 -- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc066a80 msgr2=0x7fdfdc0adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc066a80 0x7fdfdc0adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.538+0000 7fdfe1255700 1 -- 172.21.15.26:0/2612207280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc009530 con 0x7fdfdc0b2920 2022-07-20T02:35:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe1255700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7fdfdc147c70 tx=0x7fdfd4007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd4012400 con 0x7fdfdc0b2920 2022-07-20T02:35:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4012560 con 0x7fdfdc0b2920 2022-07-20T02:35:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc0ad6f0 con 0x7fdfdc0b2920 2022-07-20T02:35:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd40128c0 con 0x7fdfdc0b2920 2022-07-20T02:35:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc0ae080 con 0x7fdfdc0b2920 2022-07-20T02:35:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc0ad880 con 0x7fdfdc0b2920 2022-07-20T02:35:36.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdfd4019030 con 0x7fdfdc0b2920 2022-07-20T02:35:36.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 0x7fdfc0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:36.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fdfd406b090 con 0x7fdfdc0b2920 2022-07-20T02:35:36.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe0a54700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 0x7fdfc0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:36.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.542+0000 7fdfe0a54700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 0x7fdfc0055fc0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fdfc400b000 tx=0x7fdfc4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:36.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.546+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdfd4042130 con 0x7fdfdc0b2920 2022-07-20T02:35:36.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.674+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdfdc14ada0 con 0x7fdfdc0b2920 2022-07-20T02:35:36.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.674+0000 7fdfd9ffb700 1 -- 172.21.15.26:0/2612207280 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v153) v1 ==== 81+0+29275 (secure 0 0 0) 0x7fdfd4017070 con 0x7fdfdc0b2920 2022-07-20T02:35:36.684 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:36.684 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:35:36.685 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 153, 2022-07-20T02:35:36.685 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:35:36.686 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:35:36.686 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:35:33.642678+0000", 2022-07-20T02:35:36.687 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:36.687 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:35:36.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:35:36.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:35:36.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:35:36.688 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:35:36.688 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:35:36.688 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:35:36.689 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:35:36.689 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:36.689 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:35:36.689 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:35:36.691 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:35:36.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:35:36.698 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:35:36.699 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 28, 2022-07-20T02:35:36.699 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:35:36.699 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:35:36.699 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:35:36.700 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:35:36.700 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:35:36.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.701 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:35:36.701 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:35:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:35:36.722 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:36.723 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:36.724 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:35:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:35:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:35:36.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:35:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:35:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:36.726 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:36.727 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:35:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:36.728 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:36.729 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:36.771 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:36.772 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:36.773 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:36.774 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:36.782 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:36.787 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:36.788 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:36.789 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:36.794 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:36.795 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:36.796 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:36.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.806 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:36.808 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:36.818 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:36.819 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:36.820 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:36.821 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:35:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:35:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.823 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:35:36.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:36.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:35:36.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:35:36.826 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:35:36.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:35:36.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:35:36.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:35:36.828 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:35:36.829 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:35:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:35:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:35:36.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:35:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:35:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:35:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:35:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:35:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:35:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:35:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:35:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:35:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:35:36.887 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:35:36.887 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:35:36.887 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:35:36.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:35:36.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:35:36.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:35:36.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:35:36.891 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:35:36.892 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:35:36.892 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:35:36.893 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:35:36.893 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:35:36.893 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:35:36.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:35:36.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:35:36.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:35:36.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:35:36.896 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:35:36.896 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:35:36.896 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:35:36.897 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:35:36.928 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:35:36.929 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:35:36.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:35:36.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:35:36.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:35:36.931 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:35:36.931 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:35:36.932 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:35:36.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:35:36.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.933 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:35:36.949 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:35:36.958 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:35:36.959 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:35:36.960 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:35:36.960 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:35:36.961 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:35:36.962 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:35:36.962 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:35:36.963 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:35:36.963 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:35:36.964 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:35:36.964 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:35:36.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:35:36.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:36.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:36.966 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:36.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:35:36.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:36.968 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:35:36.968 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:36.969 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:36.969 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:36.969 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:36.970 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:36.970 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:36.971 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:36.971 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:36.971 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:36.972 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:36.972 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:36.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:36.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:36.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:35:36.974 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:36.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:36.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:35:36.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:36.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:36.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:36.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.979 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:36.979 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:36.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:36.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:35:36.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:36.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:36.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:35:36.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:36.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:36.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:36.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:36.998 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:36.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:36.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:36.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:36.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:35:37.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:37.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:35:37.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:35:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:35:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:35:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:35:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:35:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:35:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:35:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:35:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:35:37.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:35:37.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.043 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:35:37.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:35:37.048 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.050 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:35:37.052 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:35:37.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.080 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:35:37.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:35:37.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:35:37.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:35:37.091 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:35:37.091 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:35:37.092 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:35:37.092 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.092 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.094 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:37.094 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:35:37.095 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.095 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.095 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.096 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.097 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 147, 2022-07-20T02:35:37.097 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.097 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:35:37.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:35:37.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.128 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:35:37.131 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:35:37.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.135 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:35:37.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:35:37.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.141 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:35:37.142 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:35:37.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:35:37.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.145 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:35:37.145 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:35:37.145 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:35:37.146 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:35:37.146 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.146 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.146 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.148 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:37.149 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:35:37.161 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.205 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.224 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.225 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.225 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.225 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 136, 2022-07-20T02:35:37.226 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.226 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:35:37.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:35:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:35:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:35:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:35:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:35:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:35:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:35:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:35:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:35:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:35:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:35:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:35:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:35:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 147, 2022-07-20T02:35:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:35:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:35:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:35:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:35:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:35:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:35:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:35:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:35:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:35:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:35:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:35:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:35:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:35:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:35:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:35:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:35:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:35:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:35:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:35:37.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:35:37.384 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.385 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:35:37.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:35:37.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:35:37.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.426 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:35:37.438 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:35:37.439 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:35:37.439 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:35:37.439 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.439 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.440 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.440 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:37.441 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:35:37.441 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.441 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.474 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.495 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.495 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:37.496 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.496 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:35:37.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:35:37.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.511 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:35:37.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:35:37.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.519 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:35:37.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:35:37.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.555 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:35:37.557 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:35:37.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:35:37.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:35:37.561 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:35:37.562 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:35:37.563 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:35:37.563 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.564 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.564 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.565 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:37.565 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:35:37.565 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:35:37.566 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:35:37.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:35:37.566 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:35:37.567 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:35:37.567 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:35:37.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:35:37.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 148, 2022-07-20T02:35:37.581 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:35:37.581 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:35:37.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:35:37.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:35:37.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:35:37.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.603 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:35:37.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:35:37.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:35:37.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.613 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:35:37.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:35:37.616 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:35:37.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.620 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:35:37.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:35:37.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:35:37.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:35:37.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:35:37.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:35:37.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:35:37.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:35:37.638 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:35:37.639 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:35:37.639 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:35:37.640 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:35:37.640 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:35:37.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:35:37.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:37.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.642 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:37.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:35:37.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.643 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:35:37.644 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.644 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.645 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.645 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.645 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.647 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:35:37.648 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.649 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:35:37.649 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.650 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.650 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.651 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.652 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:35:37.652 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.653 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:35:37.657 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.657 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.676 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.676 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.677 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.677 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:35:37.678 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.679 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:35:37.679 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.680 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.680 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.680 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.699 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.699 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:35:37.700 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.701 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:35:37.701 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.702 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.703 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.703 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.704 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.704 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:35:37.705 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:35:37.708 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.708 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.709 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.710 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.710 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.726 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:35:37.727 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.728 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:35:37.728 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.729 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.729 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.743 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.744 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.744 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:35:37.745 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:37.746 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:35:37.747 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:35:37.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:35:37.754 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:35:37.754 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:35:37.754 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:35:37.765 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:35:37.766 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:35:37.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.767 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:35:37.767 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:35:37.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:35:37.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:35:37.768 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:35:37.769 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": { 2022-07-20T02:35:37.781 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6837/853590136": "2022-07-21T02:35:30.343246+0000", 2022-07-20T02:35:37.781 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6836/853590136": "2022-07-21T02:35:30.343246+0000" 2022-07-20T02:35:37.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.781 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:35:37.782 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:35:37.782 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:35:37.782 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:37.783 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:35:37.783 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:35:37.783 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:37.783 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:35:37.784 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:37.784 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:37.790 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:35:37.790 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:35:37.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.791 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:35:37.791 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:35:37.791 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:35:37.791 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:35:37.792 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:35:37.793 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:35:37.794 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:35:37.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:37.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:37.796 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:35:37.796 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:35:37.797 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:35:37.812 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:35:37.813 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:35:37.814 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 msgr2=0x7fdfc0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:37.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 0x7fdfc0055fc0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fdfc400b000 tx=0x7fdfc4008040).stop 2022-07-20T02:35:37.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 msgr2=0x7fdfdc0abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:37.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7fdfdc147c70 tx=0x7fdfd4007b80).stop 2022-07-20T02:35:37.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 shutdown_connections 2022-07-20T02:35:37.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfdc0b2920 0x7fdfdc0abb80 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:37.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdfc0053b00 0x7fdfc0055fc0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc066a80 0x7fdfdc0adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:37.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 --2- 172.21.15.26:0/2612207280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfdc061050 0x7fdfdc0a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:37.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 >> 172.21.15.26:0/2612207280 conn(0x7fdfdc0b60a0 msgr2=0x7fdfdc14b500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:37.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 shutdown_connections 2022-07-20T02:35:37.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:36.678+0000 7fdfe2cb8700 1 -- 172.21.15.26:0/2612207280 wait complete. 2022-07-20T02:35:37.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 Processor -- start 2022-07-20T02:35:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 -- start start 2022-07-20T02:35:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0268147000 0x7f0268143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02681474e0 con 0x7f0268060f00 2022-07-20T02:35:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680b4e40 con 0x7f026805cfd0 2022-07-20T02:35:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026f19e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026805d3b0 con 0x7f0268147000 2022-07-20T02:35:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026e99d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0268060f00 0x7f026813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43442/0 (socket says 172.21.15.26:43442) 2022-07-20T02:35:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026e99d700 1 -- 172.21.15.26:0/3100411918 learned_addr learned my addr 172.21.15.26:0/3100411918 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026d99b700 1 --2- 172.21.15.26:0/3100411918 >> v2:172.21.15.120:3300/0 conn(0x7f0268147000 0x7f0268143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.938+0000 7f026d19a700 1 -- 172.21.15.26:0/3100411918 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092049088 0 0) 0x7f02681474e0 con 0x7f0268060f00 2022-07-20T02:35:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d19a700 1 -- 172.21.15.26:0/3100411918 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0248003260 con 0x7f0268060f00 2022-07-20T02:35:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d99b700 1 -- 172.21.15.26:0/3100411918 >> v1:172.21.15.120:6789/0 conn(0x7f026805cfd0 legacy=0x7f02680b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d99b700 1 -- 172.21.15.26:0/3100411918 >> v1:172.21.15.26:6789/0 conn(0x7f0268060f00 legacy=0x7f026813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d99b700 1 -- 172.21.15.26:0/3100411918 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02681431e0 con 0x7f0268147000 2022-07-20T02:35:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d99b700 1 --2- 172.21.15.26:0/3100411918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f0268143bc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f02580032f0 tx=0x7f02580091d0).ready entity=mon.1 client_cookie=e48504321fc1b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d19a700 1 -- 172.21.15.26:0/3100411918 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f025800b710 con 0x7f0268147000 2022-07-20T02:35:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d19a700 1 -- 172.21.15.26:0/3100411918 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025800b870 con 0x7f0268147000 2022-07-20T02:35:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026d19a700 1 -- 172.21.15.26:0/3100411918 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f025800b9e0 con 0x7f0268147000 2022-07-20T02:35:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 -- 172.21.15.26:0/3100411918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 msgr2=0x7f0268143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 --2- 172.21.15.26:0/3100411918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f0268143bc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f02580032f0 tx=0x7f02580091d0).stop 2022-07-20T02:35:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 -- 172.21.15.26:0/3100411918 shutdown_connections 2022-07-20T02:35:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 --2- 172.21.15.26:0/3100411918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f0268143bc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 -- 172.21.15.26:0/3100411918 >> 172.21.15.26:0/3100411918 conn(0x7f02680620c0 msgr2=0x7f026805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 -- 172.21.15.26:0/3100411918 shutdown_connections 2022-07-20T02:35:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 -- 172.21.15.26:0/3100411918 wait complete. 2022-07-20T02:35:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.942+0000 7f026f19e700 1 Processor -- start 2022-07-20T02:35:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- start start 2022-07-20T02:35:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0268060f00 0x7f0268145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f02680b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681557d0 con 0x7f0268141300 2022-07-20T02:35:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268142310 con 0x7f0268147000 2022-07-20T02:35:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02681425b0 con 0x7f0268060f00 2022-07-20T02:35:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.055 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026d99b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f02680b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44168/0 (socket says 172.21.15.26:44168) 2022-07-20T02:35:38.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 -- 172.21.15.26:0/2427702421 learned_addr learned my addr 172.21.15.26:0/2427702421 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e19c700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0268060f00 0x7f0268145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 -- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0268060f00 msgr2=0x7f0268145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0268060f00 0x7f0268145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 -- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 msgr2=0x7f02680b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f02680b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 -- 172.21.15.26:0/2427702421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009470 con 0x7f0268141300 2022-07-20T02:35:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026d99b700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f02680b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:38.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026e99d700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f0250001110 tx=0x7f0250007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02500070d0 con 0x7f0268141300 2022-07-20T02:35:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0250007230 con 0x7f0268141300 2022-07-20T02:35:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0268142850 con 0x7f0268141300 2022-07-20T02:35:38.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0250005610 con 0x7f0268141300 2022-07-20T02:35:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.946+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0268141c70 con 0x7f0268141300 2022-07-20T02:35:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0268143480 con 0x7f0268141300 2022-07-20T02:35:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f025001a060 con 0x7f0268141300 2022-07-20T02:35:38.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f02677fe700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 0x7f024c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f025006b110 con 0x7f0268141300 2022-07-20T02:35:38.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f026e19c700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 0x7f024c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.950+0000 7f026e19c700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 0x7f024c055f30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f026000d000 tx=0x7f0260007b70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:37.954+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02500421b0 con 0x7f0268141300 2022-07-20T02:35:38.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.114+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02680574e0 con 0x7f0268141300 2022-07-20T02:35:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.114+0000 7f02677fe700 1 -- 172.21.15.26:0/2427702421 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+1553 (secure 0 0 0) 0x7f0250018020 con 0x7f0268141300 2022-07-20T02:35:38.120 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:38.120 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76},{"gid":8760,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/202857226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":202857226},{"type":"v1","addr":"172.21.15.26:6837","nonce":202857226}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":83}],"filesystems":[]} 2022-07-20T02:35:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 msgr2=0x7f024c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 0x7f024c055f30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f026000d000 tx=0x7f0260007b70).stop 2022-07-20T02:35:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 msgr2=0x7f0268141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f0250001110 tx=0x7f0250007af0).stop 2022-07-20T02:35:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 shutdown_connections 2022-07-20T02:35:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0268141300 0x7f0268141780 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f024c053a70 0x7f024c055f30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0268147000 0x7f02680b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 --2- 172.21.15.26:0/2427702421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0268060f00 0x7f0268145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 >> 172.21.15.26:0/2427702421 conn(0x7f026805cfd0 msgr2=0x7f026813bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 shutdown_connections 2022-07-20T02:35:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.118+0000 7f026f19e700 1 -- 172.21.15.26:0/2427702421 wait complete. 2022-07-20T02:35:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 83 2022-07-20T02:35:38.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:38.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 Processor -- start 2022-07-20T02:35:38.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 -- start start 2022-07-20T02:35:38.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6bd40b5ef0 0x7f6bd41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40b62d0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd414aac0 con 0x7f6bd40a7fc0 2022-07-20T02:35:38.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bda2e6700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd414ad60 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd92e4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6bd40b5ef0 0x7f6bd41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd9ae5700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6bd40a8c40 0x7f6bd4151dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43456/0 (socket says 172.21.15.26:43456) 2022-07-20T02:35:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd9ae5700 1 -- 172.21.15.26:0/3301805059 learned_addr learned my addr 172.21.15.26:0/3301805059 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498024992 0 0) 0x7f6bd40b62d0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc0003260 con 0x7f6bd40a8c40 2022-07-20T02:35:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885502872 0 0) 0x7f6bd414aac0 con 0x7f6bd40a7fc0 2022-07-20T02:35:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd40b62d0 con 0x7f6bd40a7fc0 2022-07-20T02:35:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd92e4700 1 -- 172.21.15.26:0/3301805059 >> v1:172.21.15.120:6789/0 conn(0x7f6bd40a7fc0 legacy=0x7f6bd40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd92e4700 1 -- 172.21.15.26:0/3301805059 >> v1:172.21.15.26:6789/0 conn(0x7f6bd40a8c40 legacy=0x7f6bd4151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd92e4700 1 -- 172.21.15.26:0/3301805059 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd414b000 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd92e4700 1 --2- 172.21.15.26:0/3301805059 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd41474d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f6bc8002f70 tx=0x7f6bc8006df0).ready entity=mon.0 client_cookie=ea9d4332cadc06e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc800d540 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8000e90 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.290+0000 7f6bd3fff700 1 -- 172.21.15.26:0/3301805059 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc800d890 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- 172.21.15.26:0/3301805059 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40b5ef0 msgr2=0x7f6bd41474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/3301805059 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd41474d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f6bc8002f70 tx=0x7f6bc8006df0).stop 2022-07-20T02:35:38.403 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- 172.21.15.26:0/3301805059 shutdown_connections 2022-07-20T02:35:38.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/3301805059 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd41474d0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- 172.21.15.26:0/3301805059 >> 172.21.15.26:0/3301805059 conn(0x7f6bd4060310 msgr2=0x7f6bd40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:38.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- 172.21.15.26:0/3301805059 shutdown_connections 2022-07-20T02:35:38.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- 172.21.15.26:0/3301805059 wait complete. 2022-07-20T02:35:38.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 Processor -- start 2022-07-20T02:35:38.425 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.294+0000 7f6bda2e6700 1 -- start start 2022-07-20T02:35:38.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bd40a7fc0 0x7f6bd4144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.426 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd40679c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.431 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4067eb0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd415a9e0 con 0x7f6bd40b5ef0 2022-07-20T02:35:38.435 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40627d0 con 0x7f6bd40a7fc0 2022-07-20T02:35:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44180/0 (socket says 172.21.15.26:44180) 2022-07-20T02:35:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 -- 172.21.15.26:0/352337063 learned_addr learned my addr 172.21.15.26:0/352337063 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:38.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd92e4700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bd40a7fc0 0x7f6bd4144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd9ae5700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd40679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 -- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bd40a7fc0 msgr2=0x7f6bd4144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bd40a7fc0 0x7f6bd4144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 -- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 msgr2=0x7f6bd40679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd40679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 -- 172.21.15.26:0/352337063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc800b180 con 0x7f6bd40a8c40 2022-07-20T02:35:38.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd9ae5700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd40679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:38.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd8ae3700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f6bc4001490 tx=0x7f6bc4007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc400e4a0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc400e600 con 0x7f6bd40a8c40 2022-07-20T02:35:38.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd4062a70 con 0x7f6bd40a8c40 2022-07-20T02:35:38.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc400e960 con 0x7f6bd40a8c40 2022-07-20T02:35:38.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.298+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd41453d0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb0003410 con 0x7f6bd40a8c40 2022-07-20T02:35:38.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6bc4019030 con 0x7f6bd40a8c40 2022-07-20T02:35:38.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bd27fc700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 0x7f6bb4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6bc406b5b0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bd92e4700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 0x7f6bb4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.302+0000 7f6bd92e4700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 0x7f6bb4055fc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f6bc801a000 tx=0x7f6bc8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.475 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.310+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6bc40424c0 con 0x7f6bd40a8c40 2022-07-20T02:35:38.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.474+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bb0003c40 con 0x7f6bd40a8c40 2022-07-20T02:35:38.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.474+0000 7f6bd27fc700 1 -- 172.21.15.26:0/352337063 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+1553 (secure 0 0 0) 0x7f6bc4069e60 con 0x7f6bd40a8c40 2022-07-20T02:35:38.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:38.528 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7193,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.120:6833/450075075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":450075075},{"type":"v1","addr":"172.21.15.120:6833","nonce":450075075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":77},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/13968230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":13968230},{"type":"v1","addr":"172.21.15.26:6835","nonce":13968230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":76},{"gid":8760,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6837/202857226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":202857226},{"type":"v1","addr":"172.21.15.26:6837","nonce":202857226}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":83}],"filesystems":[]} 2022-07-20T02:35:38.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 msgr2=0x7f6bb4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 0x7f6bb4055fc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f6bc801a000 tx=0x7f6bc8014000).stop 2022-07-20T02:35:38.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 msgr2=0x7f6bd40674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f6bc4001490 tx=0x7f6bc4007eb0).stop 2022-07-20T02:35:38.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 shutdown_connections 2022-07-20T02:35:38.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bd40a8c40 0x7f6bd40674d0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6bb4053b00 0x7f6bb4055fc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bd40b5ef0 0x7f6bd40679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 --2- 172.21.15.26:0/352337063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6bd40a7fc0 0x7f6bd4144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 >> 172.21.15.26:0/352337063 conn(0x7f6bd4060310 msgr2=0x7f6bd40b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 shutdown_connections 2022-07-20T02:35:38.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.478+0000 7f6bda2e6700 1 -- 172.21.15.26:0/352337063 wait complete. 2022-07-20T02:35:38.549 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 83 2022-07-20T02:35:38.551 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T02:35:38.564 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T02:35:38.565 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T02:35:38.566 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:38.566 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-20T02:35:38.567 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-20T02:35:38.567 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T02:35:38.568 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:38.568 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T02:35:38.569 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T02:35:38.569 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T02:35:38.570 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T02:35:38.593 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T02:35:38.594 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T02:35:38.596 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T02:35:38.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T02:35:38.600 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T02:35:38.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-20T02:35:38.603 INFO:tasks.ceph.mds.a:Started 2022-07-20T02:35:38.604 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:38.606 INFO:tasks.ceph.mds.b:Started 2022-07-20T02:35:38.607 INFO:tasks.ceph.mds.c:Started 2022-07-20T02:35:38.608 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:38.608 DEBUG:teuthology.parallel:result is None 2022-07-20T02:35:38.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T02:35:38.682 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2022-07-20T02:35:38.690 INFO:tasks.ceph.mds.a.smithi026.stdout:starting mds.a at 2022-07-20T02:35:38.691 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:35:38.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 Processor -- start 2022-07-20T02:35:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 -- start start 2022-07-20T02:35:38.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c8143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73c8060f00 0x7f73c813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c81474e0 con 0x7f73c805cfd0 2022-07-20T02:35:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c80b4e40 con 0x7f73c8060f00 2022-07-20T02:35:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73cf0b7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c805d3b0 con 0x7f73c8147000 2022-07-20T02:35:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.762+0000 7f73ce8b6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73c8060f00 0x7f73c813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f73c8060f00 0x7f73c813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44214/0 (socket says 172.21.15.26:44214) 2022-07-20T02:35:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 -- 172.21.15.26:0/2640113335 learned_addr learned my addr 172.21.15.26:0/2640113335 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd8b4700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c8143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 -- 172.21.15.26:0/2640113335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 msgr2=0x7f73c8143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:38.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c8143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 -- 172.21.15.26:0/2640113335 >> v1:172.21.15.26:6789/0 conn(0x7f73c805cfd0 legacy=0x7f73c80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd0b3700 1 -- 172.21.15.26:0/2640113335 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690495432 0 0) 0x7f73c81474e0 con 0x7f73c805cfd0 2022-07-20T02:35:38.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 -- 172.21.15.26:0/2640113335 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c81431e0 con 0x7f73c8060f00 2022-07-20T02:35:38.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd8b4700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:38.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73ce8b6700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c8060f00 0x7f73c813c9b0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f73c00032f0 tx=0x7f73c0009290).ready entity=mon.0 client_cookie=37fb19dfacd3d779 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd0b3700 1 -- 172.21.15.26:0/2640113335 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c000b8a0 con 0x7f73c8060f00 2022-07-20T02:35:38.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd0b3700 1 -- 172.21.15.26:0/2640113335 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c000ba00 con 0x7f73c8060f00 2022-07-20T02:35:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cd0b3700 1 -- 172.21.15.26:0/2640113335 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c000bb70 con 0x7f73c8060f00 2022-07-20T02:35:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 -- 172.21.15.26:0/2640113335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c8060f00 msgr2=0x7f73c813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c8060f00 0x7f73c813c9b0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f73c00032f0 tx=0x7f73c0009290).stop 2022-07-20T02:35:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 -- 172.21.15.26:0/2640113335 shutdown_connections 2022-07-20T02:35:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c8060f00 0x7f73c813c9b0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/2640113335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c8143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 -- 172.21.15.26:0/2640113335 >> 172.21.15.26:0/2640113335 conn(0x7f73c80620c0 msgr2=0x7f73c805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:38.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 -- 172.21.15.26:0/2640113335 shutdown_connections 2022-07-20T02:35:38.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 -- 172.21.15.26:0/2640113335 wait complete. 2022-07-20T02:35:38.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.766+0000 7f73cf0b7700 1 Processor -- start 2022-07-20T02:35:38.942 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- start start 2022-07-20T02:35:38.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 0x7f73c80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c80ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.943 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80b3450 con 0x7f73c80b2880 2022-07-20T02:35:38.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80b3680 con 0x7f73c8060f00 2022-07-20T02:35:38.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80b3920 con 0x7f73c8147000 2022-07-20T02:35:38.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.944 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44220/0 (socket says 172.21.15.26:44220) 2022-07-20T02:35:38.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 -- 172.21.15.26:0/3892142636 learned_addr learned my addr 172.21.15.26:0/3892142636 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:38.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce0b5700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 0x7f73c80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cd8b4700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c80ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.945 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 -- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 msgr2=0x7f73c80ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.954 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c80ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 -- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 msgr2=0x7f73c80b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:38.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 0x7f73c80b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:38.955 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 -- 172.21.15.26:0/3892142636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0009530 con 0x7f73c80b2880 2022-07-20T02:35:38.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce0b5700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 0x7f73c80b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:38.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73ce8b6700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f73c00032f0 tx=0x7f73c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c000b8a0 con 0x7f73c80b2880 2022-07-20T02:35:38.956 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c000bb70 con 0x7f73c80b2880 2022-07-20T02:35:38.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c0020620 con 0x7f73c80b2880 2022-07-20T02:35:38.957 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80b3bc0 con 0x7f73c80b2880 2022-07-20T02:35:38.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.770+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c8141820 con 0x7f73c80b2880 2022-07-20T02:35:38.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73c0028070 con 0x7f73c80b2880 2022-07-20T02:35:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73b77fe700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 0x7f73a8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f73c006c7a0 con 0x7f73c80b2880 2022-07-20T02:35:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73ce0b5700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 0x7f73a8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8143480 con 0x7f73c80b2880 2022-07-20T02:35:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.774+0000 7f73ce0b5700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 0x7f73a8055fc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f73c400b000 tx=0x7f73c4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:38.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.778+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73c0015070 con 0x7f73c80b2880 2022-07-20T02:35:38.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:38.918+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f73c80574e0 con 0x7f73c80b2880 2022-07-20T02:35:39.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.722+0000 7f73b77fe700 1 -- 172.21.15.26:0/3892142636 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7f73c0043840 con 0x7f73c80b2880 2022-07-20T02:35:39.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 msgr2=0x7f73a8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:39.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 0x7f73a8055fc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f73c400b000 tx=0x7f73c4009040).stop 2022-07-20T02:35:39.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 msgr2=0x7f73c8152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:39.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f73c00032f0 tx=0x7f73c0016000).stop 2022-07-20T02:35:39.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 shutdown_connections 2022-07-20T02:35:39.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73c80b2880 0x7f73c8152160 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:39.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f73a8053b00 0x7f73a8055fc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:39.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73c8147000 0x7f73c80ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:39.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 --2- 172.21.15.26:0/3892142636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8060f00 0x7f73c80b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:39.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 >> 172.21.15.26:0/3892142636 conn(0x7f73c805cfd0 msgr2=0x7f73c813bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:39.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 shutdown_connections 2022-07-20T02:35:39.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.726+0000 7f73cf0b7700 1 -- 172.21.15.26:0/3892142636 wait complete. 2022-07-20T02:35:39.753 INFO:teuthology.orchestra.run.smithi026.stderr: removed all blacklist entries 2022-07-20T02:35:39.755 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-20T02:35:39.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T02:35:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 Processor -- start 2022-07-20T02:35:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 -- start start 2022-07-20T02:35:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fddc40670c0 0x7fddc40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 0x7fddc414b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc4064880 con 0x7fddc4064450 2022-07-20T02:35:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc40b85f0 con 0x7fddc40670c0 2022-07-20T02:35:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.954+0000 7fddc3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40b80b0 con 0x7fddc40b3380 2022-07-20T02:35:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc2ffd700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fddc40670c0 0x7fddc40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 0x7fddc414b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc27fc700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fddc4064450 0x7fddc4064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:57716/0 (socket says 172.21.15.26:57716) 2022-07-20T02:35:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc27fc700 1 -- 172.21.15.26:0/2538152602 learned_addr learned my addr 172.21.15.26:0/2538152602 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc1ffb700 1 -- 172.21.15.26:0/2538152602 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050475557 0 0) 0x7fddc4064880 con 0x7fddc4064450 2022-07-20T02:35:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc1ffb700 1 -- 172.21.15.26:0/2538152602 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8003260 con 0x7fddc4064450 2022-07-20T02:35:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 -- 172.21.15.26:0/2538152602 >> v2:172.21.15.120:3300/0 conn(0x7fddc40670c0 msgr2=0x7fddc40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 --2- 172.21.15.26:0/2538152602 >> v2:172.21.15.120:3300/0 conn(0x7fddc40670c0 0x7fddc40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 -- 172.21.15.26:0/2538152602 >> v1:172.21.15.120:6789/0 conn(0x7fddc4064450 legacy=0x7fddc4064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 -- 172.21.15.26:0/2538152602 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40b8350 con 0x7fddc40b3380 2022-07-20T02:35:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc37fe700 1 --2- 172.21.15.26:0/2538152602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 0x7fddc414b840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fddbc0032f0 tx=0x7fddbc0091d0).ready entity=mon.2 client_cookie=1e70cd1b8590bd53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc1ffb700 1 -- 172.21.15.26:0/2538152602 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddbc00b600 con 0x7fddc40b3380 2022-07-20T02:35:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc1ffb700 1 -- 172.21.15.26:0/2538152602 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc00b760 con 0x7fddc40b3380 2022-07-20T02:35:40.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc1ffb700 1 -- 172.21.15.26:0/2538152602 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddbc00b8d0 con 0x7fddc40b3380 2022-07-20T02:35:40.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 -- 172.21.15.26:0/2538152602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 msgr2=0x7fddc414b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2538152602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 0x7fddc414b840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fddbc0032f0 tx=0x7fddbc0091d0).stop 2022-07-20T02:35:40.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 -- 172.21.15.26:0/2538152602 shutdown_connections 2022-07-20T02:35:40.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2538152602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40b3380 0x7fddc414b840 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2538152602 >> v2:172.21.15.120:3300/0 conn(0x7fddc40670c0 0x7fddc40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 -- 172.21.15.26:0/2538152602 >> 172.21.15.26:0/2538152602 conn(0x7fddc405f840 msgr2=0x7fddc405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:40.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 -- 172.21.15.26:0/2538152602 shutdown_connections 2022-07-20T02:35:40.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.958+0000 7fddc3fff700 1 -- 172.21.15.26:0/2538152602 wait complete. 2022-07-20T02:35:40.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 Processor -- start 2022-07-20T02:35:40.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- start start 2022-07-20T02:35:40.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40670c0 0x7fddc40b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc41470d0 0x7fddc413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4139c60 con 0x7fddc4146760 2022-07-20T02:35:40.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4139f00 con 0x7fddc41470d0 2022-07-20T02:35:40.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc413a1a0 con 0x7fddc40670c0 2022-07-20T02:35:40.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44230/0 (socket says 172.21.15.26:44230) 2022-07-20T02:35:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 -- 172.21.15.26:0/2073515641 learned_addr learned my addr 172.21.15.26:0/2073515641 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc2ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40670c0 0x7fddc40b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc37fe700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc41470d0 0x7fddc413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 -- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40670c0 msgr2=0x7fddc40b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40670c0 0x7fddc40b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 -- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc41470d0 msgr2=0x7fddc413d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc41470d0 0x7fddc413d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 -- 172.21.15.26:0/2073515641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009470 con 0x7fddc4146760 2022-07-20T02:35:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc27fc700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7fddb4001110 tx=0x7fddb4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddb400ce50 con 0x7fddc4146760 2022-07-20T02:35:40.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb40072e0 con 0x7fddc4146760 2022-07-20T02:35:40.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc40b4e90 con 0x7fddc4146760 2022-07-20T02:35:40.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddb4005800 con 0x7fddc4146760 2022-07-20T02:35:40.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.962+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc4148290 con 0x7fddc4146760 2022-07-20T02:35:40.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc413a360 con 0x7fddc4146760 2022-07-20T02:35:40.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fddb401a060 con 0x7fddc4146760 2022-07-20T02:35:40.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddb3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 0x7fdda0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fddb406b100 con 0x7fddc4146760 2022-07-20T02:35:40.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddc2ffd700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 0x7fdda0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.966+0000 7fddc2ffd700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 0x7fdda0055f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fddb800b000 tx=0x7fddb8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:39.970+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fddb4042200 con 0x7fddc4146760 2022-07-20T02:35:40.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.130+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fddc4064ab0 con 0x7fddc4146760 2022-07-20T02:35:40.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddb3fff700 1 -- 172.21.15.26:0/2073515641 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fddb4018020 con 0x7fddc4146760 2022-07-20T02:35:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:40.145 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:35:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_dump": [ 2022-07-20T02:35:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.a", 2022-07-20T02:35:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dikB/WHBAA8Byc+vhXalWSO20/ExzhLQ==", 2022-07-20T02:35:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.198 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.b", 2022-07-20T02:35:40.199 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diDTUMJxAAgZrLIzz6KszUzpymBqA/Ww==", 2022-07-20T02:35:40.199 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.199 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:40.200 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.200 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.201 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.203 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mds.c", 2022-07-20T02:35:40.203 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diriIbIxAAH/3eoch5Gn3YvxlEDa4Erg==", 2022-07-20T02:35:40.204 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.204 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:40.205 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.205 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.205 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.218 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.0", 2022-07-20T02:35:40.219 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBwZ9di59zCKhAAuzojU1bZPV7m6kcs/3ixGw==", 2022-07-20T02:35:40.219 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.220 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.220 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.220 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.222 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.1", 2022-07-20T02:35:40.222 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBzZ9di5ADqDhAAxxxxf/mMkVcH4iy6nu0uYA==", 2022-07-20T02:35:40.222 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.223 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.223 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.223 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.225 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.2", 2022-07-20T02:35:40.225 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB1Z9diQPUoMBAAQ/yiQLqBlC81axKJzDOD8g==", 2022-07-20T02:35:40.239 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.241 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.245 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.3", 2022-07-20T02:35:40.246 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB4Z9diYvPEGBAAExOLBrXYxEy+pwjCDgV2lA==", 2022-07-20T02:35:40.246 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.255 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.255 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.257 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.4", 2022-07-20T02:35:40.258 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQB9Z9diJQUkLxAAmW4qGCsZ+a6RUhQ5WwqlWg==", 2022-07-20T02:35:40.259 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.259 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.260 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.260 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.262 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.5", 2022-07-20T02:35:40.262 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCAZ9diGVdMFRAAEV2by08z+iu2rK6Ir5SKcQ==", 2022-07-20T02:35:40.263 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.263 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.264 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.264 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.267 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.6", 2022-07-20T02:35:40.267 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCCZ9diGbeeOBAA7gwOvgzLPGizYDBKTYwR0A==", 2022-07-20T02:35:40.268 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.268 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.268 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.269 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.271 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "osd.7", 2022-07-20T02:35:40.272 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQCFZ9diPcCpGhAAOKo2eZ59pm1aQxgYzQhkmQ==", 2022-07-20T02:35:40.272 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.273 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.274 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.274 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.276 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.0", 2022-07-20T02:35:40.276 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dinKShKhAArNWl+4JXqs8er/X373vDjA==", 2022-07-20T02:35:40.277 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.277 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:40.278 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:35:40.278 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:35:40.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.291 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.1", 2022-07-20T02:35:40.291 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diXLofLhAAeTtM4zvpvoFmqshk5SLhKg==", 2022-07-20T02:35:40.292 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.292 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow", 2022-07-20T02:35:40.292 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow r", 2022-07-20T02:35:40.293 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T02:35:40.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.305 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "client.admin", 2022-07-20T02:35:40.315 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBuZ9dipcTDLRAAt5d5XXqoBa51HfYrvrst7Q==", 2022-07-20T02:35:40.315 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.316 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:40.325 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": "allow *", 2022-07-20T02:35:40.347 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow *", 2022-07-20T02:35:40.348 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.375 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.x", 2022-07-20T02:35:40.376 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9diGiBNGhAAO/CdO++plOwFBB3hl+GSCQ==", 2022-07-20T02:35:40.376 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.378 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:40.379 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:35:40.379 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "entity": "mgr.y", 2022-07-20T02:35:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "AQBvZ9dihJztFRAAg0G/J/7Y5Rw4sQEyFo0kwg==", 2022-07-20T02:35:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "caps": { 2022-07-20T02:35:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "mds": "allow *", 2022-07-20T02:35:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": "allow profile mgr", 2022-07-20T02:35:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": "allow *" 2022-07-20T02:35:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:35:40.398 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:35:40.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 msgr2=0x7fdda0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 0x7fdda0055f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fddb800b000 tx=0x7fddb8006df0).stop 2022-07-20T02:35:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 msgr2=0x7fddc4146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7fddb4001110 tx=0x7fddb4007af0).stop 2022-07-20T02:35:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 shutdown_connections 2022-07-20T02:35:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddc4146760 0x7fddc4146be0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdda0053aa0 0x7fdda0055f60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc41470d0 0x7fddc413d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 --2- 172.21.15.26:0/2073515641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc40670c0 0x7fddc40b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 >> 172.21.15.26:0/2073515641 conn(0x7fddc405f840 msgr2=0x7fddc4148e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 shutdown_connections 2022-07-20T02:35:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.134+0000 7fddc3fff700 1 -- 172.21.15.26:0/2073515641 wait complete. 2022-07-20T02:35:40.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 Processor -- start 2022-07-20T02:35:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 -- start start 2022-07-20T02:35:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff9040a7fc0 0x7ff9040af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040b62d0 con 0x7ff904147be0 2022-07-20T02:35:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90414aac0 con 0x7ff9040b5ef0 2022-07-20T02:35:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90a99f700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90414ad60 con 0x7ff9040a7fc0 2022-07-20T02:35:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90919c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff904147be0 0x7ff9040aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43518/0 (socket says 172.21.15.26:43518) 2022-07-20T02:35:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.670+0000 7ff90919c700 1 -- 172.21.15.26:0/1593675804 learned_addr learned my addr 172.21.15.26:0/1593675804 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a19e700 1 --2- 172.21.15.26:0/1593675804 >> v2:172.21.15.120:3300/0 conn(0x7ff9040a7fc0 0x7ff9040af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90899b700 1 -- 172.21.15.26:0/1593675804 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916934473 0 0) 0x7ff9040b62d0 con 0x7ff904147be0 2022-07-20T02:35:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90899b700 1 -- 172.21.15.26:0/1593675804 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec003260 con 0x7ff904147be0 2022-07-20T02:35:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a19e700 1 -- 172.21.15.26:0/1593675804 >> v1:172.21.15.120:6789/0 conn(0x7ff9040b5ef0 legacy=0x7ff9041474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a19e700 1 -- 172.21.15.26:0/1593675804 >> v1:172.21.15.26:6789/0 conn(0x7ff904147be0 legacy=0x7ff9040aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a19e700 1 -- 172.21.15.26:0/1593675804 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90414b000 con 0x7ff9040a7fc0 2022-07-20T02:35:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a19e700 1 --2- 172.21.15.26:0/1593675804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9040a7fc0 0x7ff9040af550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff8fc0032f0 tx=0x7ff8fc0091d0).ready entity=mon.1 client_cookie=c7b2d2b5bb87264b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90899b700 1 -- 172.21.15.26:0/1593675804 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8fc00b6c0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90899b700 1 -- 172.21.15.26:0/1593675804 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc00b820 con 0x7ff9040a7fc0 2022-07-20T02:35:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90899b700 1 -- 172.21.15.26:0/1593675804 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8fc00b990 con 0x7ff9040a7fc0 2022-07-20T02:35:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- 172.21.15.26:0/1593675804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9040a7fc0 msgr2=0x7ff9040af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 --2- 172.21.15.26:0/1593675804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9040a7fc0 0x7ff9040af550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff8fc0032f0 tx=0x7ff8fc0091d0).stop 2022-07-20T02:35:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- 172.21.15.26:0/1593675804 shutdown_connections 2022-07-20T02:35:40.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 --2- 172.21.15.26:0/1593675804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9040a7fc0 0x7ff9040af550 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- 172.21.15.26:0/1593675804 >> 172.21.15.26:0/1593675804 conn(0x7ff904060310 msgr2=0x7ff9040b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:40.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- 172.21.15.26:0/1593675804 shutdown_connections 2022-07-20T02:35:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- 172.21.15.26:0/1593675804 wait complete. 2022-07-20T02:35:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 Processor -- start 2022-07-20T02:35:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.674+0000 7ff90a99f700 1 -- start start 2022-07-20T02:35:40.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9040b5ef0 0x7ff9040aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 0x7ff9040626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040ab3a0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90415a860 con 0x7ff904147be0 2022-07-20T02:35:40.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040b4fe0 con 0x7ff9040b5ef0 2022-07-20T02:35:40.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44244/0 (socket says 172.21.15.26:44244) 2022-07-20T02:35:40.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 -- 172.21.15.26:0/2290530485 learned_addr learned my addr 172.21.15.26:0/2290530485 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:40.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90919c700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9040b5ef0 0x7ff9040aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a19e700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 0x7ff9040626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 -- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9040b5ef0 msgr2=0x7ff9040aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9040b5ef0 0x7ff9040aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 -- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 msgr2=0x7ff9040626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 0x7ff9040626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 -- 172.21.15.26:0/2290530485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc009470 con 0x7ff9040a7fc0 2022-07-20T02:35:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a19e700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 0x7ff9040626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90999d700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7ff90406b790 tx=0x7ff8f4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f400ce30 con 0x7ff9040a7fc0 2022-07-20T02:35:40.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f400f470 con 0x7ff9040a7fc0 2022-07-20T02:35:40.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90415ab00 con 0x7ff9040a7fc0 2022-07-20T02:35:40.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f400f7d0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9040ac170 con 0x7ff9040a7fc0 2022-07-20T02:35:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90415a6a0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.678+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff8f4019030 con 0x7ff9040a7fc0 2022-07-20T02:35:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.682+0000 7ff8f2ffd700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 0x7ff8e4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.682+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff8f406b190 con 0x7ff9040a7fc0 2022-07-20T02:35:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.682+0000 7ff90919c700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 0x7ff8e4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.682+0000 7ff90919c700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 0x7ff8e4055ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7ff90000b000 tx=0x7ff900009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.686+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8f4042290 con 0x7ff9040a7fc0 2022-07-20T02:35:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.814+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff90414b2a0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff8f2ffd700 1 -- 172.21.15.26:0/2290530485 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+13472 (secure 0 0 0) 0x7ff8f40458b0 con 0x7ff9040a7fc0 2022-07-20T02:35:40.927 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:40.928 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":155,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:39.711749+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 msgr2=0x7ff8e4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 0x7ff8e4055ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7ff90000b000 tx=0x7ff900009040).stop 2022-07-20T02:35:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 msgr2=0x7ff9040b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:40.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7ff90406b790 tx=0x7ff8f4007c50).stop 2022-07-20T02:35:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 shutdown_connections 2022-07-20T02:35:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff8e4053b30 0x7ff8e4055ff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff904147be0 0x7ff9040626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9040b5ef0 0x7ff9040aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 --2- 172.21.15.26:0/2290530485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9040a7fc0 0x7ff9040b4370 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 >> 172.21.15.26:0/2290530485 conn(0x7ff904060310 msgr2=0x7ff9040ae970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 shutdown_connections 2022-07-20T02:35:40.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:40.818+0000 7ff90a99f700 1 -- 172.21.15.26:0/2290530485 wait complete. 2022-07-20T02:35:40.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T02:35:40.972 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:41.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 Processor -- start 2022-07-20T02:35:41.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 -- start start 2022-07-20T02:35:41.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07240b2b30 0x7f07241416e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07240b10f0 0x7f0724141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07240b2f10 con 0x7f0724143770 2022-07-20T02:35:41.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07240ad650 con 0x7f07240b10f0 2022-07-20T02:35:41.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07240a9230 con 0x7f07240b2b30 2022-07-20T02:35:41.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07240b10f0 0x7f0724141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f0728b45700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07240b2b30 0x7f07241416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07240b10f0 0x7f0724141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35716/0 (socket says 172.21.15.26:35716) 2022-07-20T02:35:41.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 -- 172.21.15.26:0/1893321786 learned_addr learned my addr 172.21.15.26:0/1893321786 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:41.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 -- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07240b2b30 msgr2=0x7f07241416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:41.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 --2- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07240b2b30 0x7f07241416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 -- 172.21.15.26:0/1893321786 >> v1:172.21.15.120:6789/0 conn(0x7f0724143770 legacy=0x7f0724143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 -- 172.21.15.26:0/1893321786 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724060fb0 con 0x7f07240b10f0 2022-07-20T02:35:41.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071bfff700 1 --2- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b10f0 0x7f0724141bd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f0714002f70 tx=0x7f0714006f40).ready entity=mon.1 client_cookie=4e519a95e5b2c84b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071b7fe700 1 -- 172.21.15.26:0/1893321786 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071400da80 con 0x7f07240b10f0 2022-07-20T02:35:41.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071b7fe700 1 -- 172.21.15.26:0/1893321786 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714000e90 con 0x7f07240b10f0 2022-07-20T02:35:41.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f071b7fe700 1 -- 172.21.15.26:0/1893321786 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071400de10 con 0x7f07240b10f0 2022-07-20T02:35:41.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 -- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b10f0 msgr2=0x7f0724141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.134+0000 7f072ada9700 1 --2- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b10f0 0x7f0724141bd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f0714002f70 tx=0x7f0714006f40).stop 2022-07-20T02:35:41.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- 172.21.15.26:0/1893321786 shutdown_connections 2022-07-20T02:35:41.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 --2- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b10f0 0x7f0724141bd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 --2- 172.21.15.26:0/1893321786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07240b2b30 0x7f07241416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- 172.21.15.26:0/1893321786 >> 172.21.15.26:0/1893321786 conn(0x7f07240b6110 msgr2=0x7f07240b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:41.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- 172.21.15.26:0/1893321786 shutdown_connections 2022-07-20T02:35:41.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- 172.21.15.26:0/1893321786 wait complete. 2022-07-20T02:35:41.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 Processor -- start 2022-07-20T02:35:41.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- start start 2022-07-20T02:35:41.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0724143770 0x7f072414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07241517e0 con 0x7f07240b10f0 2022-07-20T02:35:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0724151a80 con 0x7f07240b2b30 2022-07-20T02:35:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f072ada9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0724151d20 con 0x7f0724143770 2022-07-20T02:35:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f071bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0729346700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0724143770 0x7f072414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44260/0 (socket says 172.21.15.26:44260) 2022-07-20T02:35:41.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 -- 172.21.15.26:0/1756098758 learned_addr learned my addr 172.21.15.26:0/1756098758 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:41.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f071bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35718/0 (socket says 172.21.15.26:35718) 2022-07-20T02:35:41.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 -- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0724143770 msgr2=0x7f072414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0724143770 0x7f072414e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 -- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 msgr2=0x7f07241406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 -- 172.21.15.26:0/1756098758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071400b460 con 0x7f07240b10f0 2022-07-20T02:35:41.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f071bfff700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.138+0000 7f0728b45700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f072000c000 tx=0x7f0720006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0720010780 con 0x7f07240b10f0 2022-07-20T02:35:41.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07200108e0 con 0x7f07240b10f0 2022-07-20T02:35:41.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0724063ce0 con 0x7f07240b10f0 2022-07-20T02:35:41.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0720010c40 con 0x7f07240b10f0 2022-07-20T02:35:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072413f080 con 0x7f07240b10f0 2022-07-20T02:35:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07240574e0 con 0x7f07240b10f0 2022-07-20T02:35:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f072001c460 con 0x7f07240b10f0 2022-07-20T02:35:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.142+0000 7f0719ffb700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 0x7f0704055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.146+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f072006c0d0 con 0x7f07240b10f0 2022-07-20T02:35:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.146+0000 7f071bfff700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 0x7f0704055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.146+0000 7f071bfff700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 0x7f0704055e10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f071401b000 tx=0x7f07240abeb0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.150+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0720043170 con 0x7f07240b10f0 2022-07-20T02:35:41.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.278+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0724151ee0 con 0x7f07240b10f0 2022-07-20T02:35:41.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.278+0000 7f0719ffb700 1 -- 172.21.15.26:0/1756098758 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v155) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0720046790 con 0x7f07240b10f0 2022-07-20T02:35:41.349 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 msgr2=0x7f0704055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 0x7f0704055e10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f071401b000 tx=0x7f07240abeb0).stop 2022-07-20T02:35:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 msgr2=0x7f07241521c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f072000c000 tx=0x7f0720006df0).stop 2022-07-20T02:35:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 shutdown_connections 2022-07-20T02:35:41.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0704053950 0x7f0704055e10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0724143770 0x7f072414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07240b2b30 0x7f07241406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 --2- 172.21.15.26:0/1756098758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07240b10f0 0x7f07241521c0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 >> 172.21.15.26:0/1756098758 conn(0x7f07240b6110 msgr2=0x7f07240aa910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:41.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 shutdown_connections 2022-07-20T02:35:41.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.282+0000 7f072ada9700 1 -- 172.21.15.26:0/1756098758 wait complete. 2022-07-20T02:35:41.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:41.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.542+0000 7f12e359e700 1 Processor -- start 2022-07-20T02:35:41.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.542+0000 7f12e359e700 1 -- start start 2022-07-20T02:35:41.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.542+0000 7f12e359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f12e405c720 0x7f12e4147550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.542+0000 7f12e359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f12e40afe40 0x7f12e4147a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e405d4d0 con 0x7f12e40b1200 2022-07-20T02:35:41.579 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e40a9ff0 con 0x7f12e405c720 2022-07-20T02:35:41.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e4067e60 con 0x7f12e40afe40 2022-07-20T02:35:41.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f12e405c720 0x7f12e4147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f12e405c720 0x7f12e4147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44266/0 (socket says 172.21.15.26:44266) 2022-07-20T02:35:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 -- 172.21.15.26:0/347222665 learned_addr learned my addr 172.21.15.26:0/347222665 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:41.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e1d9b700 1 --2- 172.21.15.26:0/347222665 >> v2:172.21.15.120:3300/0 conn(0x7f12e40afe40 0x7f12e4147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 -- 172.21.15.26:0/347222665 >> v2:172.21.15.120:3300/0 conn(0x7f12e40afe40 msgr2=0x7f12e4147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:41.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 --2- 172.21.15.26:0/347222665 >> v2:172.21.15.120:3300/0 conn(0x7f12e40afe40 0x7f12e4147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 -- 172.21.15.26:0/347222665 >> v1:172.21.15.120:6789/0 conn(0x7f12e40b1200 legacy=0x7f12e40abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 -- 172.21.15.26:0/347222665 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4067bc0 con 0x7f12e405c720 2022-07-20T02:35:41.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e259c700 1 --2- 172.21.15.26:0/347222665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4147550 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f12cc002f70 tx=0x7f12cc006eb0).ready entity=mon.0 client_cookie=c79c3e5ea71a0ef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e159a700 1 -- 172.21.15.26:0/347222665 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12cc00d5c0 con 0x7f12e405c720 2022-07-20T02:35:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e159a700 1 -- 172.21.15.26:0/347222665 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc000e90 con 0x7f12e405c720 2022-07-20T02:35:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e159a700 1 -- 172.21.15.26:0/347222665 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12cc00d950 con 0x7f12e405c720 2022-07-20T02:35:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- 172.21.15.26:0/347222665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 msgr2=0x7f12e4147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 --2- 172.21.15.26:0/347222665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4147550 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f12cc002f70 tx=0x7f12cc006eb0).stop 2022-07-20T02:35:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- 172.21.15.26:0/347222665 shutdown_connections 2022-07-20T02:35:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 --2- 172.21.15.26:0/347222665 >> v2:172.21.15.120:3300/0 conn(0x7f12e40afe40 0x7f12e4147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 --2- 172.21.15.26:0/347222665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4147550 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- 172.21.15.26:0/347222665 >> 172.21.15.26:0/347222665 conn(0x7f12e4060310 msgr2=0x7f12e40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- 172.21.15.26:0/347222665 shutdown_connections 2022-07-20T02:35:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 -- 172.21.15.26:0/347222665 wait complete. 2022-07-20T02:35:41.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.546+0000 7f12e359e700 1 Processor -- start 2022-07-20T02:35:41.641 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- start start 2022-07-20T02:35:41.642 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40afe40 0x7f12e4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e40b1200 0x7f12e41570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e4156800 con 0x7f12e405c720 2022-07-20T02:35:41.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e4159d70 con 0x7f12e40afe40 2022-07-20T02:35:41.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40b4660 con 0x7f12e40b1200 2022-07-20T02:35:41.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44272/0 (socket says 172.21.15.26:44272) 2022-07-20T02:35:41.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 -- 172.21.15.26:0/767234623 learned_addr learned my addr 172.21.15.26:0/767234623 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:41.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e1d9b700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40afe40 0x7f12e4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e2d9d700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e40b1200 0x7f12e41570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 -- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e40b1200 msgr2=0x7f12e41570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e40b1200 0x7f12e41570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 -- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40afe40 msgr2=0x7f12e4156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.649 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40afe40 0x7f12e4156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 -- 172.21.15.26:0/767234623 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc00b1e0 con 0x7f12e405c720 2022-07-20T02:35:41.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e259c700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f12cc002f70 tx=0x7f12cc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12cc011760 con 0x7f12e405c720 2022-07-20T02:35:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc0118c0 con 0x7f12e405c720 2022-07-20T02:35:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e40b4900 con 0x7f12e405c720 2022-07-20T02:35:41.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12cc011c20 con 0x7f12e405c720 2022-07-20T02:35:41.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e40ab8a0 con 0x7f12e405c720 2022-07-20T02:35:41.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.550+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e40b4da0 con 0x7f12e405c720 2022-07-20T02:35:41.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.554+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f12cc025030 con 0x7f12e405c720 2022-07-20T02:35:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.554+0000 7f12db7fe700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 0x7f12c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.554+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f12cc06cdb0 con 0x7f12e405c720 2022-07-20T02:35:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.554+0000 7f12e1d9b700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 0x7f12c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.554+0000 7f12e1d9b700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 0x7f12c8055f30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f12d400b000 tx=0x7f12d40091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.558+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12cc043eb0 con 0x7f12e405c720 2022-07-20T02:35:41.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f12e40aa250 con 0x7f12e405c720 2022-07-20T02:35:41.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12db7fe700 1 -- 172.21.15.26:0/767234623 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v155) v1 ==== 92+0+10 (secure 0 0 0) 0x7f12cc023020 con 0x7f12e405c720 2022-07-20T02:35:41.695 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:41.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 msgr2=0x7f12c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 0x7f12c8055f30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f12d400b000 tx=0x7f12d40091d0).stop 2022-07-20T02:35:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 msgr2=0x7f12e4064770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f12cc002f70 tx=0x7f12cc015000).stop 2022-07-20T02:35:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 shutdown_connections 2022-07-20T02:35:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f12c8053a70 0x7f12c8055f30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12e40b1200 0x7f12e41570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e40afe40 0x7f12e4156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 --2- 172.21.15.26:0/767234623 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f12e405c720 0x7f12e4064770 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 >> 172.21.15.26:0/767234623 conn(0x7f12e4060310 msgr2=0x7f12e41468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 shutdown_connections 2022-07-20T02:35:41.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.686+0000 7f12e359e700 1 -- 172.21.15.26:0/767234623 wait complete. 2022-07-20T02:35:41.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T02:35:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.870+0000 7f69f48a3700 1 Processor -- start 2022-07-20T02:35:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.870+0000 7f69f48a3700 1 -- start start 2022-07-20T02:35:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.870+0000 7f69f48a3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 0x7f69f0060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.870+0000 7f69f48a3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69f00a95a0 0x7f69f00676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.870+0000 7f69f48a3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b1720 0x7f69f00b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00b3640 con 0x7f69f00a95a0 2022-07-20T02:35:41.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00615c0 con 0x7f69f00b3260 2022-07-20T02:35:41.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0061860 con 0x7f69f00b1720 2022-07-20T02:35:41.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69f00a95a0 0x7f69f00676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f69f00a95a0 0x7f69f00676b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44280/0 (socket says 172.21.15.26:44280) 2022-07-20T02:35:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 -- 172.21.15.26:0/1059522229 learned_addr learned my addr 172.21.15.26:0/1059522229 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69effff700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b1720 0x7f69f00b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/1059522229 >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 0x7f69f0060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 -- 172.21.15.26:0/1059522229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b1720 msgr2=0x7f69f00b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b1720 0x7f69f00b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 -- 172.21.15.26:0/1059522229 >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 msgr2=0x7f69f0060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:41.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 --2- 172.21.15.26:0/1059522229 >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 0x7f69f0060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 -- 172.21.15.26:0/1059522229 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0061b00 con 0x7f69f00a95a0 2022-07-20T02:35:41.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/1059522229 >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 0x7f69f0060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:41.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69eeffd700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00a95a0 0x7f69f00676b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f69e0002f70 tx=0x7f69e0006f40).ready entity=mon.0 client_cookie=5616859b68e2905a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:41.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69ee7fc700 1 -- 172.21.15.26:0/1059522229 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e000da80 con 0x7f69f00a95a0 2022-07-20T02:35:41.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69ee7fc700 1 -- 172.21.15.26:0/1059522229 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e0000e90 con 0x7f69f00a95a0 2022-07-20T02:35:41.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69ee7fc700 1 -- 172.21.15.26:0/1059522229 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e000de10 con 0x7f69f00a95a0 2022-07-20T02:35:41.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- 172.21.15.26:0/1059522229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00a95a0 msgr2=0x7f69f00676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:41.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00a95a0 0x7f69f00676b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f69e0002f70 tx=0x7f69e0006f40).stop 2022-07-20T02:35:41.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- 172.21.15.26:0/1059522229 shutdown_connections 2022-07-20T02:35:41.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b1720 0x7f69f00b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 --2- 172.21.15.26:0/1059522229 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00a95a0 0x7f69f00676b0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 --2- 172.21.15.26:0/1059522229 >> v2:172.21.15.120:3300/0 conn(0x7f69f00b3260 0x7f69f0060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:41.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- 172.21.15.26:0/1059522229 >> 172.21.15.26:0/1059522229 conn(0x7f69f00b67f0 msgr2=0x7f69f00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- 172.21.15.26:0/1059522229 shutdown_connections 2022-07-20T02:35:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.874+0000 7f69f48a3700 1 -- 172.21.15.26:0/1059522229 wait complete. 2022-07-20T02:35:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 Processor -- start 2022-07-20T02:35:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- start start 2022-07-20T02:35:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 0x7f69f00b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b3260 0x7f69f0090ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:41.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00afe00 con 0x7f69f00b1720 2022-07-20T02:35:41.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0091190 con 0x7f69f00a95a0 2022-07-20T02:35:41.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0092aa0 con 0x7f69f00b3260 2022-07-20T02:35:41.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44288/0 (socket says 172.21.15.26:44288) 2022-07-20T02:35:42.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 -- 172.21.15.26:0/2992466054 learned_addr learned my addr 172.21.15.26:0/2992466054 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:42.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 0x7f69f00b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69effff700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b3260 0x7f69f0090ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 -- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b3260 msgr2=0x7f69f0090ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b3260 0x7f69f0090ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 -- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 msgr2=0x7f69f00b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 0x7f69f00b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 -- 172.21.15.26:0/2992466054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e000b460 con 0x7f69f00b1720 2022-07-20T02:35:42.033 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 0x7f69f00b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69eeffd700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f69f00912d0 tx=0x7f69f0097e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e001d070 con 0x7f69f00b1720 2022-07-20T02:35:42.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e0010e80 con 0x7f69f00b1720 2022-07-20T02:35:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00afff0 con 0x7f69f00b1720 2022-07-20T02:35:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e0018800 con 0x7f69f00b1720 2022-07-20T02:35:42.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.878+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f0092f70 con 0x7f69f00b1720 2022-07-20T02:35:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f0093130 con 0x7f69f00b1720 2022-07-20T02:35:42.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f69e0018960 con 0x7f69f00b1720 2022-07-20T02:35:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69ecff9700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 0x7f69d0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 0x7f69d0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f69e006d040 con 0x7f69f00b1720 2022-07-20T02:35:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.882+0000 7f69ef7fe700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 0x7f69d0055e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f69e400c000 tx=0x7f69e4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:41.890+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f69e0043fd0 con 0x7f69f00b1720 2022-07-20T02:35:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.018+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f69f0067210 con 0x7f69f00b1720 2022-07-20T02:35:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.018+0000 7f69ecff9700 1 -- 172.21.15.26:0/2992466054 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v155) v1 ==== 84+0+143 (secure 0 0 0) 0x7f69e00475f0 con 0x7f69f00b1720 2022-07-20T02:35:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:42.071 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2022-07-20T02:35:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2022-07-20T02:35:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "device_health_metrics" 2022-07-20T02:35:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:35:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:35:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 2, 2022-07-20T02:35:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": "rbd" 2022-07-20T02:35:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:35:42.073 INFO:teuthology.orchestra.run.smithi026.stdout:] 2022-07-20T02:35:42.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 msgr2=0x7f69d0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 0x7f69d0055e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f69e400c000 tx=0x7f69e4008040).stop 2022-07-20T02:35:42.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 msgr2=0x7f69f00907b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f69f00912d0 tx=0x7f69f0097e70).stop 2022-07-20T02:35:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 shutdown_connections 2022-07-20T02:35:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69f00b1720 0x7f69f00907b0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f69d0053950 0x7f69d0055e10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69f00b3260 0x7f69f0090ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 --2- 172.21.15.26:0/2992466054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69f00a95a0 0x7f69f00b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 >> 172.21.15.26:0/2992466054 conn(0x7f69f00b67f0 msgr2=0x7f69f00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 shutdown_connections 2022-07-20T02:35:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.022+0000 7f69f48a3700 1 -- 172.21.15.26:0/2992466054 wait complete. 2022-07-20T02:35:42.100 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T02:35:42.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T02:35:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 Processor -- start 2022-07-20T02:35:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 -- start start 2022-07-20T02:35:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5bc0b3260 0x7ff5bc060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 0x7ff5bc095f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5bc067dc0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5bc0b3640 con 0x7ff5bc0b3260 2022-07-20T02:35:42.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.222+0000 7ff5c150d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc0615c0 con 0x7ff5bc095b10 2022-07-20T02:35:42.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 0x7ff5bc095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bbfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5bc0b3260 0x7ff5bc060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5b37fe700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff5bc0a95a0 0x7ff5bc0676b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:57788/0 (socket says 172.21.15.26:57788) 2022-07-20T02:35:42.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5b37fe700 1 -- 172.21.15.26:0/1156554323 learned_addr learned my addr 172.21.15.26:0/1156554323 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:42.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bb7fe700 1 -- 172.21.15.26:0/1156554323 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575637381 0 0) 0x7ff5bc067dc0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bb7fe700 1 -- 172.21.15.26:0/1156554323 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c003260 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 -- 172.21.15.26:0/1156554323 >> v2:172.21.15.120:3300/0 conn(0x7ff5bc0b3260 msgr2=0x7ff5bc060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 --2- 172.21.15.26:0/1156554323 >> v2:172.21.15.120:3300/0 conn(0x7ff5bc0b3260 0x7ff5bc060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 -- 172.21.15.26:0/1156554323 >> v1:172.21.15.120:6789/0 conn(0x7ff5bc0a95a0 legacy=0x7ff5bc0676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 -- 172.21.15.26:0/1156554323 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc061860 con 0x7ff5bc095b10 2022-07-20T02:35:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c0d0c700 1 --2- 172.21.15.26:0/1156554323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 0x7ff5bc095f50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff5b40032f0 tx=0x7ff5b40091d0).ready entity=mon.2 client_cookie=7f546c6e01e56ca6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bb7fe700 1 -- 172.21.15.26:0/1156554323 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5b400b600 con 0x7ff5bc095b10 2022-07-20T02:35:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bb7fe700 1 -- 172.21.15.26:0/1156554323 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b400b760 con 0x7ff5bc095b10 2022-07-20T02:35:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5bb7fe700 1 -- 172.21.15.26:0/1156554323 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5b400b8d0 con 0x7ff5bc095b10 2022-07-20T02:35:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 -- 172.21.15.26:0/1156554323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 msgr2=0x7ff5bc095f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 --2- 172.21.15.26:0/1156554323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 0x7ff5bc095f50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff5b40032f0 tx=0x7ff5b40091d0).stop 2022-07-20T02:35:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 -- 172.21.15.26:0/1156554323 shutdown_connections 2022-07-20T02:35:42.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 --2- 172.21.15.26:0/1156554323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc095b10 0x7ff5bc095f50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 --2- 172.21.15.26:0/1156554323 >> v2:172.21.15.120:3300/0 conn(0x7ff5bc0b3260 0x7ff5bc060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 -- 172.21.15.26:0/1156554323 >> 172.21.15.26:0/1156554323 conn(0x7ff5bc0b67f0 msgr2=0x7ff5bc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 -- 172.21.15.26:0/1156554323 shutdown_connections 2022-07-20T02:35:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.226+0000 7ff5c150d700 1 -- 172.21.15.26:0/1156554323 wait complete. 2022-07-20T02:35:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 Processor -- start 2022-07-20T02:35:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- start start 2022-07-20T02:35:42.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 0x7ff5bc05fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 0x7ff5bc150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc1510a0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc092420 con 0x7ff5bc095b10 2022-07-20T02:35:42.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc0926c0 con 0x7ff5bc0b3260 2022-07-20T02:35:42.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44302/0 (socket says 172.21.15.26:44302) 2022-07-20T02:35:42.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 -- 172.21.15.26:0/3387089114 learned_addr learned my addr 172.21.15.26:0/3387089114 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:42.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5bbfff700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 0x7ff5bc05fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c0d0c700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 0x7ff5bc150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 -- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 msgr2=0x7ff5bc150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 0x7ff5bc150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 -- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 msgr2=0x7ff5bc05fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 0x7ff5bc05fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 -- 172.21.15.26:0/3387089114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b4009470 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5bbfff700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 0x7ff5bc05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:42.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c0d0c700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 0x7ff5bc150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:42.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b37fe700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7ff5a4001110 tx=0x7ff5a4007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5a400cea0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a40072e0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5bc092960 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5a4005800 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.230+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5bc0b0100 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5bc092d50 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff5a4010070 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5b9ffb700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 0x7ff5a0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff5a406b530 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5bbfff700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 0x7ff5a0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.234+0000 7ff5bbfff700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 0x7ff5a0055e40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7ff5ac00b000 tx=0x7ff5ac008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.242+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff5a4042590 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.491 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.370+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7ff5bc061b00 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.710+0000 7ff5b9ffb700 1 -- 172.21.15.26:0/3387089114 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v156) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff5a4045bb0 con 0x7ff5bc0a95a0 2022-07-20T02:35:42.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 msgr2=0x7ff5a0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 0x7ff5a0055e40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7ff5ac00b000 tx=0x7ff5ac008040).stop 2022-07-20T02:35:42.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 msgr2=0x7ff5bc1506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7ff5a4001110 tx=0x7ff5a4007b80).stop 2022-07-20T02:35:42.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 shutdown_connections 2022-07-20T02:35:42.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff5bc0a95a0 0x7ff5bc1506c0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff5a0053980 0x7ff5a0055e40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5bc0b3260 0x7ff5bc150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 --2- 172.21.15.26:0/3387089114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc095b10 0x7ff5bc05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.720 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 >> 172.21.15.26:0/3387089114 conn(0x7ff5bc0b67f0 msgr2=0x7ff5bc145c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:42.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 shutdown_connections 2022-07-20T02:35:42.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.714+0000 7ff5c150d700 1 -- 172.21.15.26:0/3387089114 wait complete. 2022-07-20T02:35:42.728 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2022-07-20T02:35:42.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T02:35:42.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.858+0000 7ff73c556700 1 Processor -- start 2022-07-20T02:35:42.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.858+0000 7ff73c556700 1 -- start start 2022-07-20T02:35:42.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff7340b9c80 0x7ff7341431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff7340b5b70 0x7ff7341403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7340ba060 con 0x7ff734062710 2022-07-20T02:35:42.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73412bee0 con 0x7ff7340b9c80 2022-07-20T02:35:42.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff734061f80 con 0x7ff7340b5b70 2022-07-20T02:35:42.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff7340b9c80 0x7ff7341431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.880 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff739af1700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff734062710 0x7ff7340b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43590/0 (socket says 172.21.15.26:43590) 2022-07-20T02:35:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff739af1700 1 -- 172.21.15.26:0/4056006304 learned_addr learned my addr 172.21.15.26:0/4056006304 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73aaf3700 1 --2- 172.21.15.26:0/4056006304 >> v2:172.21.15.120:3300/0 conn(0x7ff7340b5b70 0x7ff7341403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 -- 172.21.15.26:0/4056006304 >> v2:172.21.15.120:3300/0 conn(0x7ff7340b5b70 msgr2=0x7ff7341403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:42.881 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 --2- 172.21.15.26:0/4056006304 >> v2:172.21.15.120:3300/0 conn(0x7ff7340b5b70 0x7ff7341403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.882 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 -- 172.21.15.26:0/4056006304 >> v1:172.21.15.26:6789/0 conn(0x7ff734062710 legacy=0x7ff7340b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff7392f0700 1 -- 172.21.15.26:0/4056006304 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507436070 0 0) 0x7ff7340ba060 con 0x7ff734062710 2022-07-20T02:35:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 -- 172.21.15.26:0/4056006304 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734062220 con 0x7ff7340b9c80 2022-07-20T02:35:42.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73a2f2700 1 --2- 172.21.15.26:0/4056006304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341431a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7ff730002f70 tx=0x7ff730006df0).ready entity=mon.0 client_cookie=594a97b1fb3151a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff7392f0700 1 -- 172.21.15.26:0/4056006304 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff73000d570 con 0x7ff7340b9c80 2022-07-20T02:35:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff7392f0700 1 -- 172.21.15.26:0/4056006304 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff730000e90 con 0x7ff7340b9c80 2022-07-20T02:35:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff7392f0700 1 -- 172.21.15.26:0/4056006304 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff73000d8c0 con 0x7ff7340b9c80 2022-07-20T02:35:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- 172.21.15.26:0/4056006304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 msgr2=0x7ff7341431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 --2- 172.21.15.26:0/4056006304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341431a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7ff730002f70 tx=0x7ff730006df0).stop 2022-07-20T02:35:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- 172.21.15.26:0/4056006304 shutdown_connections 2022-07-20T02:35:42.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 --2- 172.21.15.26:0/4056006304 >> v2:172.21.15.120:3300/0 conn(0x7ff7340b5b70 0x7ff7341403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 --2- 172.21.15.26:0/4056006304 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341431a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- 172.21.15.26:0/4056006304 >> 172.21.15.26:0/4056006304 conn(0x7ff734066f70 msgr2=0x7ff73405faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:42.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- 172.21.15.26:0/4056006304 shutdown_connections 2022-07-20T02:35:42.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.862+0000 7ff73c556700 1 -- 172.21.15.26:0/4056006304 wait complete. 2022-07-20T02:35:42.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 Processor -- start 2022-07-20T02:35:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 -- start start 2022-07-20T02:35:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff734062710 0x7ff7340b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b5b70 0x7ff7340b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340b3b40 con 0x7ff7340b9c80 2022-07-20T02:35:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340b3de0 con 0x7ff734062710 2022-07-20T02:35:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73c556700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340b1340 con 0x7ff7340b5b70 2022-07-20T02:35:42.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44318/0 (socket says 172.21.15.26:44318) 2022-07-20T02:35:42.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 -- 172.21.15.26:0/3257548968 learned_addr learned my addr 172.21.15.26:0/3257548968 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:42.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73a2f2700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff734062710 0x7ff7340b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff739af1700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b5b70 0x7ff7340b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 -- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b5b70 msgr2=0x7ff7340b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b5b70 0x7ff7340b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 -- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff734062710 msgr2=0x7ff7340b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:42.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff734062710 0x7ff7340b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:42.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 -- 172.21.15.26:0/3257548968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73000b180 con 0x7ff7340b9c80 2022-07-20T02:35:42.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.866+0000 7ff73aaf3700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7ff72c006e00 tx=0x7ff72c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c00c640 con 0x7ff7340b9c80 2022-07-20T02:35:42.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c00c7a0 con 0x7ff7340b9c80 2022-07-20T02:35:42.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7340b15e0 con 0x7ff7340b9c80 2022-07-20T02:35:42.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c00cb00 con 0x7ff7340b9c80 2022-07-20T02:35:42.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734146bc0 con 0x7ff7340b9c80 2022-07-20T02:35:42.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7340b6650 con 0x7ff7340b9c80 2022-07-20T02:35:42.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff72c00cc60 con 0x7ff7340b9c80 2022-07-20T02:35:42.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 0x7ff714055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:42.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5954+0+0 (secure 0 0 0) 0x7ff72c06c8b0 con 0x7ff7340b9c80 2022-07-20T02:35:42.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.870+0000 7ff73a2f2700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 0x7ff714055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:42.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.874+0000 7ff73a2f2700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 0x7ff714055ff0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7ff73001a000 tx=0x7ff730014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:42.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:42.878+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff72c043840 con 0x7ff7340b9c80 2022-07-20T02:35:43.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.006+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7ff7340574e0 con 0x7ff7340b9c80 2022-07-20T02:35:43.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.742+0000 7ff7237fe700 1 -- 172.21.15.26:0/3257548968 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v159) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff72c046e60 con 0x7ff7340b9c80 2022-07-20T02:35:43.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 msgr2=0x7ff714055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:43.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 0x7ff714055ff0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7ff73001a000 tx=0x7ff730014000).stop 2022-07-20T02:35:43.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 msgr2=0x7ff7341557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:43.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7ff72c006e00 tx=0x7ff72c0091d0).stop 2022-07-20T02:35:43.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 shutdown_connections 2022-07-20T02:35:43.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7340b9c80 0x7ff7341557a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff714053b30 0x7ff714055ff0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7340b5b70 0x7ff7340b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 --2- 172.21.15.26:0/3257548968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff734062710 0x7ff7340b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.746+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 >> 172.21.15.26:0/3257548968 conn(0x7ff734066f70 msgr2=0x7ff73405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:43.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.750+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 shutdown_connections 2022-07-20T02:35:43.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.750+0000 7ff73c556700 1 -- 172.21.15.26:0/3257548968 wait complete. 2022-07-20T02:35:43.792 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2022-07-20T02:35:43.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T02:35:43.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.958+0000 7f1d6b069700 1 Processor -- start 2022-07-20T02:35:43.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.958+0000 7f1d6b069700 1 -- start start 2022-07-20T02:35:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6b069700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1d640670c0 0x7f1d640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6b069700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d640b3380 0x7f1d6414b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6b069700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d64064880 con 0x7f1d64064450 2022-07-20T02:35:43.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6b069700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d640b85f0 con 0x7f1d640670c0 2022-07-20T02:35:43.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6b069700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d640b80b0 con 0x7f1d640b3380 2022-07-20T02:35:43.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1d640670c0 0x7f1d640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:43.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1d640670c0 0x7f1d640b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44328/0 (socket says 172.21.15.26:44328) 2022-07-20T02:35:43.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 -- 172.21.15.26:0/4200287755 learned_addr learned my addr 172.21.15.26:0/4200287755 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:43.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a868700 1 --2- 172.21.15.26:0/4200287755 >> v2:172.21.15.120:3300/0 conn(0x7f1d640b3380 0x7f1d6414b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:43.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 -- 172.21.15.26:0/4200287755 >> v2:172.21.15.120:3300/0 conn(0x7f1d640b3380 msgr2=0x7f1d6414b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:43.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 --2- 172.21.15.26:0/4200287755 >> v2:172.21.15.120:3300/0 conn(0x7f1d640b3380 0x7f1d6414b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 -- 172.21.15.26:0/4200287755 >> v1:172.21.15.120:6789/0 conn(0x7f1d64064450 legacy=0x7f1d64064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:43.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 -- 172.21.15.26:0/4200287755 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640b8350 con 0x7f1d640670c0 2022-07-20T02:35:43.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d6a067700 1 --2- 172.21.15.26:0/4200287755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d640670c0 0x7f1d640b2e90 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f1d60002f70 tx=0x7f1d60006eb0).ready entity=mon.0 client_cookie=a9610f7ba0fd6721 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:43.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d69065700 1 -- 172.21.15.26:0/4200287755 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6000d5c0 con 0x7f1d640670c0 2022-07-20T02:35:43.975 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d69065700 1 -- 172.21.15.26:0/4200287755 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d60000e90 con 0x7f1d640670c0 2022-07-20T02:35:43.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.962+0000 7f1d69065700 1 -- 172.21.15.26:0/4200287755 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6000d950 con 0x7f1d640670c0 2022-07-20T02:35:43.976 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- 172.21.15.26:0/4200287755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d640670c0 msgr2=0x7f1d640b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- 172.21.15.26:0/4200287755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d640670c0 0x7f1d640b2e90 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f1d60002f70 tx=0x7f1d60006eb0).stop 2022-07-20T02:35:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- 172.21.15.26:0/4200287755 shutdown_connections 2022-07-20T02:35:43.977 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- 172.21.15.26:0/4200287755 >> v2:172.21.15.120:3300/0 conn(0x7f1d640b3380 0x7f1d6414b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- 172.21.15.26:0/4200287755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d640670c0 0x7f1d640b2e90 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:43.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- 172.21.15.26:0/4200287755 >> 172.21.15.26:0/4200287755 conn(0x7f1d6405f840 msgr2=0x7f1d6405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:43.978 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- 172.21.15.26:0/4200287755 shutdown_connections 2022-07-20T02:35:43.979 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- 172.21.15.26:0/4200287755 wait complete. 2022-07-20T02:35:43.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 Processor -- start 2022-07-20T02:35:43.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- start start 2022-07-20T02:35:43.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d640670c0 0x7f1d64153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:43.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d640ada90 0x7f1d640b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64150230 con 0x7f1d64150900 2022-07-20T02:35:44.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d640ab590 con 0x7f1d640ada90 2022-07-20T02:35:44.000 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6b069700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64150460 con 0x7f1d640670c0 2022-07-20T02:35:44.008 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6a868700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6a868700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44336/0 (socket says 172.21.15.26:44336) 2022-07-20T02:35:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6a067700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d640670c0 0x7f1d64153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d6a868700 1 -- 172.21.15.26:0/872321824 learned_addr learned my addr 172.21.15.26:0/872321824 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.966+0000 7f1d69866700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d640ada90 0x7f1d640b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 -- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d640670c0 msgr2=0x7f1d64153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d640670c0 0x7f1d64153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 -- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d640ada90 msgr2=0x7f1d640b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d640ada90 0x7f1d640b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 -- 172.21.15.26:0/872321824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6000b1e0 con 0x7f1d64150900 2022-07-20T02:35:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6a868700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f1d5c006e00 tx=0x7f1d5c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d5c012600 con 0x7f1d64150900 2022-07-20T02:35:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c012760 con 0x7f1d64150900 2022-07-20T02:35:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d640b16e0 con 0x7f1d64150900 2022-07-20T02:35:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d5c012ac0 con 0x7f1d64150900 2022-07-20T02:35:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d64143960 con 0x7f1d64150900 2022-07-20T02:35:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d64150620 con 0x7f1d64150900 2022-07-20T02:35:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.970+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1d5c022070 con 0x7f1d64150900 2022-07-20T02:35:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.974+0000 7f1d537fe700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 0x7f1d44055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.974+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f1d5c06c620 con 0x7f1d64150900 2022-07-20T02:35:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.974+0000 7f1d6a067700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 0x7f1d44055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.974+0000 7f1d6a067700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 0x7f1d44055f60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f1d6001c000 tx=0x7f1d60015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:43.982+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d5c043410 con 0x7f1d64150900 2022-07-20T02:35:44.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.138+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1d64064ab0 con 0x7f1d64150900 2022-07-20T02:35:44.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.150+0000 7f1d537fe700 1 -- 172.21.15.26:0/872321824 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v86) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1d5c016020 con 0x7f1d64150900 2022-07-20T02:35:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 msgr2=0x7f1d44055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 0x7f1d44055f60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f1d6001c000 tx=0x7f1d60015000).stop 2022-07-20T02:35:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 msgr2=0x7f1d640acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f1d5c006e00 tx=0x7f1d5c0091d0).stop 2022-07-20T02:35:44.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 shutdown_connections 2022-07-20T02:35:44.164 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1d64150900 0x7f1d640acdd0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1d44053aa0 0x7f1d44055f60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d640ada90 0x7f1d640b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 --2- 172.21.15.26:0/872321824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d640670c0 0x7f1d64153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 >> 172.21.15.26:0/872321824 conn(0x7f1d6405f840 msgr2=0x7f1d640b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:44.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 shutdown_connections 2022-07-20T02:35:44.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.154+0000 7f1d6b069700 1 -- 172.21.15.26:0/872321824 wait complete. 2022-07-20T02:35:44.168 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 29 and data pool 30 2022-07-20T02:35:44.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 Processor -- start 2022-07-20T02:35:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- start start 2022-07-20T02:35:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1afc0b2920 0x7f1afc147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0b1bc0 con 0x7f1afc066a80 2022-07-20T02:35:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc14b040 con 0x7f1afc0b2920 2022-07-20T02:35:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0a8860 con 0x7f1afc061050 2022-07-20T02:35:44.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1afc0b2920 0x7f1afc147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.417 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1afc0b2920 0x7f1afc147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44346/0 (socket says 172.21.15.26:44346) 2022-07-20T02:35:44.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 -- 172.21.15.26:0/1465517577 learned_addr learned my addr 172.21.15.26:0/1465517577 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:44.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1afbfff700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 -- 172.21.15.26:0/1465517577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 msgr2=0x7f1afc148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:44.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 -- 172.21.15.26:0/1465517577 >> v1:172.21.15.120:6789/0 conn(0x7f1afc066a80 legacy=0x7f1afc066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.422 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 -- 172.21.15.26:0/1465517577 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc147700 con 0x7f1afc0b2920 2022-07-20T02:35:44.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b00884700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc0b2920 0x7f1afc147d90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f1ae4002f70 tx=0x7f1ae4006eb0).ready entity=mon.0 client_cookie=55da3a087cc87e9e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1afb7fe700 1 -- 172.21.15.26:0/1465517577 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae400d5c0 con 0x7f1afc0b2920 2022-07-20T02:35:44.423 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1afb7fe700 1 -- 172.21.15.26:0/1465517577 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae4000e90 con 0x7f1afc0b2920 2022-07-20T02:35:44.424 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1afb7fe700 1 -- 172.21.15.26:0/1465517577 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae400d950 con 0x7f1afc0b2920 2022-07-20T02:35:44.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1465517577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc0b2920 msgr2=0x7f1afc147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc0b2920 0x7f1afc147d90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f1ae4002f70 tx=0x7f1ae4006eb0).stop 2022-07-20T02:35:44.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1465517577 shutdown_connections 2022-07-20T02:35:44.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1465517577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc0b2920 0x7f1afc147d90 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1465517577 >> 172.21.15.26:0/1465517577 conn(0x7f1afc0b60a0 msgr2=0x7f1afc0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:44.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1465517577 shutdown_connections 2022-07-20T02:35:44.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.330+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1465517577 wait complete. 2022-07-20T02:35:44.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 Processor -- start 2022-07-20T02:35:44.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- start start 2022-07-20T02:35:44.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1afc0b2920 0x7f1afc1573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc156a70 con 0x7f1afc066a80 2022-07-20T02:35:44.464 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0aa070 con 0x7f1afc0b2920 2022-07-20T02:35:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0a95f0 con 0x7f1afc061050 2022-07-20T02:35:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.495 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44354/0 (socket says 172.21.15.26:44354) 2022-07-20T02:35:44.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 -- 172.21.15.26:0/1326048798 learned_addr learned my addr 172.21.15.26:0/1326048798 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:44.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b00884700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b01085700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1afc0b2920 0x7f1afc1573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.496 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 -- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 msgr2=0x7f1afc0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 -- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1afc0b2920 msgr2=0x7f1afc1573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1afc0b2920 0x7f1afc1573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.497 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 -- 172.21.15.26:0/1326048798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae400b1e0 con 0x7f1afc066a80 2022-07-20T02:35:44.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1afbfff700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f1af400a000 tx=0x7f1af4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1af4013620 con 0x7f1afc066a80 2022-07-20T02:35:44.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af4013780 con 0x7f1afc066a80 2022-07-20T02:35:44.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc0a9890 con 0x7f1afc066a80 2022-07-20T02:35:44.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1af4013ae0 con 0x7f1afc066a80 2022-07-20T02:35:44.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.334+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc0a9dd0 con 0x7f1afc066a80 2022-07-20T02:35:44.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc14ada0 con 0x7f1afc066a80 2022-07-20T02:35:44.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1af4013c40 con 0x7f1afc066a80 2022-07-20T02:35:44.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1af9ffb700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 0x7f1ae0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1b00884700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 0x7f1ae0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1af406be40 con 0x7f1afc066a80 2022-07-20T02:35:44.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.338+0000 7f1b00884700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 0x7f1ae0055fc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f1ae401c000 tx=0x7f1ae4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.342+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1af4042ba0 con 0x7f1afc066a80 2022-07-20T02:35:44.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.470+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1afc0574e0 con 0x7f1afc066a80 2022-07-20T02:35:44.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1af9ffb700 1 -- 172.21.15.26:0/1326048798 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16346 (secure 0 0 0) 0x7f1af40461c0 con 0x7f1afc066a80 2022-07-20T02:35:44.511 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:44.541 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":160,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:44.142199+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:42.427470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:43.067323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 msgr2=0x7f1ae0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 0x7f1ae0055fc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f1ae401c000 tx=0x7f1ae4015000).stop 2022-07-20T02:35:44.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 msgr2=0x7f1afc156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f1af400a000 tx=0x7f1af4008040).stop 2022-07-20T02:35:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 shutdown_connections 2022-07-20T02:35:44.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1afc066a80 0x7f1afc156f00 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1ae0053b00 0x7f1ae0055fc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1afc0b2920 0x7f1afc1573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 --2- 172.21.15.26:0/1326048798 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1afc061050 0x7f1afc0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 >> 172.21.15.26:0/1326048798 conn(0x7f1afc0b60a0 msgr2=0x7f1afc0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 shutdown_connections 2022-07-20T02:35:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.474+0000 7f1b02ae8700 1 -- 172.21.15.26:0/1326048798 wait complete. 2022-07-20T02:35:44.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.714+0000 7fdbaf59e700 1 Processor -- start 2022-07-20T02:35:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.714+0000 7fdbaf59e700 1 -- start start 2022-07-20T02:35:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdbb0147000 0x7fdbb0143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 0x7fdbb013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb01474e0 con 0x7fdbb005cfd0 2022-07-20T02:35:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00b4e40 con 0x7fdbb0147000 2022-07-20T02:35:44.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb005d3b0 con 0x7fdbb0060f00 2022-07-20T02:35:44.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbae59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdbb005cfd0 0x7fdbb00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43640/0 (socket says 172.21.15.26:43640) 2022-07-20T02:35:44.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbae59c700 1 -- 172.21.15.26:0/3420977417 learned_addr learned my addr 172.21.15.26:0/3420977417 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:44.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 --2- 172.21.15.26:0/3420977417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 0x7fdbb013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbadd9b700 1 --2- 172.21.15.26:0/3420977417 >> v2:172.21.15.120:3300/0 conn(0x7fdbb0147000 0x7fdbb0143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322419596 0 0) 0x7fdbb01474e0 con 0x7fdbb005cfd0 2022-07-20T02:35:44.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94003260 con 0x7fdbb005cfd0 2022-07-20T02:35:44.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 -- 172.21.15.26:0/3420977417 >> v2:172.21.15.120:3300/0 conn(0x7fdbb0147000 msgr2=0x7fdbb0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:44.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384937629 0 0) 0x7fdb94003260 con 0x7fdbb005cfd0 2022-07-20T02:35:44.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 --2- 172.21.15.26:0/3420977417 >> v2:172.21.15.120:3300/0 conn(0x7fdbb0147000 0x7fdbb0143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 -- 172.21.15.26:0/3420977417 >> v1:172.21.15.26:6789/0 conn(0x7fdbb005cfd0 legacy=0x7fdbb00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 -- 172.21.15.26:0/3420977417 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb01431e0 con 0x7fdbb0060f00 2022-07-20T02:35:44.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaed9d700 1 --2- 172.21.15.26:0/3420977417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 0x7fdbb013c9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdba0001b80 tx=0x7fdba8006df0).ready entity=mon.2 client_cookie=7180802290b86e24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdba00019e0 con 0x7fdbb0060f00 2022-07-20T02:35:44.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba8000e90 con 0x7fdbb0060f00 2022-07-20T02:35:44.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbad59a700 1 -- 172.21.15.26:0/3420977417 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdba8012460 con 0x7fdbb0060f00 2022-07-20T02:35:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3420977417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 msgr2=0x7fdbb013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3420977417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 0x7fdbb013c9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdba0001b80 tx=0x7fdba8006df0).stop 2022-07-20T02:35:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3420977417 shutdown_connections 2022-07-20T02:35:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3420977417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0060f00 0x7fdbb013c9b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3420977417 >> v2:172.21.15.120:3300/0 conn(0x7fdbb0147000 0x7fdbb0143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3420977417 >> 172.21.15.26:0/3420977417 conn(0x7fdbb00620c0 msgr2=0x7fdbb005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3420977417 shutdown_connections 2022-07-20T02:35:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.718+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3420977417 wait complete. 2022-07-20T02:35:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 Processor -- start 2022-07-20T02:35:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 -- start start 2022-07-20T02:35:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 0x7fdbb0144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 0x7fdbb0137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb00b2b70 0x7fdbb0137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0146410 con 0x7fdbb0060f00 2022-07-20T02:35:44.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0146640 con 0x7fdbb00b2b70 2022-07-20T02:35:44.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.722+0000 7fdbaf59e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb01468e0 con 0x7fdbb0147000 2022-07-20T02:35:44.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbadd9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 0x7fdbb0137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbaed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb00b2b70 0x7fdbb0137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 0x7fdbb0144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbadd9b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 0x7fdbb0137010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37650/0 (socket says 172.21.15.26:37650) 2022-07-20T02:35:44.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbadd9b700 1 -- 172.21.15.26:0/3947427821 learned_addr learned my addr 172.21.15.26:0/3947427821 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:44.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 -- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 msgr2=0x7fdbb0137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 0x7fdbb0137010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 -- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb00b2b70 msgr2=0x7fdbb0137500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb00b2b70 0x7fdbb0137500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 -- 172.21.15.26:0/3947427821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba800b180 con 0x7fdbb0060f00 2022-07-20T02:35:44.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbae59c700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 0x7fdbb0144760 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fdba0001c40 tx=0x7fdba0002160).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdba0004ab0 con 0x7fdbb0060f00 2022-07-20T02:35:44.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0004c10 con 0x7fdbb0060f00 2022-07-20T02:35:44.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0146b80 con 0x7fdbb0060f00 2022-07-20T02:35:44.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdba0007170 con 0x7fdbb0060f00 2022-07-20T02:35:44.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0144e80 con 0x7fdbb0060f00 2022-07-20T02:35:44.866 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb0143480 con 0x7fdbb0060f00 2022-07-20T02:35:44.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.726+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdba001b070 con 0x7fdbb0060f00 2022-07-20T02:35:44.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.730+0000 7fdba77fe700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 0x7fdb88055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:44.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.730+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdba006b910 con 0x7fdbb0060f00 2022-07-20T02:35:44.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.730+0000 7fdbadd9b700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 0x7fdb88055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:44.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.730+0000 7fdbadd9b700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 0x7fdb88055ff0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fdb9800b000 tx=0x7fdb98006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:44.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.734+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdba00426d0 con 0x7fdbb0060f00 2022-07-20T02:35:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.858+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbb00574e0 con 0x7fdbb0060f00 2022-07-20T02:35:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdba77fe700 1 -- 172.21.15.26:0/3947427821 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16346 (secure 0 0 0) 0x7fdba0045cf0 con 0x7fdbb0060f00 2022-07-20T02:35:44.888 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:44.889 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":160,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:44.142199+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:42.427470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:43.067323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:44.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 msgr2=0x7fdb88055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 0x7fdb88055ff0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fdb9800b000 tx=0x7fdb98006df0).stop 2022-07-20T02:35:44.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 msgr2=0x7fdbb0144760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:44.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 0x7fdbb0144760 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fdba0001c40 tx=0x7fdba0002160).stop 2022-07-20T02:35:44.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 shutdown_connections 2022-07-20T02:35:44.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdb88053b30 0x7fdb88055ff0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb00b2b70 0x7fdbb0137500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb0147000 0x7fdbb0137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 --2- 172.21.15.26:0/3947427821 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdbb0060f00 0x7fdbb0144760 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 >> 172.21.15.26:0/3947427821 conn(0x7fdbb005cfd0 msgr2=0x7fdbb00b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 shutdown_connections 2022-07-20T02:35:44.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:44.862+0000 7fdbaf59e700 1 -- 172.21.15.26:0/3947427821 wait complete. 2022-07-20T02:35:44.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T02:35:45.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.058+0000 7f5955bcc700 1 Processor -- start 2022-07-20T02:35:45.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.058+0000 7f5955bcc700 1 -- start start 2022-07-20T02:35:45.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5955bcc700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59500b24c0 0x7f59500604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:45.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5955bcc700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59500b1ae0 con 0x7f59501465b0 2022-07-20T02:35:45.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5955bcc700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59500b1d80 con 0x7f595005f840 2022-07-20T02:35:45.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5955bcc700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59500b2e20 con 0x7f59500b24c0 2022-07-20T02:35:45.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5954bca700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f59501465b0 0x7f5950066db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43654/0 (socket says 172.21.15.26:43654) 2022-07-20T02:35:45.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f5954bca700 1 -- 172.21.15.26:0/188871610 learned_addr learned my addr 172.21.15.26:0/188871610 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:45.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594ffff700 1 --2- 172.21.15.26:0/188871610 >> v2:172.21.15.120:3300/0 conn(0x7f59500b24c0 0x7f59500604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:45.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594f7fe700 1 -- 172.21.15.26:0/188871610 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930378324 0 0) 0x7f59500b1ae0 con 0x7f59501465b0 2022-07-20T02:35:45.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594f7fe700 1 -- 172.21.15.26:0/188871610 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5930003260 con 0x7f59501465b0 2022-07-20T02:35:45.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594ffff700 1 -- 172.21.15.26:0/188871610 >> v1:172.21.15.120:6789/0 conn(0x7f595005f840 legacy=0x7f5950146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:45.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594ffff700 1 -- 172.21.15.26:0/188871610 >> v1:172.21.15.26:6789/0 conn(0x7f59501465b0 legacy=0x7f5950066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:45.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594ffff700 1 -- 172.21.15.26:0/188871610 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59500b2020 con 0x7f59500b24c0 2022-07-20T02:35:45.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594ffff700 1 --2- 172.21.15.26:0/188871610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500b24c0 0x7f59500604e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f5938002f70 tx=0x7f5938006df0).ready entity=mon.1 client_cookie=2384519bfa0312d3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:45.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594f7fe700 1 -- 172.21.15.26:0/188871610 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f593800d610 con 0x7f59500b24c0 2022-07-20T02:35:45.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594f7fe700 1 -- 172.21.15.26:0/188871610 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938000e90 con 0x7f59500b24c0 2022-07-20T02:35:45.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.062+0000 7f594f7fe700 1 -- 172.21.15.26:0/188871610 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f593800d960 con 0x7f59500b24c0 2022-07-20T02:35:45.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- 172.21.15.26:0/188871610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500b24c0 msgr2=0x7f59500604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:45.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 --2- 172.21.15.26:0/188871610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500b24c0 0x7f59500604e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f5938002f70 tx=0x7f5938006df0).stop 2022-07-20T02:35:45.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- 172.21.15.26:0/188871610 shutdown_connections 2022-07-20T02:35:45.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 --2- 172.21.15.26:0/188871610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500b24c0 0x7f59500604e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:45.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- 172.21.15.26:0/188871610 >> 172.21.15.26:0/188871610 conn(0x7f59500b3050 msgr2=0x7f59500b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:45.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- 172.21.15.26:0/188871610 shutdown_connections 2022-07-20T02:35:45.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- 172.21.15.26:0/188871610 wait complete. 2022-07-20T02:35:45.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 Processor -- start 2022-07-20T02:35:45.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- start start 2022-07-20T02:35:45.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:45.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 0x7f59500ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:45.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500ac8d0 0x7f59500ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:45.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.066+0000 7f5955bcc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59501563d0 con 0x7f595005f840 2022-07-20T02:35:45.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5955bcc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950156630 con 0x7f59500ac8d0 2022-07-20T02:35:45.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5955bcc700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5950063980 con 0x7f59501465b0 2022-07-20T02:35:45.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:45.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44380/0 (socket says 172.21.15.26:44380) 2022-07-20T02:35:45.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 -- 172.21.15.26:0/2612057413 learned_addr learned my addr 172.21.15.26:0/2612057413 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:45.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594ffff700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 0x7f59500ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:45.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f59553cb700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500ac8d0 0x7f59500ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:45.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 -- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 msgr2=0x7f59500ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:45.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 0x7f59500ab350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:45.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 -- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500ac8d0 msgr2=0x7f59500ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:45.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500ac8d0 0x7f59500ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:45.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 -- 172.21.15.26:0/2612057413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593800b180 con 0x7f595005f840 2022-07-20T02:35:45.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594ffff700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 0x7f59500ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:45.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5954bca700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f5940001110 tx=0x7f5940007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:45.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59400072e0 con 0x7f595005f840 2022-07-20T02:35:45.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5940007440 con 0x7f595005f840 2022-07-20T02:35:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59500ad010 con 0x7f595005f840 2022-07-20T02:35:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5940005800 con 0x7f595005f840 2022-07-20T02:35:45.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59500ac690 con 0x7f595005f840 2022-07-20T02:35:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f594001a060 con 0x7f595005f840 2022-07-20T02:35:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.070+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59500ad1a0 con 0x7f595005f840 2022-07-20T02:35:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.074+0000 7f594dffb700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 0x7f5934055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.074+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f594006b6d0 con 0x7f595005f840 2022-07-20T02:35:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.074+0000 7f594ffff700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 0x7f5934055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.074+0000 7f594ffff700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 0x7f5934055fc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f5938018000 tx=0x7f5938014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.082+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5940042380 con 0x7f595005f840 2022-07-20T02:35:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:45.246+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5950063b40 con 0x7f595005f840 2022-07-20T02:35:46.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.166+0000 7f594dffb700 1 -- 172.21.15.26:0/2612057413 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5940018020 con 0x7f595005f840 2022-07-20T02:35:46.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 msgr2=0x7f5934055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 0x7f5934055fc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f5938018000 tx=0x7f5938014000).stop 2022-07-20T02:35:46.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 msgr2=0x7f595013fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f5940001110 tx=0x7f5940007b80).stop 2022-07-20T02:35:46.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 shutdown_connections 2022-07-20T02:35:46.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5934053b00 0x7f5934055fc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59500ac8d0 0x7f59500ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59501465b0 0x7f59500ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 --2- 172.21.15.26:0/2612057413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f595005f840 0x7f595013fcc0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 >> 172.21.15.26:0/2612057413 conn(0x7f59500b3050 msgr2=0x7f5950146990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:46.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 shutdown_connections 2022-07-20T02:35:46.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.170+0000 7f5955bcc700 1 -- 172.21.15.26:0/2612057413 wait complete. 2022-07-20T02:35:46.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:46.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 Processor -- start 2022-07-20T02:35:46.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 -- start start 2022-07-20T02:35:46.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f405c0b5ef0 0x7f405c1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0b62d0 con 0x7f405c0a7fc0 2022-07-20T02:35:46.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c14aac0 con 0x7f405c0a8c40 2022-07-20T02:35:46.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4062f30700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c14ad60 con 0x7f405c0b5ef0 2022-07-20T02:35:46.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f406172d700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f405c0a7fc0 0x7f405c0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43668/0 (socket says 172.21.15.26:43668) 2022-07-20T02:35:46.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f406172d700 1 -- 172.21.15.26:0/4291009656 learned_addr learned my addr 172.21.15.26:0/4291009656 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:46.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 --2- 172.21.15.26:0/4291009656 >> v2:172.21.15.120:3300/0 conn(0x7f405c0b5ef0 0x7f405c1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562785406 0 0) 0x7f405c0b62d0 con 0x7f405c0a7fc0 2022-07-20T02:35:46.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4040003260 con 0x7f405c0a7fc0 2022-07-20T02:35:46.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677047200 0 0) 0x7f4040003260 con 0x7f405c0a7fc0 2022-07-20T02:35:46.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 >> v2:172.21.15.120:3300/0 conn(0x7f405c0b5ef0 msgr2=0x7f405c1474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 --2- 172.21.15.26:0/4291009656 >> v2:172.21.15.120:3300/0 conn(0x7f405c0b5ef0 0x7f405c1474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 >> v1:172.21.15.120:6789/0 conn(0x7f405c0a8c40 legacy=0x7f405c151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c14b000 con 0x7f405c0a7fc0 2022-07-20T02:35:46.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f4058001680 con 0x7f405c0a7fc0 2022-07-20T02:35:46.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 >> v1:172.21.15.26:6789/0 conn(0x7f405c0a7fc0 legacy=0x7f405c0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 0x7f4040003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040004500 0x7f4040004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4040004e70 0x7f404000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c14ad60 con 0x7f4040003750 2022-07-20T02:35:46.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404000bd20 con 0x7f4040004e70 2022-07-20T02:35:46.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404000bee0 con 0x7f4040004500 2022-07-20T02:35:46.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 0x7f4040003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.485 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f406172d700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040004500 0x7f4040004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f406272f700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4040004e70 0x7f404000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.486 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 -- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040004500 msgr2=0x7f4040004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040004500 0x7f4040004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.487 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 -- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4040004e70 msgr2=0x7f404000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:46.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4040004e70 0x7f404000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.488 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f406272f700 1 -- 172.21.15.26:0/4291009656 reap_dead start 2022-07-20T02:35:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 -- 172.21.15.26:0/4291009656 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040003260 con 0x7f4040003750 2022-07-20T02:35:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.398+0000 7f4061f2e700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 0x7f4040003b70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f405000c000 tx=0x7f4050006df0).ready entity=mon.0 client_cookie=59f1b940b1ad32a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.489 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405000e830 con 0x7f4040003750 2022-07-20T02:35:46.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405000e990 con 0x7f4040003750 2022-07-20T02:35:46.494 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4060f2c700 1 -- 172.21.15.26:0/4291009656 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405000ecf0 con 0x7f4040003750 2022-07-20T02:35:46.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 msgr2=0x7f4040003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 0x7f4040003b70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f405000c000 tx=0x7f4050006df0).stop 2022-07-20T02:35:46.507 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- 172.21.15.26:0/4291009656 shutdown_connections 2022-07-20T02:35:46.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 --2- 172.21.15.26:0/4291009656 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4040003750 0x7f4040003b70 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- 172.21.15.26:0/4291009656 >> 172.21.15.26:0/4291009656 conn(0x7f405c060310 msgr2=0x7f405c0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:46.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- 172.21.15.26:0/4291009656 shutdown_connections 2022-07-20T02:35:46.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- 172.21.15.26:0/4291009656 wait complete. 2022-07-20T02:35:46.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 Processor -- start 2022-07-20T02:35:46.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- start start 2022-07-20T02:35:46.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040003750 0x7f405c0b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f405c0ab780 0x7f405c13c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c067d40 con 0x7f405c13cc30 2022-07-20T02:35:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c067850 con 0x7f405c0ab780 2022-07-20T02:35:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4062f30700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405c067aa0 con 0x7f4040003750 2022-07-20T02:35:46.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f406272f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f406272f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44404/0 (socket says 172.21.15.26:44404) 2022-07-20T02:35:46.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f406272f700 1 -- 172.21.15.26:0/3428704237 learned_addr learned my addr 172.21.15.26:0/3428704237 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f4061f2e700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040003750 0x7f405c0b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.402+0000 7f406172d700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f405c0ab780 0x7f405c13c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 -- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040003750 msgr2=0x7f405c0b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040003750 0x7f405c0b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 -- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f405c0ab780 msgr2=0x7f405c13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f405c0ab780 0x7f405c13c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.520 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 -- 172.21.15.26:0/3428704237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0abfd0 con 0x7f405c13cc30 2022-07-20T02:35:46.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f406272f700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f4058001110 tx=0x7f4054007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.521 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4054006c90 con 0x7f405c13cc30 2022-07-20T02:35:46.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4054006df0 con 0x7f405c13cc30 2022-07-20T02:35:46.529 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c13d0e0 con 0x7f405c13cc30 2022-07-20T02:35:46.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4054007220 con 0x7f405c13cc30 2022-07-20T02:35:46.530 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1444e0 con 0x7f405c13cc30 2022-07-20T02:35:46.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c0ac1d0 con 0x7f405c13cc30 2022-07-20T02:35:46.531 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.406+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4054007380 con 0x7f405c13cc30 2022-07-20T02:35:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.410+0000 7f404effd700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 0x7f4044055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.410+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f405406b630 con 0x7f405c13cc30 2022-07-20T02:35:46.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.410+0000 7f4061f2e700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 0x7f4044055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.410+0000 7f4061f2e700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 0x7f4044055f30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f405001d000 tx=0x7f4050019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.414+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f405c0ac1d0 con 0x7f405c13cc30 2022-07-20T02:35:46.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.578+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f405c0ac1d0 con 0x7f405c13cc30 2022-07-20T02:35:46.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.578+0000 7f404effd700 1 -- 172.21.15.26:0/3428704237 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2498 (secure 0 0 0) 0x7f405c0ac1d0 con 0x7f405c13cc30 2022-07-20T02:35:46.611 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:46.612 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:35:46.155980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1434658358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6835","nonce":1434658358},{"type":"v1","addr":"172.21.15.26:6837","nonce":1434658358}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:35:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 msgr2=0x7f4044055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 0x7f4044055f30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f405001d000 tx=0x7f4050019000).stop 2022-07-20T02:35:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 msgr2=0x7f405c15a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f4058001110 tx=0x7f4054007d80).stop 2022-07-20T02:35:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 shutdown_connections 2022-07-20T02:35:46.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f405c13cc30 0x7f405c15a070 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4044053a70 0x7f4044055f30 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f405c0ab780 0x7f405c13c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 --2- 172.21.15.26:0/3428704237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4040003750 0x7f405c0b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 >> 172.21.15.26:0/3428704237 conn(0x7f405c060310 msgr2=0x7f405c143590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:46.647 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 shutdown_connections 2022-07-20T02:35:46.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.582+0000 7f4062f30700 1 -- 172.21.15.26:0/3428704237 wait complete. 2022-07-20T02:35:46.662 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 89 2022-07-20T02:35:46.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 Processor -- start 2022-07-20T02:35:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 -- start start 2022-07-20T02:35:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1a340b5ef0 0x7f1a341474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 0x7f1a341479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a340b62d0 con 0x7f1a340a8c40 2022-07-20T02:35:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a34147eb0 con 0x7f1a340b5ef0 2022-07-20T02:35:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.790+0000 7f1a3b4ee700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3414aac0 con 0x7f1a340a7fc0 2022-07-20T02:35:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1a340b5ef0 0x7f1a341474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a39ceb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 0x7f1a341479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a39ceb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 0x7f1a341479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:35868/0 (socket says 172.21.15.26:35868) 2022-07-20T02:35:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3aced700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1a340a8c40 0x7f1a340aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:57904/0 (socket says 172.21.15.26:57904) 2022-07-20T02:35:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a39ceb700 1 -- 172.21.15.26:0/806971530 learned_addr learned my addr 172.21.15.26:0/806971530 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1a340b5ef0 0x7f1a341474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44408/0 (socket says 172.21.15.26:44408) 2022-07-20T02:35:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 -- 172.21.15.26:0/806971530 >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 msgr2=0x7f1a341479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/806971530 >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 0x7f1a341479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 -- 172.21.15.26:0/806971530 >> v1:172.21.15.120:6789/0 conn(0x7f1a340a8c40 legacy=0x7f1a340aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 -- 172.21.15.26:0/806971530 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3414ad60 con 0x7f1a340b5ef0 2022-07-20T02:35:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/806971530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340b5ef0 0x7f1a341474d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f1a30002f70 tx=0x7f1a30006df0).ready entity=mon.0 client_cookie=df6bf602541bd9e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a394ea700 1 -- 172.21.15.26:0/806971530 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3000d570 con 0x7f1a340b5ef0 2022-07-20T02:35:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a394ea700 1 -- 172.21.15.26:0/806971530 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30000e90 con 0x7f1a340b5ef0 2022-07-20T02:35:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a394ea700 1 -- 172.21.15.26:0/806971530 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3000d8c0 con 0x7f1a340b5ef0 2022-07-20T02:35:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/806971530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340b5ef0 msgr2=0x7f1a341474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/806971530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340b5ef0 0x7f1a341474d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f1a30002f70 tx=0x7f1a30006df0).stop 2022-07-20T02:35:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/806971530 shutdown_connections 2022-07-20T02:35:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/806971530 >> v2:172.21.15.120:3300/0 conn(0x7f1a340a7fc0 0x7f1a341479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/806971530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340b5ef0 0x7f1a341474d0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.887 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/806971530 >> 172.21.15.26:0/806971530 conn(0x7f1a34060310 msgr2=0x7f1a340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:46.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/806971530 shutdown_connections 2022-07-20T02:35:46.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/806971530 wait complete. 2022-07-20T02:35:46.888 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 Processor -- start 2022-07-20T02:35:46.889 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.794+0000 7f1a3b4ee700 1 -- start start 2022-07-20T02:35:46.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 0x7f1a34064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 0x7f1a34156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.890 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34157000 con 0x7f1a340677e0 2022-07-20T02:35:46.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34065290 con 0x7f1a340a8c40 2022-07-20T02:35:46.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34065e40 con 0x7f1a340a7fc0 2022-07-20T02:35:46.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.891 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44418/0 (socket says 172.21.15.26:44418) 2022-07-20T02:35:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 -- 172.21.15.26:0/3827669169 learned_addr learned my addr 172.21.15.26:0/3827669169 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a39ceb700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 0x7f1a34156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 0x7f1a34064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 -- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 msgr2=0x7f1a34064210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 0x7f1a34064210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 -- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 msgr2=0x7f1a34156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 0x7f1a34156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 -- 172.21.15.26:0/3827669169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3000b180 con 0x7f1a340677e0 2022-07-20T02:35:46.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 0x7f1a34064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:46.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a39ceb700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 0x7f1a34156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:46.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3aced700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f1a34060810 tx=0x7f1a2c007f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a2c004ab0 con 0x7f1a340677e0 2022-07-20T02:35:46.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c004c10 con 0x7f1a340677e0 2022-07-20T02:35:46.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a340661a0 con 0x7f1a340677e0 2022-07-20T02:35:46.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a2c007170 con 0x7f1a340677e0 2022-07-20T02:35:46.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.798+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a34064f80 con 0x7f1a340677e0 2022-07-20T02:35:46.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3405c110 con 0x7f1a340677e0 2022-07-20T02:35:46.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1a2c0072d0 con 0x7f1a340677e0 2022-07-20T02:35:46.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a237fe700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 0x7f1a14055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:46.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1a2c06bbc0 con 0x7f1a340677e0 2022-07-20T02:35:46.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 0x7f1a14055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:46.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.802+0000 7f1a3a4ec700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 0x7f1a14055e40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f1a3001a000 tx=0x7f1a30014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:46.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.810+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a3405c110 con 0x7f1a340677e0 2022-07-20T02:35:46.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.934+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a3405c110 con 0x7f1a340677e0 2022-07-20T02:35:46.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a237fe700 1 -- 172.21.15.26:0/3827669169 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f1a3405c110 con 0x7f1a340677e0 2022-07-20T02:35:46.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:46.961 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":162,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:46.156070+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:42.427470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:43.067323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:46.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 msgr2=0x7f1a14055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 0x7f1a14055e40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f1a3001a000 tx=0x7f1a30014000).stop 2022-07-20T02:35:46.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 msgr2=0x7f1a34156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:46.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f1a34060810 tx=0x7f1a2c007f30).stop 2022-07-20T02:35:46.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 shutdown_connections 2022-07-20T02:35:46.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a340677e0 0x7f1a34156b10 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:46.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1a14053980 0x7f1a14055e40 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a340a8c40 0x7f1a34156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 --2- 172.21.15.26:0/3827669169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a340a7fc0 0x7f1a34064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 >> 172.21.15.26:0/3827669169 conn(0x7f1a34060310 msgr2=0x7f1a340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:47.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 shutdown_connections 2022-07-20T02:35:47.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:46.938+0000 7f1a3b4ee700 1 -- 172.21.15.26:0/3827669169 wait complete. 2022-07-20T02:35:47.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:35:47.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 Processor -- start 2022-07-20T02:35:47.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 -- start start 2022-07-20T02:35:47.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f20200b5ef0 0x7f20201474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20200a7fc0 0x7f20201479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20200b62d0 con 0x7f20200a8c40 2022-07-20T02:35:47.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2020147eb0 con 0x7f20200b5ef0 2022-07-20T02:35:47.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.214+0000 7f2025953700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202014aac0 con 0x7f20200a7fc0 2022-07-20T02:35:47.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f20200b5ef0 0x7f20201474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f20200b5ef0 0x7f20201474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44422/0 (socket says 172.21.15.26:44422) 2022-07-20T02:35:47.234 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025152700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f20200a8c40 0x7f20200aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43704/0 (socket says 172.21.15.26:43704) 2022-07-20T02:35:47.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 -- 172.21.15.26:0/2741686815 learned_addr learned my addr 172.21.15.26:0/2741686815 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:47.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f201ffff700 1 --2- 172.21.15.26:0/2741686815 >> v2:172.21.15.120:3300/0 conn(0x7f20200a7fc0 0x7f20201479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 -- 172.21.15.26:0/2741686815 >> v2:172.21.15.120:3300/0 conn(0x7f20200a7fc0 msgr2=0x7f20201479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:47.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 --2- 172.21.15.26:0/2741686815 >> v2:172.21.15.120:3300/0 conn(0x7f20200a7fc0 0x7f20201479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 -- 172.21.15.26:0/2741686815 >> v1:172.21.15.26:6789/0 conn(0x7f20200a8c40 legacy=0x7f20200aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f201f7fe700 1 -- 172.21.15.26:0/2741686815 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980952875 0 0) 0x7f20200b62d0 con 0x7f20200a8c40 2022-07-20T02:35:47.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 -- 172.21.15.26:0/2741686815 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202014ad60 con 0x7f20200b5ef0 2022-07-20T02:35:47.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2024951700 1 --2- 172.21.15.26:0/2741686815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200b5ef0 0x7f20201474d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f2014002f70 tx=0x7f2014006eb0).ready entity=mon.0 client_cookie=42b0de30c8d45686 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f201f7fe700 1 -- 172.21.15.26:0/2741686815 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f201400d5c0 con 0x7f20200b5ef0 2022-07-20T02:35:47.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f201f7fe700 1 -- 172.21.15.26:0/2741686815 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2014000e90 con 0x7f20200b5ef0 2022-07-20T02:35:47.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f201f7fe700 1 -- 172.21.15.26:0/2741686815 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f201400d910 con 0x7f20200b5ef0 2022-07-20T02:35:47.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 -- 172.21.15.26:0/2741686815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200b5ef0 msgr2=0x7f20201474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 --2- 172.21.15.26:0/2741686815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200b5ef0 0x7f20201474d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f2014002f70 tx=0x7f2014006eb0).stop 2022-07-20T02:35:47.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 -- 172.21.15.26:0/2741686815 shutdown_connections 2022-07-20T02:35:47.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 --2- 172.21.15.26:0/2741686815 >> v2:172.21.15.120:3300/0 conn(0x7f20200a7fc0 0x7f20201479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 --2- 172.21.15.26:0/2741686815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200b5ef0 0x7f20201474d0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 -- 172.21.15.26:0/2741686815 >> 172.21.15.26:0/2741686815 conn(0x7f2020060310 msgr2=0x7f20200b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:47.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 -- 172.21.15.26:0/2741686815 shutdown_connections 2022-07-20T02:35:47.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.218+0000 7f2025953700 1 -- 172.21.15.26:0/2741686815 wait complete. 2022-07-20T02:35:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 Processor -- start 2022-07-20T02:35:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 -- start start 2022-07-20T02:35:47.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 0x7f20200b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 0x7f20200626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20200abc00 con 0x7f20200a8c40 2022-07-20T02:35:47.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2020157310 con 0x7f20200b5ef0 2022-07-20T02:35:47.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20201575b0 con 0x7f20200a7fc0 2022-07-20T02:35:47.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44430/0 (socket says 172.21.15.26:44430) 2022-07-20T02:35:47.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 -- 172.21.15.26:0/1491940249 learned_addr learned my addr 172.21.15.26:0/1491940249 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:47.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025152700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 0x7f20200626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2024951700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 0x7f20200b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 -- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 msgr2=0x7f20200b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 0x7f20200b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 -- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 msgr2=0x7f20200626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 0x7f20200626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 -- 172.21.15.26:0/1491940249 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201400b1e0 con 0x7f20200a8c40 2022-07-20T02:35:47.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025152700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 0x7f20200626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:47.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2024951700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 0x7f20200b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:47.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201ffff700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f201000c000 tx=0x7f2010006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2010011750 con 0x7f20200a8c40 2022-07-20T02:35:47.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20100118b0 con 0x7f20200a8c40 2022-07-20T02:35:47.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20200ab090 con 0x7f20200a8c40 2022-07-20T02:35:47.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.222+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2010011c10 con 0x7f20200a8c40 2022-07-20T02:35:47.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20200ab5a0 con 0x7f20200a8c40 2022-07-20T02:35:47.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202014b000 con 0x7f20200a8c40 2022-07-20T02:35:47.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2010011d70 con 0x7f20200a8c40 2022-07-20T02:35:47.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f201dffb700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 0x7f2000055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f201006c390 con 0x7f20200a8c40 2022-07-20T02:35:47.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.226+0000 7f2024951700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 0x7f2000055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.412 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.230+0000 7f2024951700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 0x7f2000055f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f201401c000 tx=0x7f2014015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.234+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20100430f0 con 0x7f20200a8c40 2022-07-20T02:35:47.415 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.402+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f20200ab220 con 0x7f20200a8c40 2022-07-20T02:35:47.416 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f201dffb700 1 -- 172.21.15.26:0/1491940249 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2010046710 con 0x7f20200a8c40 2022-07-20T02:35:47.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 msgr2=0x7f2000055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.420 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 0x7f2000055f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f201401c000 tx=0x7f2014015000).stop 2022-07-20T02:35:47.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 msgr2=0x7f20200aa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f201000c000 tx=0x7f2010006df0).stop 2022-07-20T02:35:47.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 shutdown_connections 2022-07-20T02:35:47.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f20200a8c40 0x7f20200aa810 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2000053a70 0x7f2000055f30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20200b5ef0 0x7f20200626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 --2- 172.21.15.26:0/1491940249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20200a7fc0 0x7f20200b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 >> 172.21.15.26:0/1491940249 conn(0x7f2020060310 msgr2=0x7f20200ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:47.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 shutdown_connections 2022-07-20T02:35:47.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.414+0000 7f2025953700 1 -- 172.21.15.26:0/1491940249 wait complete. 2022-07-20T02:35:47.516 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.0 2022-07-20T02:35:47.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T02:35:47.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 Processor -- start 2022-07-20T02:35:47.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 -- start start 2022-07-20T02:35:47.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f329c0b2780 0x7f329c140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c0b7ed0 con 0x7f329c0b7af0 2022-07-20T02:35:47.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c064aa0 con 0x7f329c066b10 2022-07-20T02:35:47.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f32a0d36700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c13ea50 con 0x7f329c0b2780 2022-07-20T02:35:47.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329a59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f329c0b7af0 0x7f329c064f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43716/0 (socket says 172.21.15.26:43716) 2022-07-20T02:35:47.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329a59c700 1 -- 172.21.15.26:0/1970360661 learned_addr learned my addr 172.21.15.26:0/1970360661 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:47.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f3299d9b700 1 --2- 172.21.15.26:0/1970360661 >> v2:172.21.15.120:3300/0 conn(0x7f329c0b2780 0x7f329c140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005527821 0 0) 0x7f329c0b7ed0 con 0x7f329c0b7af0 2022-07-20T02:35:47.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c003260 con 0x7f329c0b7af0 2022-07-20T02:35:47.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748086266 0 0) 0x7f329c064aa0 con 0x7f329c066b10 2022-07-20T02:35:47.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f329c0b7ed0 con 0x7f329c066b10 2022-07-20T02:35:47.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83861334 0 0) 0x7f327c003260 con 0x7f329c0b7af0 2022-07-20T02:35:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 >> v2:172.21.15.120:3300/0 conn(0x7f329c0b2780 msgr2=0x7f329c140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 --2- 172.21.15.26:0/1970360661 >> v2:172.21.15.120:3300/0 conn(0x7f329c0b2780 0x7f329c140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 >> v1:172.21.15.120:6789/0 conn(0x7f329c066b10 legacy=0x7f329c066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0b3530 con 0x7f329c0b7af0 2022-07-20T02:35:47.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f3284001660 con 0x7f329c0b7af0 2022-07-20T02:35:47.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 >> v1:172.21.15.26:6789/0 conn(0x7f329c0b7af0 legacy=0x7f329c064f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c0038e0 0x7f327c003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c004500 0x7f327c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.658+0000 7f329959a700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 0x7f327c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c00bcf0 con 0x7f327c004e70 2022-07-20T02:35:47.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c00be30 con 0x7f327c004500 2022-07-20T02:35:47.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c00bff0 con 0x7f327c0038e0 2022-07-20T02:35:47.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 0x7f327c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f3299d9b700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c004500 0x7f327c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329a59c700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c0038e0 0x7f327c003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 -- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c0038e0 msgr2=0x7f327c003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:47.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c0038e0 0x7f327c003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 -- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c004500 msgr2=0x7f327c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:47.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f327c004500 0x7f327c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 -- 172.21.15.26:0/1970360661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c003260 con 0x7f327c004e70 2022-07-20T02:35:47.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329a59c700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c0038e0 0x7f327c003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:47.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 -- 172.21.15.26:0/1970360661 reap_dead start 2022-07-20T02:35:47.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329ad9d700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 0x7f327c00b750 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f3294001490 tx=0x7f3294007d60).ready entity=mon.0 client_cookie=7eb7dab9b61431fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3294006890 con 0x7f327c004e70 2022-07-20T02:35:47.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32940069f0 con 0x7f327c004e70 2022-07-20T02:35:47.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f329959a700 1 -- 172.21.15.26:0/1970360661 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3294006d50 con 0x7f327c004e70 2022-07-20T02:35:47.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 msgr2=0x7f327c00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 0x7f327c00b750 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f3294001490 tx=0x7f3294007d60).stop 2022-07-20T02:35:47.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- 172.21.15.26:0/1970360661 shutdown_connections 2022-07-20T02:35:47.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1970360661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f327c004e70 0x7f327c00b750 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- 172.21.15.26:0/1970360661 >> 172.21.15.26:0/1970360661 conn(0x7f329c05c580 msgr2=0x7f329c05c990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:47.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- 172.21.15.26:0/1970360661 shutdown_connections 2022-07-20T02:35:47.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- 172.21.15.26:0/1970360661 wait complete. 2022-07-20T02:35:47.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 Processor -- start 2022-07-20T02:35:47.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.662+0000 7f32a0d36700 1 -- start start 2022-07-20T02:35:47.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 0x7f329c145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329c060950 0x7f329c060dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c062690 con 0x7f329c0acb20 2022-07-20T02:35:47.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0ae550 con 0x7f329c060950 2022-07-20T02:35:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c061fb0 con 0x7f327c004e70 2022-07-20T02:35:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44450/0 (socket says 172.21.15.26:44450) 2022-07-20T02:35:47.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 -- 172.21.15.26:0/1656899362 learned_addr learned my addr 172.21.15.26:0/1656899362 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:47.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f329ad9d700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329c060950 0x7f329c060dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f329a59c700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 0x7f329c145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 -- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 msgr2=0x7f329c145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 0x7f329c145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 -- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329c060950 msgr2=0x7f329c060dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329c060950 0x7f329c060dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 -- 172.21.15.26:0/1656899362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0ac300 con 0x7f329c0acb20 2022-07-20T02:35:47.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f329a59c700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 0x7f329c145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:47.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f3299d9b700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f329c139fc0 tx=0x7f328c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328c00f730 con 0x7f329c0acb20 2022-07-20T02:35:47.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c00f890 con 0x7f329c0acb20 2022-07-20T02:35:47.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c0ac5a0 con 0x7f329c0acb20 2022-07-20T02:35:47.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328c00fbf0 con 0x7f329c0acb20 2022-07-20T02:35:47.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.666+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c0b07e0 con 0x7f329c0acb20 2022-07-20T02:35:47.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329c0ae370 con 0x7f329c0acb20 2022-07-20T02:35:47.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f328c00fd50 con 0x7f329c0acb20 2022-07-20T02:35:47.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f32937fe700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 0x7f3280055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:47.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f329a59c700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 0x7f3280055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:47.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f328c06c910 con 0x7f329c0acb20 2022-07-20T02:35:47.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.670+0000 7f329a59c700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 0x7f3280055f30 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f328400d000 tx=0x7f3284009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:47.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.674+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f328c0436d0 con 0x7f329c0acb20 2022-07-20T02:35:47.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.842+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f329c0b2ff0 con 0x7f329c0acb20 2022-07-20T02:35:47.862 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.850+0000 7f32937fe700 1 -- 172.21.15.26:0/1656899362 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7f328c018020 con 0x7f329c0acb20 2022-07-20T02:35:47.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 msgr2=0x7f3280055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.867 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 0x7f3280055f30 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f328400d000 tx=0x7f3284009040).stop 2022-07-20T02:35:47.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 msgr2=0x7f329c0b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:47.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f329c139fc0 tx=0x7f328c006df0).stop 2022-07-20T02:35:47.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 shutdown_connections 2022-07-20T02:35:47.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f329c0acb20 0x7f329c0b1310 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f3280053a70 0x7f3280055f30 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329c060950 0x7f329c060dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 --2- 172.21.15.26:0/1656899362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f327c004e70 0x7f329c145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:47.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 >> 172.21.15.26:0/1656899362 conn(0x7f329c05c580 msgr2=0x7f329c0b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:47.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 shutdown_connections 2022-07-20T02:35:47.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.854+0000 7f32a0d36700 1 -- 172.21.15.26:0/1656899362 wait complete. 2022-07-20T02:35:47.877 INFO:teuthology.orchestra.run.smithi026.stderr:updated caps for client.1 2022-07-20T02:35:47.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:48.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 Processor -- start 2022-07-20T02:35:48.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- start start 2022-07-20T02:35:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfa80b5ef0 0x7fdfa81474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80a8c40 0x7fdfa80af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa80b62d0 con 0x7fdfa80a7fc0 2022-07-20T02:35:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa814aac0 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa814ad60 con 0x7fdfa80a8c40 2022-07-20T02:35:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfa80b5ef0 0x7fdfa81474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fdfa80b5ef0 0x7fdfa81474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44458/0 (socket says 172.21.15.26:44458) 2022-07-20T02:35:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 -- 172.21.15.26:0/708587142 learned_addr learned my addr 172.21.15.26:0/708587142 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:48.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa7fff700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80a8c40 0x7fdfa80af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 -- 172.21.15.26:0/708587142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80a8c40 msgr2=0x7fdfa80af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:48.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80a8c40 0x7fdfa80af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 -- 172.21.15.26:0/708587142 >> v1:172.21.15.120:6789/0 conn(0x7fdfa80a7fc0 legacy=0x7fdfa80aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 -- 172.21.15.26:0/708587142 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa814b000 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.161 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80b5ef0 0x7fdfa81474d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7fdf9c002f70 tx=0x7fdf9c006eb0).ready entity=mon.0 client_cookie=70842627a5baf993 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:48.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa67fc700 1 -- 172.21.15.26:0/708587142 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf9c00d570 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa67fc700 1 -- 172.21.15.26:0/708587142 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c000e90 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfa67fc700 1 -- 172.21.15.26:0/708587142 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf9c00d900 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- 172.21.15.26:0/708587142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80b5ef0 msgr2=0x7fdfa81474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80b5ef0 0x7fdfa81474d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7fdf9c002f70 tx=0x7fdf9c006eb0).stop 2022-07-20T02:35:48.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.994+0000 7fdfacd26700 1 -- 172.21.15.26:0/708587142 shutdown_connections 2022-07-20T02:35:48.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80a8c40 0x7fdfa80af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 --2- 172.21.15.26:0/708587142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80b5ef0 0x7fdfa81474d0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- 172.21.15.26:0/708587142 >> 172.21.15.26:0/708587142 conn(0x7fdfa8060310 msgr2=0x7fdfa80b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:48.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- 172.21.15.26:0/708587142 shutdown_connections 2022-07-20T02:35:48.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- 172.21.15.26:0/708587142 wait complete. 2022-07-20T02:35:48.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 Processor -- start 2022-07-20T02:35:48.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- start start 2022-07-20T02:35:48.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 0x7fdfa80b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80b5ef0 0x7fdfa80ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa8147c30 con 0x7fdfa80a8c40 2022-07-20T02:35:48.325 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa81451d0 con 0x7fdfa80a7fc0 2022-07-20T02:35:48.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa8145470 con 0x7fdfa80b5ef0 2022-07-20T02:35:48.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44464/0 (socket says 172.21.15.26:44464) 2022-07-20T02:35:48.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 -- 172.21.15.26:0/334348723 learned_addr learned my addr 172.21.15.26:0/334348723 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:48.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 0x7fdfa80b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa7fff700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80b5ef0 0x7fdfa80ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 -- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80b5ef0 msgr2=0x7fdfa80ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80b5ef0 0x7fdfa80ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 -- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 msgr2=0x7fdfa80b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 0x7fdfa80b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 -- 172.21.15.26:0/334348723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c00b1e0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 0x7fdfa80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:48.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa6ffd700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdfa8060810 tx=0x7fdf980014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:48.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf98010510 con 0x7fdfa80a8c40 2022-07-20T02:35:48.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf98010670 con 0x7fdfa80a8c40 2022-07-20T02:35:48.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf980107e0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa8145710 con 0x7fdfa80a8c40 2022-07-20T02:35:48.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:47.998+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa8159be0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdf98010070 con 0x7fdfa80a8c40 2022-07-20T02:35:48.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa814b2a0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfa4ff9700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 0x7fdf88055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:48.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 0x7fdf88055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:48.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdf9806be00 con 0x7fdfa80a8c40 2022-07-20T02:35:48.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.002+0000 7fdfa77fe700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 0x7fdf88055f60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fdf9c01a000 tx=0x7fdf9c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:48.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.006+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdf98042ab0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.170+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa80574e0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfa4ff9700 1 -- 172.21.15.26:0/334348723 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2498 (secure 0 0 0) 0x7fdf980460d0 con 0x7fdfa80a8c40 2022-07-20T02:35:48.380 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:48.381 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:35:46.155980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1434658358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6835","nonce":1434658358},{"type":"v1","addr":"172.21.15.26:6837","nonce":1434658358}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:35:48.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 msgr2=0x7fdf88055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 0x7fdf88055f60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fdf9c01a000 tx=0x7fdf9c015000).stop 2022-07-20T02:35:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 msgr2=0x7fdfa80abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdfa8060810 tx=0x7fdf980014a0).stop 2022-07-20T02:35:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 shutdown_connections 2022-07-20T02:35:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdfa80a8c40 0x7fdfa80abb30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fdf88053aa0 0x7fdf88055f60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfa80b5ef0 0x7fdfa80ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 --2- 172.21.15.26:0/334348723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa80a7fc0 0x7fdfa80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:48.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 >> 172.21.15.26:0/334348723 conn(0x7fdfa8060310 msgr2=0x7fdfa8148a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:48.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 shutdown_connections 2022-07-20T02:35:48.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:48.174+0000 7fdfacd26700 1 -- 172.21.15.26:0/334348723 wait complete. 2022-07-20T02:35:48.454 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 89 2022-07-20T02:35:48.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:48.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:48.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:44.142374+0000', 'modified': '2022-07-20T02:35:46.155980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8859': {'gid': 8859, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1434658358', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6835', 'nonce': 1434658358}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1434658358}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:48.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:48.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:48.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:48.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:35:48.943 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8859,"want_state":"up:active","state":"up:active","rank_uptime":4.7785021480000003,"mdsmap_epoch":89,"osdmap_epoch":162,"osdmap_epoch_barrier":161,"uptime":10.232703681} 2022-07-20T02:35:48.944 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8859, "mdsmap_epoch": 89, "osdmap_epoch": 162, "osdmap_epoch_barrier": 161, "rank_uptime": 4.778502148, "state": "up:active", "uptime": 10.232703681, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:48.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:49.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 Processor -- start 2022-07-20T02:35:49.063 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 -- start start 2022-07-20T02:35:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f26e80add30 0x7f26e80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 0x7f26e8060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80aa1e0 con 0x7f26e8061280 2022-07-20T02:35:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80ae110 con 0x7f26e80add30 2022-07-20T02:35:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ef43f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80a9f40 con 0x7f26e8060950 2022-07-20T02:35:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26ee43d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f26e80add30 0x7f26e80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26eec3e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f26e8061280 0x7f26e8143450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43754/0 (socket says 172.21.15.26:43754) 2022-07-20T02:35:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26eec3e700 1 -- 172.21.15.26:0/3774121787 learned_addr learned my addr 172.21.15.26:0/3774121787 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.058+0000 7f26edc3c700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 0x7f26e8060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ee43d700 1 -- 172.21.15.26:0/3774121787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 msgr2=0x7f26e8060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ed43b700 1 -- 172.21.15.26:0/3774121787 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014774314 0 0) 0x7f26e80aa1e0 con 0x7f26e8061280 2022-07-20T02:35:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ee43d700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 0x7f26e8060d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ee43d700 1 -- 172.21.15.26:0/3774121787 >> v1:172.21.15.26:6789/0 conn(0x7f26e8061280 legacy=0x7f26e8143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ee43d700 1 -- 172.21.15.26:0/3774121787 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80b1110 con 0x7f26e80add30 2022-07-20T02:35:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26edc3c700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 0x7f26e8060d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ee43d700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e80add30 0x7f26e80b2ab0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f26d8002f70 tx=0x7f26d8006df0).ready entity=mon.0 client_cookie=90d7fbbe6443a0af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ed43b700 1 -- 172.21.15.26:0/3774121787 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d800f970 con 0x7f26e80add30 2022-07-20T02:35:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ed43b700 1 -- 172.21.15.26:0/3774121787 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8000e90 con 0x7f26e80add30 2022-07-20T02:35:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ed43b700 1 -- 172.21.15.26:0/3774121787 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d800fcc0 con 0x7f26e80add30 2022-07-20T02:35:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 -- 172.21.15.26:0/3774121787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e80add30 msgr2=0x7f26e80b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e80add30 0x7f26e80b2ab0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f26d8002f70 tx=0x7f26d8006df0).stop 2022-07-20T02:35:49.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 -- 172.21.15.26:0/3774121787 shutdown_connections 2022-07-20T02:35:49.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e8060950 0x7f26e8060d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 --2- 172.21.15.26:0/3774121787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e80add30 0x7f26e80b2ab0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 -- 172.21.15.26:0/3774121787 >> 172.21.15.26:0/3774121787 conn(0x7f26e80b3050 msgr2=0x7f26e80b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 -- 172.21.15.26:0/3774121787 shutdown_connections 2022-07-20T02:35:49.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.062+0000 7f26ef43f700 1 -- 172.21.15.26:0/3774121787 wait complete. 2022-07-20T02:35:49.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 Processor -- start 2022-07-20T02:35:49.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- start start 2022-07-20T02:35:49.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e8060950 0x7f26e8144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80add30 0x7f26e81435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80a9a70 con 0x7f26e8061280 2022-07-20T02:35:49.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8140c10 con 0x7f26e8060950 2022-07-20T02:35:49.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8140510 con 0x7f26e80add30 2022-07-20T02:35:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44478/0 (socket says 172.21.15.26:44478) 2022-07-20T02:35:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 -- 172.21.15.26:0/565715263 learned_addr learned my addr 172.21.15.26:0/565715263 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26eec3e700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80add30 0x7f26e81435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ee43d700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e8060950 0x7f26e8144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 -- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80add30 msgr2=0x7f26e81435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80add30 0x7f26e81435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 -- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e8060950 msgr2=0x7f26e8144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e8060950 0x7f26e8144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 -- 172.21.15.26:0/565715263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d800b180 con 0x7f26e8061280 2022-07-20T02:35:49.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26edc3c700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f26dc006cc0 tx=0x7f26dc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26dc013620 con 0x7f26e8061280 2022-07-20T02:35:49.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc013780 con 0x7f26e8061280 2022-07-20T02:35:49.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26dc013ae0 con 0x7f26e8061280 2022-07-20T02:35:49.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e81424a0 con 0x7f26e8061280 2022-07-20T02:35:49.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.066+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e8142200 con 0x7f26e8061280 2022-07-20T02:35:49.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e8140910 con 0x7f26e8061280 2022-07-20T02:35:49.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f26dc013c40 con 0x7f26e8061280 2022-07-20T02:35:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26e77fe700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 0x7f26c8055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f26dc06bf60 con 0x7f26e8061280 2022-07-20T02:35:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26ee43d700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 0x7f26c8055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.070+0000 7f26ee43d700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 0x7f26c8055e40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f26d801c000 tx=0x7f26d8017000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.074+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f26dc042bb0 con 0x7f26e8061280 2022-07-20T02:35:49.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.202+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e80b13b0 con 0x7f26e8061280 2022-07-20T02:35:49.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.202+0000 7f26e77fe700 1 -- 172.21.15.26:0/565715263 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16333 (secure 0 0 0) 0x7f26dc0461d0 con 0x7f26e8061280 2022-07-20T02:35:49.208 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:49.209 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":162,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:46.156070+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:42.427470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:43.067323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:49.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 msgr2=0x7f26c8055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 0x7f26c8055e40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f26d801c000 tx=0x7f26d8017000).stop 2022-07-20T02:35:49.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 msgr2=0x7f26e81430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f26dc006cc0 tx=0x7f26dc008040).stop 2022-07-20T02:35:49.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 shutdown_connections 2022-07-20T02:35:49.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26e8061280 0x7f26e81430d0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f26c8053980 0x7f26c8055e40 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e80add30 0x7f26e81435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 --2- 172.21.15.26:0/565715263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e8060950 0x7f26e8144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 >> 172.21.15.26:0/565715263 conn(0x7f26e80b3050 msgr2=0x7f26e8060030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 shutdown_connections 2022-07-20T02:35:49.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.206+0000 7f26ef43f700 1 -- 172.21.15.26:0/565715263 wait complete. 2022-07-20T02:35:49.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:35:42.427470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:35:43.067323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:35:49.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 Processor -- start 2022-07-20T02:35:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 -- start start 2022-07-20T02:35:49.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c05cfd0 0x7fa02c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa02c0b1160 0x7fa02c140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02c147a00 con 0x7fa02c146e80 2022-07-20T02:35:49.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02c1435b0 con 0x7fa02c0b1160 2022-07-20T02:35:49.402 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.342+0000 7fa0334fb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c05d3b0 con 0x7fa02c05cfd0 2022-07-20T02:35:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa031a98700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa02c146e80 0x7fa02c1472f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43766/0 (socket says 172.21.15.26:43766) 2022-07-20T02:35:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa031a98700 1 -- 172.21.15.26:0/339275680 learned_addr learned my addr 172.21.15.26:0/339275680 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:49.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa02c0b1160 0x7fa02c140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa031297700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c05cfd0 0x7fa02c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa017fff700 1 -- 172.21.15.26:0/339275680 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039920384 0 0) 0x7fa02c147a00 con 0x7fa02c146e80 2022-07-20T02:35:49.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa017fff700 1 -- 172.21.15.26:0/339275680 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa010003260 con 0x7fa02c146e80 2022-07-20T02:35:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 -- 172.21.15.26:0/339275680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c05cfd0 msgr2=0x7fa02c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c05cfd0 0x7fa02c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 -- 172.21.15.26:0/339275680 >> v1:172.21.15.26:6789/0 conn(0x7fa02c146e80 legacy=0x7fa02c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 -- 172.21.15.26:0/339275680 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0b4bc0 con 0x7fa02c0b1160 2022-07-20T02:35:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa030a96700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c0b1160 0x7fa02c140cb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fa020002f70 tx=0x7fa020006df0).ready entity=mon.0 client_cookie=fef4f1bb0a84aa5c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa017fff700 1 -- 172.21.15.26:0/339275680 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa02000d4a0 con 0x7fa02c0b1160 2022-07-20T02:35:49.409 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa017fff700 1 -- 172.21.15.26:0/339275680 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa020000e90 con 0x7fa02c0b1160 2022-07-20T02:35:49.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa017fff700 1 -- 172.21.15.26:0/339275680 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa02000d7f0 con 0x7fa02c0b1160 2022-07-20T02:35:49.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 -- 172.21.15.26:0/339275680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c0b1160 msgr2=0x7fa02c140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c0b1160 0x7fa02c140cb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fa020002f70 tx=0x7fa020006df0).stop 2022-07-20T02:35:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 -- 172.21.15.26:0/339275680 shutdown_connections 2022-07-20T02:35:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c0b1160 0x7fa02c140cb0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 --2- 172.21.15.26:0/339275680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c05cfd0 0x7fa02c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 -- 172.21.15.26:0/339275680 >> 172.21.15.26:0/339275680 conn(0x7fa02c066f70 msgr2=0x7fa02c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 -- 172.21.15.26:0/339275680 shutdown_connections 2022-07-20T02:35:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.346+0000 7fa0334fb700 1 -- 172.21.15.26:0/339275680 wait complete. 2022-07-20T02:35:49.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 Processor -- start 2022-07-20T02:35:49.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 -- start start 2022-07-20T02:35:49.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c0b1160 0x7fa02c142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa02c146e80 0x7fa02c0b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c13a1a0 con 0x7fa02c05cfd0 2022-07-20T02:35:49.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1429b0 con 0x7fa02c146e80 2022-07-20T02:35:49.518 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa0334fb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1421c0 con 0x7fa02c0b1160 2022-07-20T02:35:49.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa031297700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa031297700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44490/0 (socket says 172.21.15.26:44490) 2022-07-20T02:35:49.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa031297700 1 -- 172.21.15.26:0/403040699 learned_addr learned my addr 172.21.15.26:0/403040699 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:49.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa030a96700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c0b1160 0x7fa02c142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa031a98700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa02c146e80 0x7fa02c0b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.350+0000 7fa031297700 1 -- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c0b1160 msgr2=0x7fa02c142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa031297700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c0b1160 0x7fa02c142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa031297700 1 -- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa02c146e80 msgr2=0x7fa02c0b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa031297700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa02c146e80 0x7fa02c0b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa031297700 1 -- 172.21.15.26:0/403040699 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02000b180 con 0x7fa02c05cfd0 2022-07-20T02:35:49.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa031297700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7fa02800c000 tx=0x7fa028006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa028010780 con 0x7fa02c05cfd0 2022-07-20T02:35:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0280108e0 con 0x7fa02c05cfd0 2022-07-20T02:35:49.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c13e680 con 0x7fa02c05cfd0 2022-07-20T02:35:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa028010c40 con 0x7fa02c05cfd0 2022-07-20T02:35:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c13f450 con 0x7fa02c05cfd0 2022-07-20T02:35:49.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c0574e0 con 0x7fa02c05cfd0 2022-07-20T02:35:49.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa02801c460 con 0x7fa02c05cfd0 2022-07-20T02:35:49.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 0x7fa008055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa02806c4b0 con 0x7fa02c05cfd0 2022-07-20T02:35:49.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa030a96700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 0x7fa008055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.354+0000 7fa030a96700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 0x7fa008055fc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fa020018000 tx=0x7fa020014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.358+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa028043210 con 0x7fa02c05cfd0 2022-07-20T02:35:49.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa02c060030 con 0x7fa02c05cfd0 2022-07-20T02:35:49.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa01effd700 1 -- 172.21.15.26:0/403040699 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v162) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa028046830 con 0x7fa02c05cfd0 2022-07-20T02:35:49.590 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:49.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 msgr2=0x7fa008055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 0x7fa008055fc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fa020018000 tx=0x7fa020014000).stop 2022-07-20T02:35:49.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 msgr2=0x7fa02c141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7fa02800c000 tx=0x7fa028006df0).stop 2022-07-20T02:35:49.609 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 shutdown_connections 2022-07-20T02:35:49.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa008053b00 0x7fa008055fc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa02c146e80 0x7fa02c0b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa02c0b1160 0x7fa02c142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 --2- 172.21.15.26:0/403040699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa02c05cfd0 0x7fa02c141cd0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 >> 172.21.15.26:0/403040699 conn(0x7fa02c066f70 msgr2=0x7fa02c13d920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 shutdown_connections 2022-07-20T02:35:49.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.486+0000 7fa0334fb700 1 -- 172.21.15.26:0/403040699 wait complete. 2022-07-20T02:35:49.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:49.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.742+0000 7f48cdb97700 1 Processor -- start 2022-07-20T02:35:49.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.742+0000 7f48cdb97700 1 -- start start 2022-07-20T02:35:49.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f48c805cfd0 0x7f48c80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 0x7f48c8144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80b27a0 con 0x7f48c80a9ed0 2022-07-20T02:35:49.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c805d3b0 con 0x7f48c805cfd0 2022-07-20T02:35:49.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c80b29d0 con 0x7f48c805f840 2022-07-20T02:35:49.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cd396700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f48c80a9ed0 0x7f48c80ac770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43780/0 (socket says 172.21.15.26:43780) 2022-07-20T02:35:49.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cd396700 1 -- 172.21.15.26:0/1159228096 learned_addr learned my addr 172.21.15.26:0/1159228096 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:49.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 --2- 172.21.15.26:0/1159228096 >> v2:172.21.15.26:3300/0 conn(0x7f48c805cfd0 0x7f48c80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c7fff700 1 --2- 172.21.15.26:0/1159228096 >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 0x7f48c8144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c77fe700 1 -- 172.21.15.26:0/1159228096 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910199292 0 0) 0x7f48c80b27a0 con 0x7f48c80a9ed0 2022-07-20T02:35:49.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c77fe700 1 -- 172.21.15.26:0/1159228096 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b4003260 con 0x7f48c80a9ed0 2022-07-20T02:35:49.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 -- 172.21.15.26:0/1159228096 >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 msgr2=0x7f48c8144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:49.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 --2- 172.21.15.26:0/1159228096 >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 0x7f48c8144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 -- 172.21.15.26:0/1159228096 >> v1:172.21.15.26:6789/0 conn(0x7f48c80a9ed0 legacy=0x7f48c80ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 -- 172.21.15.26:0/1159228096 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80b2c70 con 0x7f48c805cfd0 2022-07-20T02:35:49.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c7fff700 1 --2- 172.21.15.26:0/1159228096 >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 0x7f48c8144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:49.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48ccb95700 1 --2- 172.21.15.26:0/1159228096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c80b2150 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f48b8002f70 tx=0x7f48b8006df0).ready entity=mon.0 client_cookie=ed2a4fdd351cf24a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c77fe700 1 -- 172.21.15.26:0/1159228096 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b800d570 con 0x7f48c805cfd0 2022-07-20T02:35:49.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c77fe700 1 -- 172.21.15.26:0/1159228096 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8000e90 con 0x7f48c805cfd0 2022-07-20T02:35:49.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48c77fe700 1 -- 172.21.15.26:0/1159228096 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b800d8c0 con 0x7f48c805cfd0 2022-07-20T02:35:49.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- 172.21.15.26:0/1159228096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 msgr2=0x7f48c80b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 --2- 172.21.15.26:0/1159228096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c80b2150 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f48b8002f70 tx=0x7f48b8006df0).stop 2022-07-20T02:35:49.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- 172.21.15.26:0/1159228096 shutdown_connections 2022-07-20T02:35:49.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 --2- 172.21.15.26:0/1159228096 >> v2:172.21.15.120:3300/0 conn(0x7f48c805f840 0x7f48c8144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 --2- 172.21.15.26:0/1159228096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c80b2150 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- 172.21.15.26:0/1159228096 >> 172.21.15.26:0/1159228096 conn(0x7f48c80b7090 msgr2=0x7f48c805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- 172.21.15.26:0/1159228096 shutdown_connections 2022-07-20T02:35:49.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.746+0000 7f48cdb97700 1 -- 172.21.15.26:0/1159228096 wait complete. 2022-07-20T02:35:49.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 Processor -- start 2022-07-20T02:35:49.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- start start 2022-07-20T02:35:49.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c805f840 0x7f48c8152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48c80a9ed0 0x7f48c8152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8063980 con 0x7f48c805cfd0 2022-07-20T02:35:49.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8063c20 con 0x7f48c805f840 2022-07-20T02:35:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8063ec0 con 0x7f48c80a9ed0 2022-07-20T02:35:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44504/0 (socket says 172.21.15.26:44504) 2022-07-20T02:35:49.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48c7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c805f840 0x7f48c8152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 -- 172.21.15.26:0/376168297 learned_addr learned my addr 172.21.15.26:0/376168297 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:49.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cd396700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48c80a9ed0 0x7f48c8152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 -- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48c80a9ed0 msgr2=0x7f48c8152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48c80a9ed0 0x7f48c8152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 -- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c805f840 msgr2=0x7f48c8152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c805f840 0x7f48c8152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 -- 172.21.15.26:0/376168297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b800b180 con 0x7f48c805cfd0 2022-07-20T02:35:49.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48ccb95700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f48b8002f70 tx=0x7f48b8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b800dea0 con 0x7f48c805cfd0 2022-07-20T02:35:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8013de0 con 0x7f48c805cfd0 2022-07-20T02:35:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b801e5f0 con 0x7f48c805cfd0 2022-07-20T02:35:49.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8064160 con 0x7f48c805cfd0 2022-07-20T02:35:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.750+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c8141060 con 0x7f48c805cfd0 2022-07-20T02:35:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c8144730 con 0x7f48c805cfd0 2022-07-20T02:35:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f48b801c070 con 0x7f48c805cfd0 2022-07-20T02:35:49.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48c5ffb700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 0x7f48a8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:49.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f48b806ca40 con 0x7f48c805cfd0 2022-07-20T02:35:49.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48c7fff700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 0x7f48a8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:49.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.754+0000 7f48c7fff700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 0x7f48a8055f60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f48bc00c000 tx=0x7f48bc008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:49.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.758+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f48b80436f0 con 0x7f48c805cfd0 2022-07-20T02:35:49.893 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.886+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f48c80574e0 con 0x7f48c805cfd0 2022-07-20T02:35:49.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.886+0000 7f48c5ffb700 1 -- 172.21.15.26:0/376168297 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v162) v1 ==== 92+0+10 (secure 0 0 0) 0x7f48b8023070 con 0x7f48c805cfd0 2022-07-20T02:35:49.894 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:49.897 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 msgr2=0x7f48a8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 0x7f48a8055f60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f48bc00c000 tx=0x7f48bc008040).stop 2022-07-20T02:35:49.898 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 msgr2=0x7f48c8140b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:49.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f48b8002f70 tx=0x7f48b8014000).stop 2022-07-20T02:35:49.899 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 shutdown_connections 2022-07-20T02:35:49.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f48a8053aa0 0x7f48a8055f60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f48c80a9ed0 0x7f48c8152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c805f840 0x7f48c8152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 --2- 172.21.15.26:0/376168297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48c805cfd0 0x7f48c8140b70 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:49.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 >> 172.21.15.26:0/376168297 conn(0x7f48c80b7090 msgr2=0x7f48c8143e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:49.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 shutdown_connections 2022-07-20T02:35:49.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:49.890+0000 7f48cdb97700 1 -- 172.21.15.26:0/376168297 wait complete. 2022-07-20T02:35:49.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:35:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 Processor -- start 2022-07-20T02:35:50.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 -- start start 2022-07-20T02:35:50.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb02805cfd0 0x7fb028143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028060f00 0x7fb02813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0281474e0 con 0x7fb028147000 2022-07-20T02:35:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0280b4e40 con 0x7fb02805cfd0 2022-07-20T02:35:50.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02f2ca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02805d3b0 con 0x7fb028060f00 2022-07-20T02:35:50.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb02805cfd0 0x7fb028143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb02805cfd0 0x7fb028143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44512/0 (socket says 172.21.15.26:44512) 2022-07-20T02:35:50.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 -- 172.21.15.26:0/579830483 learned_addr learned my addr 172.21.15.26:0/579830483 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:50.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02eac9700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028060f00 0x7fb02813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.182 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 -- 172.21.15.26:0/579830483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028060f00 msgr2=0x7fb02813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:50.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028060f00 0x7fb02813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.183 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 -- 172.21.15.26:0/579830483 >> v1:172.21.15.26:6789/0 conn(0x7fb028147000 legacy=0x7fb0280b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.184 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 -- 172.21.15.26:0/579830483 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0281431e0 con 0x7fb02805cfd0 2022-07-20T02:35:50.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02d2c6700 1 -- 172.21.15.26:0/579830483 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626792334 0 0) 0x7fb0281474e0 con 0x7fb028147000 2022-07-20T02:35:50.185 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb02805cfd0 0x7fb028143bc0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fb024002f70 tx=0x7fb024006eb0).ready entity=mon.0 client_cookie=29ca01edf36206f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.042+0000 7fb02d2c6700 1 -- 172.21.15.26:0/579830483 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb02400d5c0 con 0x7fb02805cfd0 2022-07-20T02:35:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02d2c6700 1 -- 172.21.15.26:0/579830483 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024000e90 con 0x7fb02805cfd0 2022-07-20T02:35:50.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02d2c6700 1 -- 172.21.15.26:0/579830483 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb02400d910 con 0x7fb02805cfd0 2022-07-20T02:35:50.187 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- 172.21.15.26:0/579830483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb02805cfd0 msgr2=0x7fb028143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.188 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb02805cfd0 0x7fb028143bc0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fb024002f70 tx=0x7fb024006eb0).stop 2022-07-20T02:35:50.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- 172.21.15.26:0/579830483 shutdown_connections 2022-07-20T02:35:50.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028060f00 0x7fb02813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/579830483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb02805cfd0 0x7fb028143bc0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- 172.21.15.26:0/579830483 >> 172.21.15.26:0/579830483 conn(0x7fb0280620c0 msgr2=0x7fb02805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:50.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- 172.21.15.26:0/579830483 shutdown_connections 2022-07-20T02:35:50.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- 172.21.15.26:0/579830483 wait complete. 2022-07-20T02:35:50.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 Processor -- start 2022-07-20T02:35:50.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- start start 2022-07-20T02:35:50.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 0x7fb0280b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028147000 0x7fb028155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0281492b0 con 0x7fb0280aef80 2022-07-20T02:35:50.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb028149fe0 con 0x7fb028060f00 2022-07-20T02:35:50.223 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.046+0000 7fb02f2ca700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb028140b50 con 0x7fb028147000 2022-07-20T02:35:50.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44522/0 (socket says 172.21.15.26:44522) 2022-07-20T02:35:50.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 -- 172.21.15.26:0/1270530582 learned_addr learned my addr 172.21.15.26:0/1270530582 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:50.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 0x7fb0280b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02dac7700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028147000 0x7fb028155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 -- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028147000 msgr2=0x7fb028155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028147000 0x7fb028155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 -- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 msgr2=0x7fb0280b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 0x7fb0280b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 -- 172.21.15.26:0/1270530582 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02400b1e0 con 0x7fb0280aef80 2022-07-20T02:35:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 0x7fb0280b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02eac9700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fb028140e20 tx=0x7fb020008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0200127e0 con 0x7fb0280aef80 2022-07-20T02:35:50.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb020012940 con 0x7fb0280aef80 2022-07-20T02:35:50.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02814a210 con 0x7fb0280aef80 2022-07-20T02:35:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb020012ca0 con 0x7fb0280aef80 2022-07-20T02:35:50.233 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb028152f90 con 0x7fb0280aef80 2022-07-20T02:35:50.235 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb020013480 con 0x7fb0280aef80 2022-07-20T02:35:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb028149e10 con 0x7fb0280aef80 2022-07-20T02:35:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 0x7fb008055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 0x7fb008055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.050+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb02006c490 con 0x7fb0280aef80 2022-07-20T02:35:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.054+0000 7fb02e2c8700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 0x7fb008055ed0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fb02401a000 tx=0x7fb024015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.058+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0200430e0 con 0x7fb0280aef80 2022-07-20T02:35:50.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.182+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb028143480 con 0x7fb0280aef80 2022-07-20T02:35:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.182+0000 7fb0177fe700 1 -- 172.21.15.26:0/1270530582 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v162) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb020012e00 con 0x7fb0280aef80 2022-07-20T02:35:50.370 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 msgr2=0x7fb008055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 0x7fb008055ed0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fb02401a000 tx=0x7fb024015000).stop 2022-07-20T02:35:50.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 msgr2=0x7fb028155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fb028140e20 tx=0x7fb020008040).stop 2022-07-20T02:35:50.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 shutdown_connections 2022-07-20T02:35:50.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0280aef80 0x7fb028155af0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb008053a10 0x7fb008055ed0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb028147000 0x7fb028155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 --2- 172.21.15.26:0/1270530582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb028060f00 0x7fb0280b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 >> 172.21.15.26:0/1270530582 conn(0x7fb02805cfd0 msgr2=0x7fb028138f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:50.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 shutdown_connections 2022-07-20T02:35:50.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.186+0000 7fb02f2ca700 1 -- 172.21.15.26:0/1270530582 wait complete. 2022-07-20T02:35:50.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:35:50.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.514+0000 7fbd05f1e700 1 Processor -- start 2022-07-20T02:35:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- start start 2022-07-20T02:35:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fbd000b31a0 0x7fbd000b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd000a9690 0x7fbd000b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd000613a0 con 0x7fbd0009b9f0 2022-07-20T02:35:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd000615f0 con 0x7fbd000b31a0 2022-07-20T02:35:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd000b1980 con 0x7fbd000a9690 2022-07-20T02:35:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbd0009b9f0 0x7fbd00067070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43806/0 (socket says 172.21.15.26:43806) 2022-07-20T02:35:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfffff700 1 -- 172.21.15.26:0/1868134525 learned_addr learned my addr 172.21.15.26:0/1868134525 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 --2- 172.21.15.26:0/1868134525 >> v2:172.21.15.26:3300/0 conn(0x7fbd000b31a0 0x7fbd000b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfeffd700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd000a9690 0x7fbd000b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfe7fc700 1 -- 172.21.15.26:0/1868134525 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059835885 0 0) 0x7fbd000613a0 con 0x7fbd0009b9f0 2022-07-20T02:35:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfe7fc700 1 -- 172.21.15.26:0/1868134525 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce0003260 con 0x7fbd0009b9f0 2022-07-20T02:35:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 -- 172.21.15.26:0/1868134525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd000a9690 msgr2=0x7fbd000b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd000a9690 0x7fbd000b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 -- 172.21.15.26:0/1868134525 >> v1:172.21.15.26:6789/0 conn(0x7fbd0009b9f0 legacy=0x7fbd00067070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 -- 172.21.15.26:0/1868134525 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd000b3580 con 0x7fbd000b31a0 2022-07-20T02:35:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcff7fe700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd000b1e20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fbce8002f70 tx=0x7fbce8006df0).ready entity=mon.0 client_cookie=fc8b30c2ba33b68d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfe7fc700 1 -- 172.21.15.26:0/1868134525 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbce800d4a0 con 0x7fbd000b31a0 2022-07-20T02:35:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfe7fc700 1 -- 172.21.15.26:0/1868134525 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce8000e90 con 0x7fbd000b31a0 2022-07-20T02:35:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbcfe7fc700 1 -- 172.21.15.26:0/1868134525 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbce800d7f0 con 0x7fbd000b31a0 2022-07-20T02:35:50.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- 172.21.15.26:0/1868134525 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 msgr2=0x7fbd000b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd000b1e20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fbce8002f70 tx=0x7fbce8006df0).stop 2022-07-20T02:35:50.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 -- 172.21.15.26:0/1868134525 shutdown_connections 2022-07-20T02:35:50.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd000a9690 0x7fbd000b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.518+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/1868134525 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd000b1e20 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- 172.21.15.26:0/1868134525 >> 172.21.15.26:0/1868134525 conn(0x7fbd000b6920 msgr2=0x7fbd000b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:50.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- 172.21.15.26:0/1868134525 shutdown_connections 2022-07-20T02:35:50.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- 172.21.15.26:0/1868134525 wait complete. 2022-07-20T02:35:50.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 Processor -- start 2022-07-20T02:35:50.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- start start 2022-07-20T02:35:50.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0009b9f0 0x7fbd00094640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 0x7fbd00156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0009a540 con 0x7fbd000b31a0 2022-07-20T02:35:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0009a770 con 0x7fbd000a9690 2022-07-20T02:35:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbd05f1e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0009aa10 con 0x7fbd0009b9f0 2022-07-20T02:35:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbcfffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbcfffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44536/0 (socket says 172.21.15.26:44536) 2022-07-20T02:35:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbcfffff700 1 -- 172.21.15.26:0/3005348374 learned_addr learned my addr 172.21.15.26:0/3005348374 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbcff7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0009b9f0 0x7fbd00094640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.522+0000 7fbcfeffd700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 0x7fbd00156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 -- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0009b9f0 msgr2=0x7fbd00094640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0009b9f0 0x7fbd00094640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 -- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 msgr2=0x7fbd00156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 0x7fbd00156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 -- 172.21.15.26:0/3005348374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce800b180 con 0x7fbd000b31a0 2022-07-20T02:35:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfeffd700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 0x7fbd00156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfffff700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fbcf8001490 tx=0x7fbcf8007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf8005ba0 con 0x7fbd000b31a0 2022-07-20T02:35:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf8005d00 con 0x7fbd000b31a0 2022-07-20T02:35:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd00155d70 con 0x7fbd000b31a0 2022-07-20T02:35:50.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf80072c0 con 0x7fbd000b31a0 2022-07-20T02:35:50.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd00095020 con 0x7fbd000b31a0 2022-07-20T02:35:50.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0009ac10 con 0x7fbd000b31a0 2022-07-20T02:35:50.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.526+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbcf8010070 con 0x7fbd000b31a0 2022-07-20T02:35:50.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.530+0000 7fbcfcff9700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 0x7fbce4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.530+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbcf806ba90 con 0x7fbd000b31a0 2022-07-20T02:35:50.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.530+0000 7fbcff7fe700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 0x7fbce4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:50.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.530+0000 7fbcff7fe700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 0x7fbce4055f30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fbce801a000 tx=0x7fbce8014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:50.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.534+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbcf8042850 con 0x7fbd000b31a0 2022-07-20T02:35:50.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd00155f00 con 0x7fbd000b31a0 2022-07-20T02:35:50.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbcfcff9700 1 -- 172.21.15.26:0/3005348374 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v162) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbcf8045e70 con 0x7fbd000b31a0 2022-07-20T02:35:50.719 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 msgr2=0x7fbce4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 0x7fbce4055f30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fbce801a000 tx=0x7fbce8014000).stop 2022-07-20T02:35:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 msgr2=0x7fbd00156700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fbcf8001490 tx=0x7fbcf8007e70).stop 2022-07-20T02:35:50.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 shutdown_connections 2022-07-20T02:35:50.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.662+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbd000b31a0 0x7fbd00156700 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fbce4053a70 0x7fbce4055f30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd000a9690 0x7fbd00156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 --2- 172.21.15.26:0/3005348374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0009b9f0 0x7fbd00094640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:50.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 >> 172.21.15.26:0/3005348374 conn(0x7fbd000b6920 msgr2=0x7fbd000b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:50.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 shutdown_connections 2022-07-20T02:35:50.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.666+0000 7fbd05f1e700 1 -- 172.21.15.26:0/3005348374 wait complete. 2022-07-20T02:35:50.728 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:35:50.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 Processor -- start 2022-07-20T02:35:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 -- start start 2022-07-20T02:35:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a44147be0 0x7f8a440aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a440a7fc0 0x7f8a440af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a440b62d0 con 0x7f8a440b5ef0 2022-07-20T02:35:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4414aac0 con 0x7f8a44147be0 2022-07-20T02:35:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a43fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4414ad60 con 0x7f8a440a7fc0 2022-07-20T02:35:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a44147be0 0x7f8a440aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f8a44147be0 0x7f8a440aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44542/0 (socket says 172.21.15.26:44542) 2022-07-20T02:35:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 -- 172.21.15.26:0/59841234 learned_addr learned my addr 172.21.15.26:0/59841234 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a437fe700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a440a7fc0 0x7f8a440af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 -- 172.21.15.26:0/59841234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a440a7fc0 msgr2=0x7f8a440af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a440a7fc0 0x7f8a440af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 -- 172.21.15.26:0/59841234 >> v1:172.21.15.26:6789/0 conn(0x7f8a440b5ef0 legacy=0x7f8a441474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.854+0000 7f8a427fc700 1 -- 172.21.15.26:0/59841234 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4414b000 con 0x7f8a44147be0 2022-07-20T02:35:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a427fc700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a44147be0 0x7f8a440aee40 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f8a38002f70 tx=0x7f8a38006eb0).ready entity=mon.0 client_cookie=a2756c9b8c87ab48 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a41ffb700 1 -- 172.21.15.26:0/59841234 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a3800d5c0 con 0x7f8a44147be0 2022-07-20T02:35:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a41ffb700 1 -- 172.21.15.26:0/59841234 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a38000e90 con 0x7f8a44147be0 2022-07-20T02:35:51.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a41ffb700 1 -- 172.21.15.26:0/59841234 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a3800d950 con 0x7f8a44147be0 2022-07-20T02:35:51.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 -- 172.21.15.26:0/59841234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a44147be0 msgr2=0x7f8a440aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a44147be0 0x7f8a440aee40 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f8a38002f70 tx=0x7f8a38006eb0).stop 2022-07-20T02:35:51.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 -- 172.21.15.26:0/59841234 shutdown_connections 2022-07-20T02:35:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a440a7fc0 0x7f8a440af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 --2- 172.21.15.26:0/59841234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a44147be0 0x7f8a440aee40 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 -- 172.21.15.26:0/59841234 >> 172.21.15.26:0/59841234 conn(0x7f8a44060310 msgr2=0x7f8a440b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 -- 172.21.15.26:0/59841234 shutdown_connections 2022-07-20T02:35:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.858+0000 7f8a43fff700 1 -- 172.21.15.26:0/59841234 wait complete. 2022-07-20T02:35:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 Processor -- start 2022-07-20T02:35:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- start start 2022-07-20T02:35:51.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:51.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 0x7f8a440b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:51.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 0x7f8a440abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:51.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44156690 con 0x7f8a440a7fc0 2022-07-20T02:35:51.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440ac290 con 0x7f8a440b5ef0 2022-07-20T02:35:51.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440aa9e0 con 0x7f8a44147be0 2022-07-20T02:35:51.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44546/0 (socket says 172.21.15.26:44546) 2022-07-20T02:35:51.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 -- 172.21.15.26:0/1305414774 learned_addr learned my addr 172.21.15.26:0/1305414774 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:51.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a427fc700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 0x7f8a440b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a437fe700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 0x7f8a440abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 -- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 msgr2=0x7f8a440abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 0x7f8a440abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 -- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 msgr2=0x7f8a440b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 0x7f8a440b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 -- 172.21.15.26:0/1305414774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3800b1e0 con 0x7f8a440a7fc0 2022-07-20T02:35:51.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a437fe700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 0x7f8a440abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:35:51.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a427fc700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 0x7f8a440b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:35:51.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a42ffd700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f8a440ac4e0 tx=0x7f8a3c007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a3c006aa0 con 0x7f8a440a7fc0 2022-07-20T02:35:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c006c00 con 0x7f8a440a7fc0 2022-07-20T02:35:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a440aac10 con 0x7f8a440a7fc0 2022-07-20T02:35:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a3c007030 con 0x7f8a440a7fc0 2022-07-20T02:35:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a44156a80 con 0x7f8a440a7fc0 2022-07-20T02:35:51.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a440ab0b0 con 0x7f8a440a7fc0 2022-07-20T02:35:51.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.862+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a3c019030 con 0x7f8a440a7fc0 2022-07-20T02:35:51.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.866+0000 7f8a33fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 0x7f8a20055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:51.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.866+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8a3c06b810 con 0x7f8a440a7fc0 2022-07-20T02:35:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.866+0000 7f8a427fc700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 0x7f8a20055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.866+0000 7f8a427fc700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 0x7f8a20055f60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f8a38017000 tx=0x7f8a38015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:50.874+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a3c0425d0 con 0x7f8a440a7fc0 2022-07-20T02:35:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a4414b2a0 con 0x7f8a440a7fc0 2022-07-20T02:35:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a33fff700 1 -- 172.21.15.26:0/1305414774 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2498 (secure 0 0 0) 0x7f8a3c017030 con 0x7f8a440a7fc0 2022-07-20T02:35:51.243 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:51.244 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:35:46.155980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1434658358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6835","nonce":1434658358},{"type":"v1","addr":"172.21.15.26:6837","nonce":1434658358}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:35:51.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 msgr2=0x7f8a20055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 0x7f8a20055f60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f8a38017000 tx=0x7f8a38015000).stop 2022-07-20T02:35:51.275 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 msgr2=0x7f8a440630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:51.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f8a440ac4e0 tx=0x7f8a3c007f90).stop 2022-07-20T02:35:51.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.034+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 shutdown_connections 2022-07-20T02:35:51.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f8a20053aa0 0x7f8a20055f60 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a44147be0 0x7f8a440abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a440b5ef0 0x7f8a440b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 --2- 172.21.15.26:0/1305414774 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a440a7fc0 0x7f8a440630c0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:51.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 >> 172.21.15.26:0/1305414774 conn(0x7f8a44060310 msgr2=0x7f8a440af050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:51.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 shutdown_connections 2022-07-20T02:35:51.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:51.038+0000 7f8a43fff700 1 -- 172.21.15.26:0/1305414774 wait complete. 2022-07-20T02:35:51.279 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 89 2022-07-20T02:35:51.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:44.142374+0000', 'modified': '2022-07-20T02:35:46.155980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8859': {'gid': 8859, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1434658358', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6835', 'nonce': 1434658358}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1434658358}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:51.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:51.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:51.740 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8859,"want_state":"up:active","state":"up:active","rank_uptime":7.5758674490000004,"mdsmap_epoch":89,"osdmap_epoch":162,"osdmap_epoch_barrier":161,"uptime":13.030068164999999} 2022-07-20T02:35:51.741 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8859, "mdsmap_epoch": 89, "osdmap_epoch": 162, "osdmap_epoch_barrier": 161, "rank_uptime": 7.575867449, "state": "up:active", "uptime": 13.030068165, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:51.752 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:35:51.764 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T02:35:51.764 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi159.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T02:35:51.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:51.774 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:51.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:35:51.863 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:51.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:51.902 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:51.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:51.903 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:35:51.960 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:35:51.969 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:51.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo modprobe fuse 2022-07-20T02:35:52.020 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:52.021 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:52.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:2022-07-20T02:35:52.056+0000 7f729204a080 -1 init, newargv = 0x55bbca1f7d60 newargc=9ceph-fuse[23550]: starting ceph client 2022-07-20T02:35:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr: 2022-07-20T02:35:52.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi159.stderr:ceph-fuse[23550]: starting fuse 2022-07-20T02:35:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:52.102 INFO:teuthology.orchestra.run.smithi159.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:52.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:52.103 DEBUG:teuthology.orchestra.run.smithi159:> ls /sys/fs/fuse/connections 2022-07-20T02:35:52.153 INFO:teuthology.orchestra.run.smithi159.stdout:52 2022-07-20T02:35:52.153 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:35:52.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:52.154 DEBUG:teuthology.orchestra.run.smithi159:> sudo python3 -c ' 2022-07-20T02:35:52.155 DEBUG:teuthology.orchestra.run.smithi159:> import glob 2022-07-20T02:35:52.155 DEBUG:teuthology.orchestra.run.smithi159:> import re 2022-07-20T02:35:52.156 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-20T02:35:52.156 DEBUG:teuthology.orchestra.run.smithi159:> import subprocess 2022-07-20T02:35:52.157 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:52.157 DEBUG:teuthology.orchestra.run.smithi159:> def find_socket(client_name): 2022-07-20T02:35:52.158 DEBUG:teuthology.orchestra.run.smithi159:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T02:35:52.158 DEBUG:teuthology.orchestra.run.smithi159:> files = glob.glob(asok_path) 2022-07-20T02:35:52.159 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:52.159 DEBUG:teuthology.orchestra.run.smithi159:> # Given a non-glob path, it better be there 2022-07-20T02:35:52.160 DEBUG:teuthology.orchestra.run.smithi159:> if "*" not in asok_path: 2022-07-20T02:35:52.160 DEBUG:teuthology.orchestra.run.smithi159:> assert(len(files) == 1) 2022-07-20T02:35:52.161 DEBUG:teuthology.orchestra.run.smithi159:> return files[0] 2022-07-20T02:35:52.170 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:52.171 DEBUG:teuthology.orchestra.run.smithi159:> for f in files: 2022-07-20T02:35:52.172 DEBUG:teuthology.orchestra.run.smithi159:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:35:52.172 DEBUG:teuthology.orchestra.run.smithi159:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:35:52.173 DEBUG:teuthology.orchestra.run.smithi159:> return f 2022-07-20T02:35:52.179 DEBUG:teuthology.orchestra.run.smithi159:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:35:52.181 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-20T02:35:52.182 DEBUG:teuthology.orchestra.run.smithi159:> print(find_socket("client.0")) 2022-07-20T02:35:52.183 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-20T02:35:52.229 INFO:teuthology.orchestra.run.smithi159.stdout:/var/run/ceph/ceph-client.0.23550.asok 2022-07-20T02:35:52.240 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23550.asok 2022-07-20T02:35:52.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:52.257 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23550.asok status 2022-07-20T02:35:52.401 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2022-07-20T02:35:52.411 INFO:teuthology.orchestra.run.smithi159.stdout: "metadata": { 2022-07-20T02:35:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:35:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:35:52.412 INFO:teuthology.orchestra.run.smithi159.stdout: "entity_id": "0", 2022-07-20T02:35:52.413 INFO:teuthology.orchestra.run.smithi159.stdout: "hostname": "smithi159", 2022-07-20T02:35:52.413 INFO:teuthology.orchestra.run.smithi159.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T02:35:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: "pid": "23550", 2022-07-20T02:35:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: "root": "/" 2022-07-20T02:35:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:52.421 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_count": 0, 2022-07-20T02:35:52.422 INFO:teuthology.orchestra.run.smithi159.stdout: "dentry_pinned_count": 0, 2022-07-20T02:35:52.422 INFO:teuthology.orchestra.run.smithi159.stdout: "id": 9021, 2022-07-20T02:35:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: "inst": { 2022-07-20T02:35:52.446 INFO:teuthology.orchestra.run.smithi159.stdout: "name": { 2022-07-20T02:35:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "client", 2022-07-20T02:35:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "num": 9021 2022-07-20T02:35:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:52.447 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:35:52.448 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:35:52.478 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:35:52.479 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 1705012474 2022-07-20T02:35:52.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2022-07-20T02:35:52.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:52.487 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": { 2022-07-20T02:35:52.487 INFO:teuthology.orchestra.run.smithi159.stdout: "type": "v1", 2022-07-20T02:35:52.488 INFO:teuthology.orchestra.run.smithi159.stdout: "addr": "172.21.15.159:0", 2022-07-20T02:35:52.488 INFO:teuthology.orchestra.run.smithi159.stdout: "nonce": 1705012474 2022-07-20T02:35:52.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2022-07-20T02:35:52.488 INFO:teuthology.orchestra.run.smithi159.stdout: "inst_str": "client.9021 172.21.15.159:0/1705012474", 2022-07-20T02:35:52.488 INFO:teuthology.orchestra.run.smithi159.stdout: "addr_str": "172.21.15.159:0/1705012474", 2022-07-20T02:35:52.489 INFO:teuthology.orchestra.run.smithi159.stdout: "inode_count": 1, 2022-07-20T02:35:52.489 INFO:teuthology.orchestra.run.smithi159.stdout: "mds_epoch": 89, 2022-07-20T02:35:52.489 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch": 162, 2022-07-20T02:35:52.499 INFO:teuthology.orchestra.run.smithi159.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:35:52.499 INFO:teuthology.orchestra.run.smithi159.stdout: "blacklisted": false, 2022-07-20T02:35:52.500 INFO:teuthology.orchestra.run.smithi159.stdout: "fs_name": "cephfs" 2022-07-20T02:35:52.500 INFO:teuthology.orchestra.run.smithi159.stdout:} 2022-07-20T02:35:52.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:52.514 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:52.537 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:52.556 INFO:teuthology.orchestra.run.smithi159.stdout:fuseblk 2022-07-20T02:35:52.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T02:35:52.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:52.559 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:35:52.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T02:35:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 Processor -- start 2022-07-20T02:35:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 -- start start 2022-07-20T02:35:52.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d0061a20 0x7fa9d01432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:52.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9d01468c0 0x7fa9d012c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:52.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 0x7fa9d00b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:52.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d0061e00 con 0x7fa9d01468c0 2022-07-20T02:35:52.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d0147040 con 0x7fa9d012c530 2022-07-20T02:35:52.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d6f06700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d01472e0 con 0x7fa9d0061a20 2022-07-20T02:35:52.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9d01468c0 0x7fa9d012c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:52.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa9d01468c0 0x7fa9d012c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44558/0 (socket says 172.21.15.26:44558) 2022-07-20T02:35:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d54a3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 0x7fa9d00b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d4ca2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d0061a20 0x7fa9d01432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 -- 172.21.15.26:0/1421868297 learned_addr learned my addr 172.21.15.26:0/1421868297 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:53.003 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 -- 172.21.15.26:0/1421868297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d0061a20 msgr2=0x7fa9d01432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:53.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 --2- 172.21.15.26:0/1421868297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d0061a20 0x7fa9d01432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.004 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 -- 172.21.15.26:0/1421868297 >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 msgr2=0x7fa9d00b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 --2- 172.21.15.26:0/1421868297 >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 0x7fa9d00b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 -- 172.21.15.26:0/1421868297 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0147580 con 0x7fa9d01468c0 2022-07-20T02:35:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9d54a3700 1 --2- 172.21.15.26:0/1421868297 >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 0x7fa9d00b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cffff700 1 --2- 172.21.15.26:0/1421868297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d01468c0 0x7fa9d012c040 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7fa9c0002f70 tx=0x7fa9c0006eb0).ready entity=mon.0 client_cookie=a9eca8cc38cbea75 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/1421868297 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c000d570 con 0x7fa9d01468c0 2022-07-20T02:35:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/1421868297 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c0000e90 con 0x7fa9d01468c0 2022-07-20T02:35:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.782+0000 7fa9cf7fe700 1 -- 172.21.15.26:0/1421868297 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c000d900 con 0x7fa9d01468c0 2022-07-20T02:35:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- 172.21.15.26:0/1421868297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d01468c0 msgr2=0x7fa9d012c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/1421868297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d01468c0 0x7fa9d012c040 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7fa9c0002f70 tx=0x7fa9c0006eb0).stop 2022-07-20T02:35:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- 172.21.15.26:0/1421868297 shutdown_connections 2022-07-20T02:35:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/1421868297 >> v2:172.21.15.120:3300/0 conn(0x7fa9d012c530 0x7fa9d00b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/1421868297 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d01468c0 0x7fa9d012c040 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/1421868297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d0061a20 0x7fa9d01432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- 172.21.15.26:0/1421868297 >> 172.21.15.26:0/1421868297 conn(0x7fa9d005faf0 msgr2=0x7fa9d0066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:53.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- 172.21.15.26:0/1421868297 shutdown_connections 2022-07-20T02:35:53.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- 172.21.15.26:0/1421868297 wait complete. 2022-07-20T02:35:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 Processor -- start 2022-07-20T02:35:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.786+0000 7fa9d6f06700 1 -- start start 2022-07-20T02:35:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d012c530 0x7fa9d0127190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d01468c0 0x7fa9d0127680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d00b8350 con 0x7fa9d0061a20 2022-07-20T02:35:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0141380 con 0x7fa9d01468c0 2022-07-20T02:35:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0141620 con 0x7fa9d012c530 2022-07-20T02:35:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44560/0 (socket says 172.21.15.26:44560) 2022-07-20T02:35:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 -- 172.21.15.26:0/348703148 learned_addr learned my addr 172.21.15.26:0/348703148 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d54a3700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d01468c0 0x7fa9d0127680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9cffff700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d012c530 0x7fa9d0127190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 -- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d012c530 msgr2=0x7fa9d0127190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d012c530 0x7fa9d0127190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 -- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d01468c0 msgr2=0x7fa9d0127680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d01468c0 0x7fa9d0127680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 -- 172.21.15.26:0/348703148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009470 con 0x7fa9d0061a20 2022-07-20T02:35:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d4ca2700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fa9b800c000 tx=0x7fa9b8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b8010780 con 0x7fa9d0061a20 2022-07-20T02:35:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b80108e0 con 0x7fa9d0061a20 2022-07-20T02:35:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c000b1e0 con 0x7fa9d0061a20 2022-07-20T02:35:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b8010c40 con 0x7fa9d0061a20 2022-07-20T02:35:53.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d00b7cb0 con 0x7fa9d0061a20 2022-07-20T02:35:53.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.790+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d00b80f0 con 0x7fa9d0061a20 2022-07-20T02:35:53.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.794+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9b801c460 con 0x7fa9d0061a20 2022-07-20T02:35:53.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.794+0000 7fa9cdffb700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 0x7fa9b4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.794+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa9b806c410 con 0x7fa9d0061a20 2022-07-20T02:35:53.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.794+0000 7fa9cffff700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 0x7fa9b4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.794+0000 7fa9cffff700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 0x7fa9b4055ff0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fa9c001c000 tx=0x7fa9c0015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.798+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9b8043170 con 0x7fa9d0061a20 2022-07-20T02:35:53.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9d0146d20 con 0x7fa9d0061a20 2022-07-20T02:35:53.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9cdffb700 1 -- 172.21.15.26:0/348703148 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16333 (secure 0 0 0) 0x7fa9b8046790 con 0x7fa9d0061a20 2022-07-20T02:35:53.339 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:53.339 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":162,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:35:46.156070+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T02:35:42.427470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T02:35:43.067323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T02:35:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 msgr2=0x7fa9b4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 0x7fa9b4055ff0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fa9c001c000 tx=0x7fa9c0015000).stop 2022-07-20T02:35:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 msgr2=0x7fa9d0148160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.930+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fa9b800c000 tx=0x7fa9b8006df0).stop 2022-07-20T02:35:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 shutdown_connections 2022-07-20T02:35:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa9b4053b30 0x7fa9b4055ff0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9d01468c0 0x7fa9d0127680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9d012c530 0x7fa9d0127190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 --2- 172.21.15.26:0/348703148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9d0061a20 0x7fa9d0148160 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 >> 172.21.15.26:0/348703148 conn(0x7fa9d005faf0 msgr2=0x7fa9d0142b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 shutdown_connections 2022-07-20T02:35:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:52.934+0000 7fa9d6f06700 1 -- 172.21.15.26:0/348703148 wait complete. 2022-07-20T02:35:53.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T02:25:19.431119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T02:25:28.375708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T02:35:42.427470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T02:35:43.067323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T02:35:53.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T02:35:53.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.526+0000 7f5e26e09700 1 Processor -- start 2022-07-20T02:35:53.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.526+0000 7f5e26e09700 1 -- start start 2022-07-20T02:35:53.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e20067520 0x7f5e20067900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 0x7f5e20097350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e20096c80 con 0x7f5e200b22c0 2022-07-20T02:35:53.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e20098070 con 0x7f5e20067520 2022-07-20T02:35:53.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e200b48a0 con 0x7f5e200b5600 2022-07-20T02:35:53.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e20067520 0x7f5e20067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5e20067520 0x7f5e20067900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44568/0 (socket says 172.21.15.26:44568) 2022-07-20T02:35:53.722 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 -- 172.21.15.26:0/2859809658 learned_addr learned my addr 172.21.15.26:0/2859809658 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e17fff700 1 --2- 172.21.15.26:0/2859809658 >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 0x7f5e20097350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 -- 172.21.15.26:0/2859809658 >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 msgr2=0x7f5e20097350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:53.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 --2- 172.21.15.26:0/2859809658 >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 0x7f5e20097350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 -- 172.21.15.26:0/2859809658 >> v1:172.21.15.120:6789/0 conn(0x7f5e200b22c0 legacy=0x7f5e20097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 -- 172.21.15.26:0/2859809658 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e200b4b40 con 0x7f5e20067520 2022-07-20T02:35:53.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e17fff700 1 --2- 172.21.15.26:0/2859809658 >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 0x7f5e20097350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:35:53.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e24ba5700 1 --2- 172.21.15.26:0/2859809658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e20067900 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f5e1c002f70 tx=0x7f5e1c006eb0).ready entity=mon.0 client_cookie=b1ab899f2304e865 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e177fe700 1 -- 172.21.15.26:0/2859809658 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c00d610 con 0x7f5e20067520 2022-07-20T02:35:53.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e177fe700 1 -- 172.21.15.26:0/2859809658 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c000e90 con 0x7f5e20067520 2022-07-20T02:35:53.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e177fe700 1 -- 172.21.15.26:0/2859809658 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c00d9a0 con 0x7f5e20067520 2022-07-20T02:35:53.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- 172.21.15.26:0/2859809658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 msgr2=0x7f5e20067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 --2- 172.21.15.26:0/2859809658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e20067900 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f5e1c002f70 tx=0x7f5e1c006eb0).stop 2022-07-20T02:35:53.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- 172.21.15.26:0/2859809658 shutdown_connections 2022-07-20T02:35:53.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 --2- 172.21.15.26:0/2859809658 >> v2:172.21.15.120:3300/0 conn(0x7f5e200b5600 0x7f5e20097350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 --2- 172.21.15.26:0/2859809658 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e20067900 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- 172.21.15.26:0/2859809658 >> 172.21.15.26:0/2859809658 conn(0x7f5e200b5ea0 msgr2=0x7f5e200b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:53.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- 172.21.15.26:0/2859809658 shutdown_connections 2022-07-20T02:35:53.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.530+0000 7f5e26e09700 1 -- 172.21.15.26:0/2859809658 wait complete. 2022-07-20T02:35:53.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 Processor -- start 2022-07-20T02:35:53.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- start start 2022-07-20T02:35:53.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e200b22c0 0x7f5e20092f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e200b5600 0x7f5e20145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20152790 con 0x7f5e20067520 2022-07-20T02:35:53.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200aa130 con 0x7f5e200b22c0 2022-07-20T02:35:53.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200afc80 con 0x7f5e200b5600 2022-07-20T02:35:53.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44576/0 (socket says 172.21.15.26:44576) 2022-07-20T02:35:53.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 -- 172.21.15.26:0/3780259553 learned_addr learned my addr 172.21.15.26:0/3780259553 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:53.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e17fff700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e200b22c0 0x7f5e20092f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e253a6700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e200b5600 0x7f5e20145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 -- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e200b5600 msgr2=0x7f5e20145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e200b5600 0x7f5e20145b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 -- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e200b22c0 msgr2=0x7f5e20092f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e200b22c0 0x7f5e20092f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 -- 172.21.15.26:0/3780259553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c00b1e0 con 0x7f5e20067520 2022-07-20T02:35:53.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e24ba5700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f5e1c002f70 tx=0x7f5e1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c011760 con 0x7f5e20067520 2022-07-20T02:35:53.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c0118c0 con 0x7f5e20067520 2022-07-20T02:35:53.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e20092630 con 0x7f5e20067520 2022-07-20T02:35:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c011c20 con 0x7f5e20067520 2022-07-20T02:35:53.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.534+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e20092b30 con 0x7f5e20067520 2022-07-20T02:35:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e20096eb0 con 0x7f5e20067520 2022-07-20T02:35:53.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e1c025030 con 0x7f5e20067520 2022-07-20T02:35:53.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e15ffb700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 0x7f5e00055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:53.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5e1c06d160 con 0x7f5e20067520 2022-07-20T02:35:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e17fff700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 0x7f5e00055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:53.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.538+0000 7f5e17fff700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 0x7f5e00055f30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f5e1000c000 tx=0x7f5e10008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:53.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.546+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e1c043ec0 con 0x7f5e20067520 2022-07-20T02:35:53.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.670+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5e200574e0 con 0x7f5e20067520 2022-07-20T02:35:53.797 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e15ffb700 1 -- 172.21.15.26:0/3780259553 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v162) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5e1c0230b0 con 0x7f5e20067520 2022-07-20T02:35:53.798 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2022-07-20T02:35:53.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 msgr2=0x7f5e00055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 0x7f5e00055f30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f5e1000c000 tx=0x7f5e10008040).stop 2022-07-20T02:35:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 msgr2=0x7f5e200b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f5e1c002f70 tx=0x7f5e1c015000).stop 2022-07-20T02:35:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 shutdown_connections 2022-07-20T02:35:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5e00053a70 0x7f5e00055f30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e200b5600 0x7f5e20145b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e200b22c0 0x7f5e20092f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 --2- 172.21.15.26:0/3780259553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e20067520 0x7f5e200b11c0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 >> 172.21.15.26:0/3780259553 conn(0x7f5e200b5ea0 msgr2=0x7f5e20097270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 shutdown_connections 2022-07-20T02:35:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.674+0000 7f5e26e09700 1 -- 172.21.15.26:0/3780259553 wait complete. 2022-07-20T02:35:53.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T02:35:54.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.950+0000 7ff9ceaa7700 1 Processor -- start 2022-07-20T02:35:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.950+0000 7ff9ceaa7700 1 -- start start 2022-07-20T02:35:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9ceaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9c81431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9ceaa7700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff9c80b5b70 0x7ff9c81403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9ceaa7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80ba060 con 0x7ff9c80b9c80 2022-07-20T02:35:54.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9ceaa7700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c812bee0 con 0x7ff9c80b5b70 2022-07-20T02:35:54.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9ceaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8061f80 con 0x7ff9c8062710 2022-07-20T02:35:54.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cc843700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff9c80b9c80 0x7ff9c80b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43862/0 (socket says 172.21.15.26:43862) 2022-07-20T02:35:54.032 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cc843700 1 -- 172.21.15.26:0/1756770776 learned_addr learned my addr 172.21.15.26:0/1756770776 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cd044700 1 --2- 172.21.15.26:0/1756770776 >> v2:172.21.15.120:3300/0 conn(0x7ff9c80b5b70 0x7ff9c81403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9c81431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122016127 0 0) 0x7ff9c80ba060 con 0x7ff9c80b9c80 2022-07-20T02:35:54.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a8003260 con 0x7ff9c80b9c80 2022-07-20T02:35:54.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394469609 0 0) 0x7ff9a8003260 con 0x7ff9c80b9c80 2022-07-20T02:35:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 msgr2=0x7ff9c81431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9c81431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 >> v2:172.21.15.120:3300/0 conn(0x7ff9c80b5b70 msgr2=0x7ff9c81403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 --2- 172.21.15.26:0/1756770776 >> v2:172.21.15.120:3300/0 conn(0x7ff9c80b5b70 0x7ff9c81403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8062220 con 0x7ff9c80b9c80 2022-07-20T02:35:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7ff9b0001660 con 0x7ff9c80b9c80 2022-07-20T02:35:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 >> v1:172.21.15.26:6789/0 conn(0x7ff9c80b9c80 legacy=0x7ff9c80b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9a8003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9a8003c80 0x7ff9a8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a80049f0 0x7ff9a8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c812bee0 con 0x7ff9a8003c80 2022-07-20T02:35:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8061f80 con 0x7ff9a80049f0 2022-07-20T02:35:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a800b750 con 0x7ff9c8062710 2022-07-20T02:35:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9a8003c80 0x7ff9a8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cc843700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9a8003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cd044700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a80049f0 0x7ff9a8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 -- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 msgr2=0x7ff9a8003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c8062710 0x7ff9a8003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 -- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a80049f0 msgr2=0x7ff9a8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a80049f0 0x7ff9a8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 -- 172.21.15.26:0/1756770776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8003260 con 0x7ff9a8003c80 2022-07-20T02:35:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9cd044700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a80049f0 0x7ff9a8004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:54.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9a8003c80 0x7ff9a8004500 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7ff9a80054d0 tx=0x7ff9b8006df0).ready entity=mon.0 client_cookie=46f6569711859c09 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9b800e830 con 0x7ff9a8003c80 2022-07-20T02:35:54.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b800be90 con 0x7ff9a8003c80 2022-07-20T02:35:54.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.954+0000 7ff9c77fe700 1 -- 172.21.15.26:0/1756770776 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9b800eb80 con 0x7ff9a8003c80 2022-07-20T02:35:54.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/1756770776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9a8003c80 msgr2=0x7ff9a8004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/1756770776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9a8003c80 0x7ff9a8004500 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7ff9a80054d0 tx=0x7ff9b8006df0).stop 2022-07-20T02:35:54.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 -- 172.21.15.26:0/1756770776 reap_dead start 2022-07-20T02:35:54.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/1756770776 shutdown_connections 2022-07-20T02:35:54.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/1756770776 >> 172.21.15.26:0/1756770776 conn(0x7ff9c8066f70 msgr2=0x7ff9c805faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:54.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/1756770776 shutdown_connections 2022-07-20T02:35:54.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/1756770776 wait complete. 2022-07-20T02:35:54.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 Processor -- start 2022-07-20T02:35:54.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- start start 2022-07-20T02:35:54.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 0x7ff9c81564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8155b80 con 0x7ff9c80b1d30 2022-07-20T02:35:54.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80b19a0 con 0x7ff9a8003c80 2022-07-20T02:35:54.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9ceaa7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8155db0 con 0x7ff9c812d990 2022-07-20T02:35:54.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cc843700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 0x7ff9c81564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9c7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9c7fff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37880/0 (socket says 172.21.15.26:37880) 2022-07-20T02:35:54.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9c7fff700 1 -- 172.21.15.26:0/4199366865 learned_addr learned my addr 172.21.15.26:0/4199366865 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:54.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44598/0 (socket says 172.21.15.26:44598) 2022-07-20T02:35:54.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 -- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 msgr2=0x7ff9c80b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 -- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 msgr2=0x7ff9c81564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 0x7ff9c81564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cd044700 1 -- 172.21.15.26:0/4199366865 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8156050 con 0x7ff9c80b1d30 2022-07-20T02:35:54.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9cc843700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 0x7ff9c81564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:54.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.958+0000 7ff9c7fff700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:54.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9cd044700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7ff9c000b000 tx=0x7ff9c00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c0019070 con 0x7ff9c80b1d30 2022-07-20T02:35:54.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c000f910 con 0x7ff9c80b1d30 2022-07-20T02:35:54.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c81569e0 con 0x7ff9c80b1d30 2022-07-20T02:35:54.335 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c000fc70 con 0x7ff9c80b1d30 2022-07-20T02:35:54.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c8152aa0 con 0x7ff9c80b1d30 2022-07-20T02:35:54.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c80b1760 con 0x7ff9c80b1d30 2022-07-20T02:35:54.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff9c0014440 con 0x7ff9c80b1d30 2022-07-20T02:35:54.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 0x7ff9ac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff9c006cb30 con 0x7ff9c80b1d30 2022-07-20T02:35:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9cc843700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 0x7ff9ac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.962+0000 7ff9cc843700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 0x7ff9ac055fc0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7ff9b0005000 tx=0x7ff9b0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:53.970+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff9c80b1760 con 0x7ff9c80b1d30 2022-07-20T02:35:54.413 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.098+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff9c80b1760 con 0x7ff9c80b1d30 2022-07-20T02:35:54.414 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.098+0000 7ff9c5ffb700 1 -- 172.21.15.26:0/4199366865 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v162) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff9c80b1760 con 0x7ff9c80b1d30 2022-07-20T02:35:54.415 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:54.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 msgr2=0x7ff9ac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 0x7ff9ac055fc0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7ff9b0005000 tx=0x7ff9b0009040).stop 2022-07-20T02:35:54.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 msgr2=0x7ff9c81525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7ff9c000b000 tx=0x7ff9c00091d0).stop 2022-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 shutdown_connections 2022-07-20T02:35:54.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9c80b1d30 0x7ff9c81525b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff9ac053b00 0x7ff9ac055fc0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9c812d990 0x7ff9c80b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.458 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 --2- 172.21.15.26:0/4199366865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9a8003c80 0x7ff9c81564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 >> 172.21.15.26:0/4199366865 conn(0x7ff9c8066f70 msgr2=0x7ff9c80b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:54.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 shutdown_connections 2022-07-20T02:35:54.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.102+0000 7ff9ceaa7700 1 -- 172.21.15.26:0/4199366865 wait complete. 2022-07-20T02:35:54.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T02:35:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 Processor -- start 2022-07-20T02:35:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 -- start start 2022-07-20T02:35:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29780add30 0x7f29780b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2978060950 0x7f2978060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2978061280 0x7f2978143450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29780aa1e0 con 0x7f2978061280 2022-07-20T02:35:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29780ae110 con 0x7f29780add30 2022-07-20T02:35:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297f0f4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29780a9f40 con 0x7f2978060950 2022-07-20T02:35:54.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297e8f3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2978061280 0x7f2978143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297e8f3700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2978061280 0x7f2978143450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44606/0 (socket says 172.21.15.26:44606) 2022-07-20T02:35:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297d8f1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2978060950 0x7f2978060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297e8f3700 1 -- 172.21.15.26:0/3810302302 learned_addr learned my addr 172.21.15.26:0/3810302302 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:54.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.630+0000 7f297e0f2700 1 --2- 172.21.15.26:0/3810302302 >> v2:172.21.15.120:3300/0 conn(0x7f29780add30 0x7f29780b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 -- 172.21.15.26:0/3810302302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2978060950 msgr2=0x7f2978060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 --2- 172.21.15.26:0/3810302302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2978060950 0x7f2978060d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 -- 172.21.15.26:0/3810302302 >> v2:172.21.15.120:3300/0 conn(0x7f29780add30 msgr2=0x7f29780b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:54.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 --2- 172.21.15.26:0/3810302302 >> v2:172.21.15.120:3300/0 conn(0x7f29780add30 0x7f29780b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 -- 172.21.15.26:0/3810302302 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780b1110 con 0x7f2978061280 2022-07-20T02:35:54.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297e8f3700 1 --2- 172.21.15.26:0/3810302302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978061280 0x7f2978143450 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f29700032f0 tx=0x7f2970009290).ready entity=mon.0 client_cookie=d27bb08aa31b4fed server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297d0f0700 1 -- 172.21.15.26:0/3810302302 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f297000b680 con 0x7f2978061280 2022-07-20T02:35:54.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297d0f0700 1 -- 172.21.15.26:0/3810302302 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297000b7e0 con 0x7f2978061280 2022-07-20T02:35:54.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297d0f0700 1 -- 172.21.15.26:0/3810302302 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f297000b950 con 0x7f2978061280 2022-07-20T02:35:54.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- 172.21.15.26:0/3810302302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978061280 msgr2=0x7f2978143450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 --2- 172.21.15.26:0/3810302302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978061280 0x7f2978143450 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f29700032f0 tx=0x7f2970009290).stop 2022-07-20T02:35:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- 172.21.15.26:0/3810302302 shutdown_connections 2022-07-20T02:35:54.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 --2- 172.21.15.26:0/3810302302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978061280 0x7f2978143450 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 --2- 172.21.15.26:0/3810302302 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2978060950 0x7f2978060d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 --2- 172.21.15.26:0/3810302302 >> v2:172.21.15.120:3300/0 conn(0x7f29780add30 0x7f29780b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- 172.21.15.26:0/3810302302 >> 172.21.15.26:0/3810302302 conn(0x7f29780b3050 msgr2=0x7f29780b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- 172.21.15.26:0/3810302302 shutdown_connections 2022-07-20T02:35:54.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- 172.21.15.26:0/3810302302 wait complete. 2022-07-20T02:35:54.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 Processor -- start 2022-07-20T02:35:54.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.634+0000 7f297f0f4700 1 -- start start 2022-07-20T02:35:54.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2978061280 0x7f297813a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29780add30 0x7f29781534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978139830 con 0x7f2978060950 2022-07-20T02:35:54.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978150b40 con 0x7f2978061280 2022-07-20T02:35:54.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2978150de0 con 0x7f29780add30 2022-07-20T02:35:54.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44608/0 (socket says 172.21.15.26:44608) 2022-07-20T02:35:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 -- 172.21.15.26:0/2639487124 learned_addr learned my addr 172.21.15.26:0/2639487124 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e8f3700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29780add30 0x7f29781534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297d8f1700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2978061280 0x7f297813a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 -- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29780add30 msgr2=0x7f29781534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29780add30 0x7f29781534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 -- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2978061280 msgr2=0x7f297813a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2978061280 0x7f297813a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 -- 172.21.15.26:0/2639487124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2970009530 con 0x7f2978060950 2022-07-20T02:35:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297e0f2700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f297400c000 tx=0x7f2974006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2974010780 con 0x7f2978060950 2022-07-20T02:35:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29740108e0 con 0x7f2978060950 2022-07-20T02:35:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29781390f0 con 0x7f2978060950 2022-07-20T02:35:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2974010c40 con 0x7f2978060950 2022-07-20T02:35:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.638+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2978066380 con 0x7f2978060950 2022-07-20T02:35:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29780b13b0 con 0x7f2978060950 2022-07-20T02:35:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f297401c460 con 0x7f2978060950 2022-07-20T02:35:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f29677fe700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 0x7f2958055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f297406c560 con 0x7f2978060950 2022-07-20T02:35:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f297d8f1700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 0x7f2958055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:54.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.642+0000 7f297d8f1700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 0x7f2958055e10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f296800b000 tx=0x7f2968008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:54.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.650+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29740431b0 con 0x7f2978060950 2022-07-20T02:35:54.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.778+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29780574e0 con 0x7f2978060950 2022-07-20T02:35:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.778+0000 7f29677fe700 1 -- 172.21.15.26:0/2639487124 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v162) v1 ==== 104+0+10 (secure 0 0 0) 0x7f29740467d0 con 0x7f2978060950 2022-07-20T02:35:54.828 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 msgr2=0x7f2958055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 0x7f2958055e10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f296800b000 tx=0x7f2968008040).stop 2022-07-20T02:35:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 msgr2=0x7f2978138b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:54.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f297400c000 tx=0x7f2974006df0).stop 2022-07-20T02:35:54.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 shutdown_connections 2022-07-20T02:35:54.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2958053950 0x7f2958055e10 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29780add30 0x7f29781534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2978061280 0x7f297813a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 --2- 172.21.15.26:0/2639487124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2978060950 0x7f2978138b00 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 >> 172.21.15.26:0/2639487124 conn(0x7f29780b3050 msgr2=0x7f29780af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 shutdown_connections 2022-07-20T02:35:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.782+0000 7f297f0f4700 1 -- 172.21.15.26:0/2639487124 wait complete. 2022-07-20T02:35:54.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T02:35:55.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 Processor -- start 2022-07-20T02:35:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 -- start start 2022-07-20T02:35:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 0x7f64fc140700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0ab530 con 0x7f64fc147020 2022-07-20T02:35:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64fc0b1ce0 con 0x7f64fc05cf80 2022-07-20T02:35:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc05cc90 con 0x7f64fc140320 2022-07-20T02:35:55.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fbfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44620/0 (socket says 172.21.15.26:44620) 2022-07-20T02:35:55.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fbfff700 1 -- 172.21.15.26:0/438222068 learned_addr learned my addr 172.21.15.26:0/438222068 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:55.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/438222068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 0x7f64fc140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64faffd700 1 -- 172.21.15.26:0/438222068 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802877287 0 0) 0x7f64e4001110 con 0x7f64fc147020 2022-07-20T02:35:55.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64faffd700 1 -- 172.21.15.26:0/438222068 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64dc003260 con 0x7f64fc147020 2022-07-20T02:35:55.067 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 -- 172.21.15.26:0/438222068 >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 msgr2=0x7f64fc143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/438222068 >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 -- 172.21.15.26:0/438222068 >> v1:172.21.15.120:6789/0 conn(0x7f64fc147020 legacy=0x7f64fc05c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.068 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 -- 172.21.15.26:0/438222068 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0aab50 con 0x7f64fc140320 2022-07-20T02:35:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fbfff700 1 --2- 172.21.15.26:0/438222068 >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/438222068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 0x7f64fc140700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f64ec002f70 tx=0x7f64ec006df0).ready entity=mon.2 client_cookie=b93d64c476527980 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.069 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64faffd700 1 -- 172.21.15.26:0/438222068 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ec00d4a0 con 0x7f64fc140320 2022-07-20T02:35:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64faffd700 1 -- 172.21.15.26:0/438222068 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec000e90 con 0x7f64fc140320 2022-07-20T02:35:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f64faffd700 1 -- 172.21.15.26:0/438222068 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ec00d7f0 con 0x7f64fc140320 2022-07-20T02:35:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 -- 172.21.15.26:0/438222068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 msgr2=0x7f64fc140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.070 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.990+0000 7f65009eb700 1 --2- 172.21.15.26:0/438222068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 0x7f64fc140700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f64ec002f70 tx=0x7f64ec006df0).stop 2022-07-20T02:35:55.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- 172.21.15.26:0/438222068 shutdown_connections 2022-07-20T02:35:55.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 --2- 172.21.15.26:0/438222068 >> v2:172.21.15.26:3300/0 conn(0x7f64fc05cf80 0x7f64fc143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 --2- 172.21.15.26:0/438222068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc140320 0x7f64fc140700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- 172.21.15.26:0/438222068 >> 172.21.15.26:0/438222068 conn(0x7f64fc060310 msgr2=0x7f64fc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:55.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- 172.21.15.26:0/438222068 shutdown_connections 2022-07-20T02:35:55.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- 172.21.15.26:0/438222068 wait complete. 2022-07-20T02:35:55.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 Processor -- start 2022-07-20T02:35:55.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- start start 2022-07-20T02:35:55.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc05cf80 0x7f64fc1453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 0x7f64fc13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc13e4d0 con 0x7f64fc140320 2022-07-20T02:35:55.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc156870 con 0x7f64fc05cf80 2022-07-20T02:35:55.098 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f65009eb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0adcb0 con 0x7f64fc147020 2022-07-20T02:35:55.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64f2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.099 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64f2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44624/0 (socket says 172.21.15.26:44624) 2022-07-20T02:35:55.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64f2ffd700 1 -- 172.21.15.26:0/2948286993 learned_addr learned my addr 172.21.15.26:0/2948286993 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:55.100 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 0x7f64fc13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc05cf80 0x7f64fc1453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 -- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc05cf80 msgr2=0x7f64fc1453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc05cf80 0x7f64fc1453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 -- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 msgr2=0x7f64fc13daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.118 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64fbfff700 1 -- 172.21.15.26:0/2948286993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4009530 con 0x7f64fc147020 2022-07-20T02:35:55.119 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.994+0000 7f64f2ffd700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:35:55.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64fbfff700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 0x7f64fc13dfe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f64f400c000 tx=0x7f64f4009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.120 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64f400c770 con 0x7f64fc147020 2022-07-20T02:35:55.130 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f400c8d0 con 0x7f64fc147020 2022-07-20T02:35:55.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec00b180 con 0x7f64fc147020 2022-07-20T02:35:55.131 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64f400cc30 con 0x7f64fc147020 2022-07-20T02:35:55.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc0b45c0 con 0x7f64fc147020 2022-07-20T02:35:55.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64f4022070 con 0x7f64fc147020 2022-07-20T02:35:55.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc0ae2d0 con 0x7f64fc147020 2022-07-20T02:35:55.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 0x7f64e0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f64f406c850 con 0x7f64fc147020 2022-07-20T02:35:55.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 0x7f64e0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:54.998+0000 7f64fb7fe700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 0x7f64e0055ed0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f64ec01a000 tx=0x7f64ec014000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.006+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64f4043500 con 0x7f64fc147020 2022-07-20T02:35:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.138+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64fc0aadf0 con 0x7f64fc147020 2022-07-20T02:35:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.138+0000 7f64f97fa700 1 -- 172.21.15.26:0/2948286993 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v162) v1 ==== 100+0+10 (secure 0 0 0) 0x7f64f40160a0 con 0x7f64fc147020 2022-07-20T02:35:55.180 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2022-07-20T02:35:55.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 msgr2=0x7f64e0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 0x7f64e0055ed0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f64ec01a000 tx=0x7f64ec014000).stop 2022-07-20T02:35:55.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 msgr2=0x7f64fc13dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 0x7f64fc13dfe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f64f400c000 tx=0x7f64f4009290).stop 2022-07-20T02:35:55.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 shutdown_connections 2022-07-20T02:35:55.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f64fc140320 0x7f64fc13daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f64e0053a10 0x7f64e0055ed0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64fc147020 0x7f64fc13dfe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 --2- 172.21.15.26:0/2948286993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64fc05cf80 0x7f64fc1453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 >> 172.21.15.26:0/2948286993 conn(0x7f64fc060310 msgr2=0x7f64fc0ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:55.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 shutdown_connections 2022-07-20T02:35:55.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.142+0000 7f65009eb700 1 -- 172.21.15.26:0/2948286993 wait complete. 2022-07-20T02:35:55.294 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T02:35:55.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:35:55.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 Processor -- start 2022-07-20T02:35:55.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 -- start start 2022-07-20T02:35:55.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c90061050 0x7f7c90067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c900b1bc0 con 0x7f7c900b2920 2022-07-20T02:35:55.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c90147700 con 0x7f7c901484a0 2022-07-20T02:35:55.500 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c94e17700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9014b040 con 0x7f7c90061050 2022-07-20T02:35:55.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8e59c700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7c900b2920 0x7f7c90147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43910/0 (socket says 172.21.15.26:43910) 2022-07-20T02:35:55.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8e59c700 1 -- 172.21.15.26:0/2999892947 learned_addr learned my addr 172.21.15.26:0/2999892947 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:35:55.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8ed9d700 1 --2- 172.21.15.26:0/2999892947 >> v2:172.21.15.120:3300/0 conn(0x7f7c90061050 0x7f7c90067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.501 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282036723 0 0) 0x7f7c900b1bc0 con 0x7f7c900b2920 2022-07-20T02:35:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c70003260 con 0x7f7c900b2920 2022-07-20T02:35:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565546104 0 0) 0x7f7c90147700 con 0x7f7c901484a0 2022-07-20T02:35:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c900b1bc0 con 0x7f7c901484a0 2022-07-20T02:35:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259934026 0 0) 0x7f7c70003260 con 0x7f7c900b2920 2022-07-20T02:35:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 >> v2:172.21.15.120:3300/0 conn(0x7f7c90061050 msgr2=0x7f7c90067190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 --2- 172.21.15.26:0/2999892947 >> v2:172.21.15.120:3300/0 conn(0x7f7c90061050 0x7f7c90067190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 >> v1:172.21.15.120:6789/0 conn(0x7f7c901484a0 legacy=0x7f7c90066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90147930 con 0x7f7c900b2920 2022-07-20T02:35:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f7c78001660 con 0x7f7c900b2920 2022-07-20T02:35:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 >> v1:172.21.15.26:6789/0 conn(0x7f7c900b2920 legacy=0x7f7c90147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c700038e0 0x7f7c70003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 0x7f7c70004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004e70 0x7f7c7000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9014b040 con 0x7f7c70004500 2022-07-20T02:35:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7000bcf0 con 0x7f7c700038e0 2022-07-20T02:35:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7000beb0 con 0x7f7c70004e70 2022-07-20T02:35:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 0x7f7c70004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8ed9d700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004e70 0x7f7c7000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8e59c700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c700038e0 0x7f7c70003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 -- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004e70 msgr2=0x7f7c7000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004e70 0x7f7c7000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 -- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c700038e0 msgr2=0x7f7c70003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:35:55.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c700038e0 0x7f7c70003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 -- 172.21.15.26:0/2999892947 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70003260 con 0x7f7c70004500 2022-07-20T02:35:55.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8ed9d700 1 -- 172.21.15.26:0/2999892947 reap_dead start 2022-07-20T02:35:55.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8dd9b700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 0x7f7c70004980 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f7c88001110 tx=0x7f7c88007a00).ready entity=mon.0 client_cookie=28b3d50cca01a829 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c88007d10 con 0x7f7c70004500 2022-07-20T02:35:55.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88007030 con 0x7f7c70004500 2022-07-20T02:35:55.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.422+0000 7f7c8d59a700 1 -- 172.21.15.26:0/2999892947 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c88007390 con 0x7f7c70004500 2022-07-20T02:35:55.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 msgr2=0x7f7c70004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.569 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 0x7f7c70004980 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f7c88001110 tx=0x7f7c88007a00).stop 2022-07-20T02:35:55.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- 172.21.15.26:0/2999892947 shutdown_connections 2022-07-20T02:35:55.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 --2- 172.21.15.26:0/2999892947 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c70004500 0x7f7c70004980 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- 172.21.15.26:0/2999892947 >> 172.21.15.26:0/2999892947 conn(0x7f7c900b60a0 msgr2=0x7f7c900b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:55.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- 172.21.15.26:0/2999892947 shutdown_connections 2022-07-20T02:35:55.571 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- 172.21.15.26:0/2999892947 wait complete. 2022-07-20T02:35:55.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 Processor -- start 2022-07-20T02:35:55.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.426+0000 7f7c94e17700 1 -- start start 2022-07-20T02:35:55.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 0x7f7c9014a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c90065070 0x7f7c90066280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9015a9e0 con 0x7f7c90066770 2022-07-20T02:35:55.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c90065d90 con 0x7f7c90065070 2022-07-20T02:35:55.573 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9015ac10 con 0x7f7c70004500 2022-07-20T02:35:55.574 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8e59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 0x7f7c9014a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44646/0 (socket says 172.21.15.26:44646) 2022-07-20T02:35:55.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 -- 172.21.15.26:0/1522217066 learned_addr learned my addr 172.21.15.26:0/1522217066 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:35:55.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8dd9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c90065070 0x7f7c90066280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8e59c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 0x7f7c9014a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:37926/0 (socket says 172.21.15.26:37926) 2022-07-20T02:35:55.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 -- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 msgr2=0x7f7c9014a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 0x7f7c9014a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 -- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c90065070 msgr2=0x7f7c90066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c90065070 0x7f7c90066280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 -- 172.21.15.26:0/1522217066 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900a9820 con 0x7f7c90066770 2022-07-20T02:35:55.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c8ed9d700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f7c8000b000 tx=0x7f7c80008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80011e40 con 0x7f7c90066770 2022-07-20T02:35:55.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80013550 con 0x7f7c90066770 2022-07-20T02:35:55.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c90064b60 con 0x7f7c90066770 2022-07-20T02:35:55.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c800136c0 con 0x7f7c90066770 2022-07-20T02:35:55.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.430+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c90156a60 con 0x7f7c90066770 2022-07-20T02:35:55.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7c8000f070 con 0x7f7c90066770 2022-07-20T02:35:55.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c877fe700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 0x7f7c74055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:35:55.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9015add0 con 0x7f7c90066770 2022-07-20T02:35:55.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7c8006c1a0 con 0x7f7c90066770 2022-07-20T02:35:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c8e59c700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 0x7f7c74055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:35:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.434+0000 7f7c8e59c700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 0x7f7c74055ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f7c7800d000 tx=0x7f7c78007040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:35:55.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.442+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c80042f60 con 0x7f7c90066770 2022-07-20T02:35:55.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c9014ada0 con 0x7f7c90066770 2022-07-20T02:35:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c877fe700 1 -- 172.21.15.26:0/1522217066 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2498 (secure 0 0 0) 0x7f7c80046580 con 0x7f7c90066770 2022-07-20T02:35:55.740 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:35:55.743 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:35:46.155980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1434658358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6835","nonce":1434658358},{"type":"v1","addr":"172.21.15.26:6837","nonce":1434658358}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:35:55.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 msgr2=0x7f7c74055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 0x7f7c74055ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f7c7800d000 tx=0x7f7c78007040).stop 2022-07-20T02:35:55.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 msgr2=0x7f7c90156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:35:55.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f7c8000b000 tx=0x7f7c80008040).stop 2022-07-20T02:35:55.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 shutdown_connections 2022-07-20T02:35:55.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7c90066770 0x7f7c90156fa0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7c74053b30 0x7f7c74055ff0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c90065070 0x7f7c90066280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 --2- 172.21.15.26:0/1522217066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c70004500 0x7f7c9014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:35:55.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 >> 172.21.15.26:0/1522217066 conn(0x7f7c900b60a0 msgr2=0x7f7c900b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:35:55.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 shutdown_connections 2022-07-20T02:35:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:35:55.610+0000 7f7c94e17700 1 -- 172.21.15.26:0/1522217066 wait complete. 2022-07-20T02:35:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 89 2022-07-20T02:35:56.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T02:35:44.142374+0000', 'modified': '2022-07-20T02:35:46.155980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8859': {'gid': 8859, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.26:6837/1434658358', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6835', 'nonce': 1434658358}, {'type': 'v1', 'addr': '172.21.15.26:6837', 'nonce': 1434658358}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T02:35:56.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T02:35:56.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-20T02:35:56.267 INFO:teuthology.orchestra.run.smithi026.stdout:{"cluster_fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","whoami":0,"id":8859,"want_state":"up:active","state":"up:active","rank_uptime":12.112672337999999,"mdsmap_epoch":89,"osdmap_epoch":162,"osdmap_epoch_barrier":161,"uptime":17.566876471} 2022-07-20T02:35:56.268 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", "id": 8859, "mdsmap_epoch": 89, "osdmap_epoch": 162, "osdmap_epoch_barrier": 161, "rank_uptime": 12.112672338, "state": "up:active", "uptime": 17.566876471, "want_state": "up:active", "whoami": 0 } 2022-07-20T02:35:56.269 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T02:35:56.269 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T02:35:56.304 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi176.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T02:35:56.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.311 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:56.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:35:56.418 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:56.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:56.442 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:56.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.447 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:35:56.498 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T02:35:56.499 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:56.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2022-07-20T02:35:56.529 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T02:35:56.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T02:35:56.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2022-07-20T02:35:56.582+0000 7f365121f080 -1 init, newargv = 0x55749039f780 newargc=9ceph-fuse[23200 2022-07-20T02:35:56.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:]: starting ceph client 2022-07-20T02:35:56.594 INFO:teuthology.orchestra.run.smithi176.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T02:35:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T02:35:56.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.616 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2022-07-20T02:35:56.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23200]: starting fuse 2022-07-20T02:35:56.638 INFO:teuthology.orchestra.run.smithi176.stdout:52 2022-07-20T02:35:56.639 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T02:35:56.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo python3 -c ' 2022-07-20T02:35:56.658 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2022-07-20T02:35:56.658 DEBUG:teuthology.orchestra.run.smithi176:> import re 2022-07-20T02:35:56.659 DEBUG:teuthology.orchestra.run.smithi176:> import os 2022-07-20T02:35:56.660 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2022-07-20T02:35:56.661 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:56.679 DEBUG:teuthology.orchestra.run.smithi176:> def find_socket(client_name): 2022-07-20T02:35:56.679 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T02:35:56.680 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2022-07-20T02:35:56.681 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:56.702 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2022-07-20T02:35:56.703 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2022-07-20T02:35:56.704 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2022-07-20T02:35:56.705 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2022-07-20T02:35:56.705 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:56.706 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2022-07-20T02:35:56.707 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T02:35:56.708 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T02:35:56.709 DEBUG:teuthology.orchestra.run.smithi176:> return f 2022-07-20T02:35:56.710 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T02:35:56.711 DEBUG:teuthology.orchestra.run.smithi176:> 2022-07-20T02:35:56.712 DEBUG:teuthology.orchestra.run.smithi176:> print(find_socket("client.1")) 2022-07-20T02:35:56.713 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-07-20T02:35:56.768 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.23200.asok 2022-07-20T02:35:56.780 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23200.asok 2022-07-20T02:35:56.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23200.asok status 2022-07-20T02:35:56.877 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2022-07-20T02:35:56.877 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2022-07-20T02:35:56.878 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T02:35:56.878 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T02:35:56.879 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2022-07-20T02:35:56.879 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2022-07-20T02:35:56.880 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T02:35:56.880 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "23200", 2022-07-20T02:35:56.881 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2022-07-20T02:35:56.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:56.881 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2022-07-20T02:35:56.882 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2022-07-20T02:35:56.882 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 8773, 2022-07-20T02:35:56.883 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2022-07-20T02:35:56.883 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2022-07-20T02:35:56.884 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2022-07-20T02:35:56.884 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 8773 2022-07-20T02:35:56.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:56.885 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:35:56.886 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:35:56.887 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:35:56.887 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3861345726 2022-07-20T02:35:56.887 INFO:teuthology.orchestra.run.smithi176.stdout: } 2022-07-20T02:35:56.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:56.888 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2022-07-20T02:35:56.889 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2022-07-20T02:35:56.889 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "172.21.15.176:0", 2022-07-20T02:35:56.900 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 3861345726 2022-07-20T02:35:56.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2022-07-20T02:35:56.901 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.8773 172.21.15.176:0/3861345726", 2022-07-20T02:35:56.902 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "172.21.15.176:0/3861345726", 2022-07-20T02:35:56.902 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2022-07-20T02:35:56.903 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 89, 2022-07-20T02:35:56.903 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 162, 2022-07-20T02:35:56.904 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2022-07-20T02:35:56.908 INFO:teuthology.orchestra.run.smithi176.stdout: "blacklisted": false, 2022-07-20T02:35:56.909 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "cephfs" 2022-07-20T02:35:56.909 INFO:teuthology.orchestra.run.smithi176.stdout:} 2022-07-20T02:35:56.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.912 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:56.913 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:56.948 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:35:56.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:56.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:56.952 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:57.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:57.026 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:57.076 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:57.108 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:35:57.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:57.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:57.110 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:57.115 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:57.130 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2022-07-20T02:35:57.131 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T02:35:57.140 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2022-07-20T02:35:57.140 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T02:35:57.141 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:57.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[23200]: fuse finished with error 0 and tester_r 0 2022-07-20T02:35:57.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:35:57.209 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:35:57.209 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:35:57.243 INFO:teuthology.orchestra.run.smithi176.stdout:ext2/ext3 2022-07-20T02:35:57.244 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T02:35:57.244 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T02:36:03.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T02:36:03.351 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T02:36:03.387 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T02:36:03.388 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-20T02:36:03.431 INFO:teuthology.orchestra.run.smithi159.stdout: File: "." 2022-07-20T02:36:03.431 INFO:teuthology.orchestra.run.smithi159.stdout: ID: 0 Namelen: 255 Type: fuseblk 2022-07-20T02:36:03.432 INFO:teuthology.orchestra.run.smithi159.stdout:Block size: 4194304 Fundamental block size: 4194304 2022-07-20T02:36:03.432 INFO:teuthology.orchestra.run.smithi159.stdout:Blocks: Total: 86526 Free: 86526 Available: 86526 2022-07-20T02:36:03.433 INFO:teuthology.orchestra.run.smithi159.stdout:Inodes: Total: 1 Free: 0 2022-07-20T02:36:03.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T02:36:03.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.550+0000 7f18bf576700 1 Processor -- start 2022-07-20T02:36:03.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.550+0000 7f18bf576700 1 -- start start 2022-07-20T02:36:03.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f18b80b2920 0x7f18b8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.558 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8066a80 0x7f18b8066ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b814b040 con 0x7f18b80b2920 2022-07-20T02:36:03.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80a8860 con 0x7f18b8061050 2022-07-20T02:36:03.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80b1bc0 con 0x7f18b8066a80 2022-07-20T02:36:03.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f18b80b2920 0x7f18b8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.559 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bcb11700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bcb11700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:36112/0 (socket says 172.21.15.26:36112) 2022-07-20T02:36:03.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bcb11700 1 -- 172.21.15.26:0/306788119 learned_addr learned my addr 172.21.15.26:0/306788119 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:03.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bdb13700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8066a80 0x7f18b8066ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.560 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 -- 172.21.15.26:0/306788119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8066a80 msgr2=0x7f18b8066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:03.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8066a80 0x7f18b8066ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 -- 172.21.15.26:0/306788119 >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 msgr2=0x7f18b8148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:03.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 --2- 172.21.15.26:0/306788119 >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.561 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 -- 172.21.15.26:0/306788119 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8147700 con 0x7f18b80b2920 2022-07-20T02:36:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bcb11700 1 --2- 172.21.15.26:0/306788119 >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:36:03.562 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bd312700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b80b2920 0x7f18b8147d90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f18ac002f70 tx=0x7f18ac006df0).ready entity=mon.0 client_cookie=8eace566b7602105 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:03.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18abfff700 1 -- 172.21.15.26:0/306788119 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18ac00d4a0 con 0x7f18b80b2920 2022-07-20T02:36:03.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18abfff700 1 -- 172.21.15.26:0/306788119 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac000e90 con 0x7f18b80b2920 2022-07-20T02:36:03.563 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18abfff700 1 -- 172.21.15.26:0/306788119 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18ac00d7f0 con 0x7f18b80b2920 2022-07-20T02:36:03.564 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- 172.21.15.26:0/306788119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b80b2920 msgr2=0x7f18b8147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:03.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b80b2920 0x7f18b8147d90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f18ac002f70 tx=0x7f18ac006df0).stop 2022-07-20T02:36:03.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- 172.21.15.26:0/306788119 shutdown_connections 2022-07-20T02:36:03.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b8066a80 0x7f18b8066ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.565 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- 172.21.15.26:0/306788119 >> v2:172.21.15.120:3300/0 conn(0x7f18b8061050 0x7f18b8148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 --2- 172.21.15.26:0/306788119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b80b2920 0x7f18b8147d90 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- 172.21.15.26:0/306788119 >> 172.21.15.26:0/306788119 conn(0x7f18b80b60a0 msgr2=0x7f18b80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- 172.21.15.26:0/306788119 shutdown_connections 2022-07-20T02:36:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.554+0000 7f18bf576700 1 -- 172.21.15.26:0/306788119 wait complete. 2022-07-20T02:36:03.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 Processor -- start 2022-07-20T02:36:03.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 -- start start 2022-07-20T02:36:03.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8066a80 0x7f18b8062810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b80b2920 0x7f18b8062d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8159ff0 con 0x7f18b8061050 2022-07-20T02:36:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b815a220 con 0x7f18b8066a80 2022-07-20T02:36:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bf576700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b815a4c0 con 0x7f18b80b2920 2022-07-20T02:36:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44656/0 (socket says 172.21.15.26:44656) 2022-07-20T02:36:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 -- 172.21.15.26:0/1213266736 learned_addr learned my addr 172.21.15.26:0/1213266736 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bdb13700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b80b2920 0x7f18b8062d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bcb11700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8066a80 0x7f18b8062810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 -- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b80b2920 msgr2=0x7f18b8062d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b80b2920 0x7f18b8062d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 -- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8066a80 msgr2=0x7f18b8062810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8066a80 0x7f18b8062810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 -- 172.21.15.26:0/1213266736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b400b1e0 con 0x7f18b8061050 2022-07-20T02:36:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.558+0000 7f18bd312700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f18ac002f70 tx=0x7f18ac014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18ac013ca0 con 0x7f18b8061050 2022-07-20T02:36:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac013e00 con 0x7f18b8061050 2022-07-20T02:36:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac00b180 con 0x7f18b8061050 2022-07-20T02:36:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18ac01e5f0 con 0x7f18b8061050 2022-07-20T02:36:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8065350 con 0x7f18b8061050 2022-07-20T02:36:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b814ada0 con 0x7f18b8061050 2022-07-20T02:36:03.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f18ac025030 con 0x7f18b8061050 2022-07-20T02:36:03.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 0x7f18a0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:03.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f18ac06cca0 con 0x7f18b8061050 2022-07-20T02:36:03.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18bcb11700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 0x7f18a0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:03.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.562+0000 7f18bcb11700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 0x7f18a0055ea0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f18b400b000 tx=0x7f18b4006eb0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:03.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.570+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f18ac071070 con 0x7f18b8061050 2022-07-20T02:36:03.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.694+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f18b80574e0 con 0x7f18b8061050 2022-07-20T02:36:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18aa7fc700 1 -- 172.21.15.26:0/1213266736 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v162) v1 ==== 81+0+34099 (secure 0 0 0) 0x7f18ac023070 con 0x7f18b8061050 2022-07-20T02:36:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:36:03.713 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T02:36:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 162, 2022-07-20T02:36:03.713 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "36c84aa7-9150-4850-bbd5-e28fb0ff0eb0", 2022-07-20T02:36:03.726 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2022-07-20T02:25:16.528661+0000", 2022-07-20T02:36:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2022-07-20T02:35:46.156070+0000", 2022-07-20T02:36:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:36:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2022-07-20T02:25:24.287145+0000", 2022-07-20T02:36:03.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T02:36:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2022-07-20T02:36:03.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2022-07-20T02:36:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2022-07-20T02:36:03.736 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2022-07-20T02:36:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2022-07-20T02:36:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2022-07-20T02:36:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:36:03.737 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 5, 2022-07-20T02:36:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T02:36:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T02:36:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T02:36:03.738 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2022-07-20T02:36:03.739 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 30, 2022-07-20T02:36:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 8, 2022-07-20T02:36:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "jewel", 2022-07-20T02:36:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2022-07-20T02:36:03.740 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "octopus", 2022-07-20T02:36:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2022-07-20T02:36:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2022-07-20T02:36:03.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "device_health_metrics", 2022-07-20T02:36:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:19.431119+0000", 2022-07-20T02:36:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:36:03.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:36:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:36:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2022-07-20T02:36:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2022-07-20T02:36:03.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2022-07-20T02:36:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2022-07-20T02:36:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2022-07-20T02:36:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:36:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:36:03.745 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:36:03.755 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:36:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:36:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:36:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:36:03.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "15", 2022-07-20T02:36:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:36:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:36:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:36:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:36:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:36:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:36:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:36:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:36:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:36:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:36:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:36:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:36:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:36:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:36:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:36:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:36:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:36:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:36:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:36:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:36:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:36:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:36:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:36:03.800 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:36:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:36:03.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:36:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:36:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:36:03.812 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:36:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:36:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:36:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:36:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:36:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:36:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:36:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:36:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:36:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:36:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2022-07-20T02:36:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:36:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_devicehealth": {} 2022-07-20T02:36:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2022-07-20T02:36:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "rbd", 2022-07-20T02:36:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:25:28.375708+0000", 2022-07-20T02:36:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:36:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:36:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:36:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:36:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:36:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:36:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:36:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:36:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:36:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:36:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:36:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:36:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:36:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:36:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:36:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:36:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:36:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:36:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:36:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:36:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "18", 2022-07-20T02:36:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:36:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:36:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:36:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:36:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:36:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:36:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:36:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:36:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:36:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:36:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:36:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:36:03.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:36:03.963 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:36:03.963 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:36:03.964 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:36:03.964 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:36:03.964 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:36:03.964 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:36:03.965 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:36:03.965 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:36:03.965 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:36:03.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:36:03.976 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:36:03.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:36:03.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:36:03.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.978 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:36:03.978 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:36:03.978 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:36:03.978 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:36:03.979 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:36:03.979 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:36:03.979 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:36:03.979 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:36:03.980 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:36:03.980 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:36:03.980 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:36:03.980 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:36:03.980 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:36:03.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd": {} 2022-07-20T02:36:03.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:03.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:03.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 29, 2022-07-20T02:36:03.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_metadata", 2022-07-20T02:36:03.982 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:35:42.427470+0000", 2022-07-20T02:36:03.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:36:03.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:36:03.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:36:03.984 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:36:03.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:36:03.986 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:36:03.987 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:36:03.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:36:03.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:36:03.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:36:03.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:36:03.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:36:03.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:36:03.988 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:36:03.989 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:36:03.989 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:36:03.989 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:36:03.990 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:36:03.990 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:36:03.990 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:36:03.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "160", 2022-07-20T02:36:03.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:36:03.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:36:03.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:36:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:36:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:36:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:36:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:36:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:36:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:36:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:36:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:36:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:36:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:36:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:36:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:36:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:36:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:36:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:36:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:36:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:36:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:36:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:36:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:36:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:36:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:36:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:36:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:36:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:36:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:36:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:36:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:36:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:36:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:36:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:36:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:36:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:36:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:36:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2022-07-20T02:36:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_bias": 4, 2022-07-20T02:36:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 16, 2022-07-20T02:36:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_priority": 5 2022-07-20T02:36:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:36:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:36:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "metadata": "cephfs" 2022-07-20T02:36:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 30, 2022-07-20T02:36:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "cephfs_data", 2022-07-20T02:36:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2022-07-20T02:35:43.067323+0000", 2022-07-20T02:36:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2022-07-20T02:36:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2022-07-20T02:36:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2022-07-20T02:36:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 2, 2022-07-20T02:36:04.036 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 1, 2022-07-20T02:36:04.037 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2022-07-20T02:36:04.037 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2022-07-20T02:36:04.057 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2022-07-20T02:36:04.057 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 8, 2022-07-20T02:36:04.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 8, 2022-07-20T02:36:04.071 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 8, 2022-07-20T02:36:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 8, 2022-07-20T02:36:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 8, 2022-07-20T02:36:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2022-07-20T02:36:04.102 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2022-07-20T02:36:04.103 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2022-07-20T02:36:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2022-07-20T02:36:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2022-07-20T02:36:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2022-07-20T02:36:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2022-07-20T02:36:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "162", 2022-07-20T02:36:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2022-07-20T02:36:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T02:36:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T02:36:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2022-07-20T02:36:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2022-07-20T02:36:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2022-07-20T02:36:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2022-07-20T02:36:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2022-07-20T02:36:04.154 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2022-07-20T02:36:04.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2022-07-20T02:36:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2022-07-20T02:36:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2022-07-20T02:36:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2022-07-20T02:36:04.159 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2022-07-20T02:36:04.160 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2022-07-20T02:36:04.165 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2022-07-20T02:36:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2022-07-20T02:36:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2022-07-20T02:36:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T02:36:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T02:36:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T02:36:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2022-07-20T02:36:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2022-07-20T02:36:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2022-07-20T02:36:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2022-07-20T02:36:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2022-07-20T02:36:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2022-07-20T02:36:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2022-07-20T02:36:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2022-07-20T02:36:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2022-07-20T02:36:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2022-07-20T02:36:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T02:36:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2022-07-20T02:36:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2022-07-20T02:36:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2022-07-20T02:36:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2022-07-20T02:36:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2022-07-20T02:36:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2022-07-20T02:36:04.200 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2022-07-20T02:36:04.201 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs": { 2022-07-20T02:36:04.201 INFO:teuthology.orchestra.run.smithi026.stdout: "data": "cephfs" 2022-07-20T02:36:04.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.202 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:36:04.202 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2022-07-20T02:36:04.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.203 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:36:04.203 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b99f86d-740c-4e86-aa14-2c202b408772", 2022-07-20T02:36:04.203 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:04.203 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:04.204 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:04.204 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:04.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:04.204 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:04.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:04.205 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 159, 2022-07-20T02:36:04.205 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:04.205 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2022-07-20T02:36:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2022-07-20T02:36:04.279 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.280 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.280 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:04.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.282 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2022-07-20T02:36:04.282 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2022-07-20T02:36:04.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2022-07-20T02:36:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2022-07-20T02:36:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.318 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:04.319 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.319 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.320 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2022-07-20T02:36:04.320 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.321 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2022-07-20T02:36:04.321 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23049 2022-07-20T02:36:04.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/23049", 2022-07-20T02:36:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/23049", 2022-07-20T02:36:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/23049", 2022-07-20T02:36:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/23049", 2022-07-20T02:36:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.387 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:36:04.388 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "0256d8ae-ca5e-445d-8c9e-25203aeea92b", 2022-07-20T02:36:04.388 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:04.389 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:04.389 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:04.417 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:04.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:04.417 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:04.420 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:04.420 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 158, 2022-07-20T02:36:04.420 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:04.421 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:04.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:04.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.434 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2022-07-20T02:36:04.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2022-07-20T02:36:04.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:04.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2022-07-20T02:36:04.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2022-07-20T02:36:04.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.481 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2022-07-20T02:36:04.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2022-07-20T02:36:04.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.535 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:04.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2022-07-20T02:36:04.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2022-07-20T02:36:04.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23050 2022-07-20T02:36:04.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.586 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/23050", 2022-07-20T02:36:04.587 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/23050", 2022-07-20T02:36:04.587 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/23050", 2022-07-20T02:36:04.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/23050", 2022-07-20T02:36:04.596 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:04.597 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:04.597 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:04.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.620 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:36:04.620 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e7012e19-7787-4117-be57-55c92bd2e289", 2022-07-20T02:36:04.621 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:04.622 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:04.623 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:04.623 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:04.623 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:04.624 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:04.624 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:04.625 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 159, 2022-07-20T02:36:04.625 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:04.625 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:04.626 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:04.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2022-07-20T02:36:04.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2022-07-20T02:36:04.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.712 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2022-07-20T02:36:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2022-07-20T02:36:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2022-07-20T02:36:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2022-07-20T02:36:04.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.834 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:04.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:04.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:04.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2022-07-20T02:36:04.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:04.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2022-07-20T02:36:04.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23051 2022-07-20T02:36:04.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:04.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/23051", 2022-07-20T02:36:04.901 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/23051", 2022-07-20T02:36:04.902 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/23051", 2022-07-20T02:36:04.902 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/23051", 2022-07-20T02:36:04.903 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:04.903 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:04.904 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:04.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:04.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:04.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:04.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:36:05.026 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "8abca3ca-a3f0-44e2-80e9-a3d7ff287442", 2022-07-20T02:36:05.041 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:05.041 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:05.042 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:05.042 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:05.076 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:05.076 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:05.077 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:05.078 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 159, 2022-07-20T02:36:05.078 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:05.079 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:05.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:05.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2022-07-20T02:36:05.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2022-07-20T02:36:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2022-07-20T02:36:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2022-07-20T02:36:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2022-07-20T02:36:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2022-07-20T02:36:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2022-07-20T02:36:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2022-07-20T02:36:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23052 2022-07-20T02:36:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/23052", 2022-07-20T02:36:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/23052", 2022-07-20T02:36:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/23052", 2022-07-20T02:36:05.174 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/23052", 2022-07-20T02:36:05.180 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:05.180 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:05.180 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:05.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.181 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:36:05.181 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf", 2022-07-20T02:36:05.182 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:05.182 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:05.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:05.182 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:05.183 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:05.209 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:05.209 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:05.209 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 156, 2022-07-20T02:36:05.210 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:05.210 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:05.210 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:05.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6816", 2022-07-20T02:36:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6817", 2022-07-20T02:36:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.218 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:05.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6818", 2022-07-20T02:36:05.219 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6819", 2022-07-20T02:36:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:05.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6822", 2022-07-20T02:36:05.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6823", 2022-07-20T02:36:05.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.241 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:05.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6820", 2022-07-20T02:36:05.266 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6821", 2022-07-20T02:36:05.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23448 2022-07-20T02:36:05.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.283 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6817/23448", 2022-07-20T02:36:05.283 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6819/23448", 2022-07-20T02:36:05.283 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6823/23448", 2022-07-20T02:36:05.283 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6821/23448", 2022-07-20T02:36:05.284 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:05.284 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:05.284 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:05.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:36:05.285 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3", 2022-07-20T02:36:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 160, 2022-07-20T02:36:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6800", 2022-07-20T02:36:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6801", 2022-07-20T02:36:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6802", 2022-07-20T02:36:05.305 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.306 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6803", 2022-07-20T02:36:05.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6806", 2022-07-20T02:36:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6807", 2022-07-20T02:36:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:05.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6804", 2022-07-20T02:36:05.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6805", 2022-07-20T02:36:05.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23447 2022-07-20T02:36:05.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6801/23447", 2022-07-20T02:36:05.359 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6803/23447", 2022-07-20T02:36:05.359 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6807/23447", 2022-07-20T02:36:05.359 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6805/23447", 2022-07-20T02:36:05.359 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:05.360 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:05.360 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:05.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:36:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "87a7cb28-e512-4d57-910f-5b4b8bc96eef", 2022-07-20T02:36:05.362 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:05.363 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:05.365 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:05.366 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:05.367 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 159, 2022-07-20T02:36:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:05.379 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:05.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6808", 2022-07-20T02:36:05.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6809", 2022-07-20T02:36:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:05.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6810", 2022-07-20T02:36:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6811", 2022-07-20T02:36:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.445 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:05.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6814", 2022-07-20T02:36:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6815", 2022-07-20T02:36:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6812", 2022-07-20T02:36:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6813", 2022-07-20T02:36:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23449 2022-07-20T02:36:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6809/23449", 2022-07-20T02:36:05.503 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6811/23449", 2022-07-20T02:36:05.503 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6815/23449", 2022-07-20T02:36:05.503 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6813/23449", 2022-07-20T02:36:05.504 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:05.504 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:05.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:05.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.564 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:36:05.565 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b1607f07-e4fb-42df-84d9-b5c855babf2e", 2022-07-20T02:36:05.565 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2022-07-20T02:36:05.566 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2022-07-20T02:36:05.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2022-07-20T02:36:05.567 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2022-07-20T02:36:05.567 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2022-07-20T02:36:05.567 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2022-07-20T02:36:05.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2022-07-20T02:36:05.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 159, 2022-07-20T02:36:05.568 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2022-07-20T02:36:05.569 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2022-07-20T02:36:05.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2022-07-20T02:36:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6824", 2022-07-20T02:36:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6825", 2022-07-20T02:36:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2022-07-20T02:36:05.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6826", 2022-07-20T02:36:05.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6827", 2022-07-20T02:36:05.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.623 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2022-07-20T02:36:05.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6830", 2022-07-20T02:36:05.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6831", 2022-07-20T02:36:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2022-07-20T02:36:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2022-07-20T02:36:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2022-07-20T02:36:05.635 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6828", 2022-07-20T02:36:05.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2022-07-20T02:36:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.120:6829", 2022-07-20T02:36:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 23460 2022-07-20T02:36:05.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.120:6825/23460", 2022-07-20T02:36:05.639 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.120:6827/23460", 2022-07-20T02:36:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.120:6831/23460", 2022-07-20T02:36:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.120:6829/23460", 2022-07-20T02:36:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2022-07-20T02:36:05.640 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2022-07-20T02:36:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2022-07-20T02:36:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2022-07-20T02:36:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:36:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2022-07-20T02:36:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2022-07-20T02:36:05.642 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.643 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.444208+0000", 2022-07-20T02:36:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.644 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2022-07-20T02:36:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.645 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.626271+0000", 2022-07-20T02:36:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.647 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2022-07-20T02:36:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.648 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.733012+0000", 2022-07-20T02:36:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2022-07-20T02:36:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.650 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.651 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.679666+0000", 2022-07-20T02:36:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2022-07-20T02:36:05.652 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.653 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.704323+0000", 2022-07-20T02:36:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2022-07-20T02:36:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.588747+0000", 2022-07-20T02:36:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2022-07-20T02:36:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.961520+0000", 2022-07-20T02:36:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2022-07-20T02:36:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2022-07-20T02:36:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2022-07-20T02:36:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2022-07-20T02:36:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2022-07-20T02:36:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540138292840890367, 2022-07-20T02:36:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2022-07-20T02:36:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2022-07-20T02:25:22.657760+0000", 2022-07-20T02:36:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2022-07-20T02:36:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2022-07-20T02:36:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2022-07-20T02:36:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2022-07-20T02:36:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2022-07-20T02:36:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2022-07-20T02:36:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "blacklist": {}, 2022-07-20T02:36:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2022-07-20T02:36:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_data": { 2022-07-20T02:36:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-device-class": "", 2022-07-20T02:36:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:36:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-root": "default", 2022-07-20T02:36:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-20T02:36:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:36:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "2", 2022-07-20T02:36:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:36:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:36:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van", 2022-07-20T02:36:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "w": "8" 2022-07-20T02:36:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2022-07-20T02:36:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2022-07-20T02:36:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2022-07-20T02:36:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2022-07-20T02:36:05.686 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2022-07-20T02:36:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "ruleset-failure-domain": "osd", 2022-07-20T02:36:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2022-07-20T02:36:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2022-07-20T02:36:05.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2022-07-20T02:36:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2022-07-20T02:36:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2022-07-20T02:36:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2022-07-20T02:36:05.688 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2022-07-20T02:36:05.689 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {} 2022-07-20T02:36:05.689 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T02:36:05.691 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 msgr2=0x7f18a0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 0x7f18a0055ea0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f18b400b000 tx=0x7f18b4006eb0).stop 2022-07-20T02:36:05.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 msgr2=0x7f18b8065cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f18ac002f70 tx=0x7f18ac014000).stop 2022-07-20T02:36:05.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 shutdown_connections 2022-07-20T02:36:05.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f18a00539e0 0x7f18a0055ea0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18b80b2920 0x7f18b8062d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b8066a80 0x7f18b8062810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 --2- 172.21.15.26:0/1213266736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f18b8061050 0x7f18b8065cc0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 >> 172.21.15.26:0/1213266736 conn(0x7f18b80b60a0 msgr2=0x7f18b8147cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:05.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 shutdown_connections 2022-07-20T02:36:05.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:03.698+0000 7f18bf576700 1 -- 172.21.15.26:0/1213266736 wait complete. 2022-07-20T02:36:05.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:36:05.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 Processor -- start 2022-07-20T02:36:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- start start 2022-07-20T02:36:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40adf90 0x7fe1a40b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1a40614b0 0x7fe1a4061940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a40ae370 con 0x7fe1a40b14b0 2022-07-20T02:36:05.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a40b28b0 con 0x7fe1a40614b0 2022-07-20T02:36:05.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40b0610 con 0x7fe1a40adf90 2022-07-20T02:36:05.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1a40614b0 0x7fe1a4061940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fe1a40614b0 0x7fe1a4061940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44668/0 (socket says 172.21.15.26:44668) 2022-07-20T02:36:05.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 -- 172.21.15.26:0/1903060418 learned_addr learned my addr 172.21.15.26:0/1903060418 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:05.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a8edb700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40adf90 0x7fe1a40b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 -- 172.21.15.26:0/1903060418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40adf90 msgr2=0x7fe1a40b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:05.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40adf90 0x7fe1a40b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 -- 172.21.15.26:0/1903060418 >> v1:172.21.15.120:6789/0 conn(0x7fe1a40b14b0 legacy=0x7fe1a4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 -- 172.21.15.26:0/1903060418 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40b08b0 con 0x7fe1a40614b0 2022-07-20T02:36:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a37fe700 1 -- 172.21.15.26:0/1903060418 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406380373 0 0) 0x7fe1a40ae370 con 0x7fe1a40b14b0 2022-07-20T02:36:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a96dc700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40614b0 0x7fe1a4061940 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7fe19c002f70 tx=0x7fe19c006eb0).ready entity=mon.0 client_cookie=3d8fb4b4d5039dc4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:05.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a37fe700 1 -- 172.21.15.26:0/1903060418 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe19c00d570 con 0x7fe1a40614b0 2022-07-20T02:36:05.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a37fe700 1 -- 172.21.15.26:0/1903060418 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c000e90 con 0x7fe1a40614b0 2022-07-20T02:36:05.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1a37fe700 1 -- 172.21.15.26:0/1903060418 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe19c00d8c0 con 0x7fe1a40614b0 2022-07-20T02:36:05.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- 172.21.15.26:0/1903060418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40614b0 msgr2=0x7fe1a4061940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40614b0 0x7fe1a4061940 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7fe19c002f70 tx=0x7fe19c006eb0).stop 2022-07-20T02:36:05.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- 172.21.15.26:0/1903060418 shutdown_connections 2022-07-20T02:36:05.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40614b0 0x7fe1a4061940 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/1903060418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40adf90 0x7fe1a40b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- 172.21.15.26:0/1903060418 >> 172.21.15.26:0/1903060418 conn(0x7fe1a40b66e0 msgr2=0x7fe1a40b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:05.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- 172.21.15.26:0/1903060418 shutdown_connections 2022-07-20T02:36:05.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.818+0000 7fe1ab13f700 1 -- 172.21.15.26:0/1903060418 wait complete. 2022-07-20T02:36:05.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 Processor -- start 2022-07-20T02:36:05.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 -- start start 2022-07-20T02:36:05.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40614b0 0x7fe1a4152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a40b14b0 0x7fe1a40ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40a7d30 con 0x7fe1a40adf90 2022-07-20T02:36:05.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40a54d0 con 0x7fe1a40b14b0 2022-07-20T02:36:05.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1ab13f700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40a5770 con 0x7fe1a40614b0 2022-07-20T02:36:05.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a8edb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40614b0 0x7fe1a4152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44674/0 (socket says 172.21.15.26:44674) 2022-07-20T02:36:05.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 -- 172.21.15.26:0/2073233934 learned_addr learned my addr 172.21.15.26:0/2073233934 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:05.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a96dc700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a40b14b0 0x7fe1a40ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 -- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40614b0 msgr2=0x7fe1a4152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40614b0 0x7fe1a4152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 -- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a40b14b0 msgr2=0x7fe1a40ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:05.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a40b14b0 0x7fe1a40ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:05.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.822+0000 7fe1a3fff700 1 -- 172.21.15.26:0/2073233934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c00b1e0 con 0x7fe1a40adf90 2022-07-20T02:36:05.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a3fff700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fe194001360 tx=0x7fe194007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:05.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe194010510 con 0x7fe1a40adf90 2022-07-20T02:36:05.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe194010670 con 0x7fe1a40adf90 2022-07-20T02:36:05.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1940107e0 con 0x7fe1a40adf90 2022-07-20T02:36:05.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a40a5a10 con 0x7fe1a40adf90 2022-07-20T02:36:05.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a40a7640 con 0x7fe1a40adf90 2022-07-20T02:36:05.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a40a5ba0 con 0x7fe1a40adf90 2022-07-20T02:36:05.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe194010070 con 0x7fe1a40adf90 2022-07-20T02:36:05.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 0x7fe188055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:05.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe19406bdf0 con 0x7fe1a40adf90 2022-07-20T02:36:05.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.826+0000 7fe1a8edb700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 0x7fe188055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:05.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.830+0000 7fe1a8edb700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 0x7fe188055f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fe18c00b000 tx=0x7fe18c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:05.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.834+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe194042bb0 con 0x7fe1a40adf90 2022-07-20T02:36:05.998 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.994+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1a40574e0 con 0x7fe1a40adf90 2022-07-20T02:36:05.999 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.994+0000 7fe1a1ffb700 1 -- 172.21.15.26:0/2073233934 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2498 (secure 0 0 0) 0x7fe1940461d0 con 0x7fe1a40adf90 2022-07-20T02:36:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:36:06.000 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:35:46.155980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.26:6837/1434658358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6835","nonce":1434658358},{"type":"v1","addr":"172.21.15.26:6837","nonce":1434658358}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:36:06.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 msgr2=0x7fe188055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.010 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 0x7fe188055f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fe18c00b000 tx=0x7fe18c006df0).stop 2022-07-20T02:36:06.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 msgr2=0x7fe1a4143a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.011 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fe194001360 tx=0x7fe194007ed0).stop 2022-07-20T02:36:06.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 shutdown_connections 2022-07-20T02:36:06.012 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe1a40adf90 0x7fe1a4143a80 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fe188053a70 0x7fe188055f30 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1a40b14b0 0x7fe1a40ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.013 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 --2- 172.21.15.26:0/2073233934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1a40614b0 0x7fe1a4152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 >> 172.21.15.26:0/2073233934 conn(0x7fe1a40b66e0 msgr2=0x7fe1a40b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:06.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 shutdown_connections 2022-07-20T02:36:06.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:05.998+0000 7fe1ab13f700 1 -- 172.21.15.26:0/2073233934 wait complete. 2022-07-20T02:36:06.016 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 89 2022-07-20T02:36:06.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T02:36:06.179 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 Processor -- start 2022-07-20T02:36:06.180 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 -- start start 2022-07-20T02:36:06.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffafc0b5ef0 0x7ffafc1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.181 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffafc0b62d0 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffafc14aac0 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc14ad60 con 0x7ffafc0a8c40 2022-07-20T02:36:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafb7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffafc0a7fc0 0x7ffafc0aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43958/0 (socket says 172.21.15.26:43958) 2022-07-20T02:36:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafb7fe700 1 -- 172.21.15.26:0/985971315 learned_addr learned my addr 172.21.15.26:0/985971315 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:36:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 --2- 172.21.15.26:0/985971315 >> v2:172.21.15.120:3300/0 conn(0x7ffafc0b5ef0 0x7ffafc1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb00b48700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342668664 0 0) 0x7ffafc0b62d0 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.228 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae0003260 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.247 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 -- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 msgr2=0x7ffafc0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:06.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 -- 172.21.15.26:0/985971315 >> v1:172.21.15.26:6789/0 conn(0x7ffafc0a7fc0 legacy=0x7ffafc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.248 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 385493936 0 0) 0x7ffae0003260 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 -- 172.21.15.26:0/985971315 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc14b000 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafbfff700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc1474d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7ffaf4001b80 tx=0x7ffae4006df0).ready entity=mon.1 client_cookie=780452b4e7948c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffaf40019e0 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae4000e90 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.249 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffafaffd700 1 -- 172.21.15.26:0/985971315 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae4012460 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 -- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 msgr2=0x7ffafc1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.134+0000 7ffb01349700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc1474d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7ffaf4001b80 tx=0x7ffae4006df0).stop 2022-07-20T02:36:06.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- 172.21.15.26:0/985971315 shutdown_connections 2022-07-20T02:36:06.250 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 --2- 172.21.15.26:0/985971315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc1474d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- 172.21.15.26:0/985971315 >> 172.21.15.26:0/985971315 conn(0x7ffafc060310 msgr2=0x7ffafc0b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:06.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- 172.21.15.26:0/985971315 shutdown_connections 2022-07-20T02:36:06.251 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- 172.21.15.26:0/985971315 wait complete. 2022-07-20T02:36:06.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 Processor -- start 2022-07-20T02:36:06.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- start start 2022-07-20T02:36:06.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.252 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc0ab670 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc0abda0 con 0x7ffafc0b5ef0 2022-07-20T02:36:06.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb01349700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc0ab860 con 0x7ffafc0a8c40 2022-07-20T02:36:06.253 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44684/0 (socket says 172.21.15.26:44684) 2022-07-20T02:36:06.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 -- 172.21.15.26:0/1887250789 learned_addr learned my addr 172.21.15.26:0/1887250789 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:06.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafb7fe700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 -- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 msgr2=0x7ffafc15a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc15a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 -- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 msgr2=0x7ffafc15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb00b48700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffafbfff700 1 -- 172.21.15.26:0/1887250789 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae400b180 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.138+0000 7ffb00b48700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T02:36:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffafbfff700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7ffaf4001b80 tx=0x7ffae4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae400da20 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae400db80 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc0abb00 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae401e400 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc0655f0 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc159e20 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffae4025030 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.142+0000 7ffaf97fa700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 0x7ffad4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.146+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ffae406ce30 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.146+0000 7ffafb7fe700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 0x7ffad4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.146+0000 7ffafb7fe700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 0x7ffad4055fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7ffaf400b000 tx=0x7ffaf4009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.146+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffafc159e20 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.310+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffafc159e20 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.620 INFO:tasks.ceph.mon.a.smithi026.stderr:2022-07-20T02:36:06.618+0000 7f559233c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T02:36:06.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.634+0000 7ffaf97fa700 1 -- 172.21.15.26:0/1887250789 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffafc159e20 con 0x7ffafc0a7fc0 2022-07-20T02:36:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 msgr2=0x7ffad4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 0x7ffad4055fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7ffaf400b000 tx=0x7ffaf4009040).stop 2022-07-20T02:36:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 msgr2=0x7ffafc149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7ffaf4001b80 tx=0x7ffae4014000).stop 2022-07-20T02:36:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 shutdown_connections 2022-07-20T02:36:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ffad4053b00 0x7ffad4055fc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffafc0b5ef0 0x7ffafc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffafc0a8c40 0x7ffafc15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 --2- 172.21.15.26:0/1887250789 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffafc0a7fc0 0x7ffafc149a60 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 >> 172.21.15.26:0/1887250789 conn(0x7ffafc060310 msgr2=0x7ffafc067950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 shutdown_connections 2022-07-20T02:36:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.638+0000 7ffb01349700 1 -- 172.21.15.26:0/1887250789 wait complete. 2022-07-20T02:36:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T02:36:06.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T02:36:06.678 INFO:tasks.ceph.mds.c.smithi026.stdout:starting mds.c at 2022-07-20T02:36:06.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.786+0000 7f90e26b9700 1 Processor -- start 2022-07-20T02:36:06.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.786+0000 7f90e26b9700 1 -- start start 2022-07-20T02:36:06.868 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90dc064450 0x7f90dc064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f90dc0b3380 0x7f90dc065440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc0b85f0 con 0x7f90dc0b3380 2022-07-20T02:36:06.869 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc064880 con 0x7f90dc064450 2022-07-20T02:36:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0b80b0 con 0x7f90dc0670c0 2022-07-20T02:36:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f90dc0b3380 0x7f90dc065440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e16b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f90dc0b3380 0x7f90dc065440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44706/0 (socket says 172.21.15.26:44706) 2022-07-20T02:36:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 -- 172.21.15.26:0/321608907 learned_addr learned my addr 172.21.15.26:0/321608907 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e0eb6700 1 --2- 172.21.15.26:0/321608907 >> v2:172.21.15.120:3300/0 conn(0x7f90dc064450 0x7f90dc064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 -- 172.21.15.26:0/321608907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 msgr2=0x7f90dc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:06.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 -- 172.21.15.26:0/321608907 >> v2:172.21.15.120:3300/0 conn(0x7f90dc064450 msgr2=0x7f90dc064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:06.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/321608907 >> v2:172.21.15.120:3300/0 conn(0x7f90dc064450 0x7f90dc064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 -- 172.21.15.26:0/321608907 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc0b8350 con 0x7f90dc0b3380 2022-07-20T02:36:06.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e16b7700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:36:06.901 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc0b3380 0x7f90dc065440 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f90d40032f0 tx=0x7f90d4009290).ready entity=mon.0 client_cookie=54626a02c0b38b06 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90cbfff700 1 -- 172.21.15.26:0/321608907 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d400b680 con 0x7f90dc0b3380 2022-07-20T02:36:06.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90cbfff700 1 -- 172.21.15.26:0/321608907 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d400b7e0 con 0x7f90dc0b3380 2022-07-20T02:36:06.903 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90cbfff700 1 -- 172.21.15.26:0/321608907 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d400b950 con 0x7f90dc0b3380 2022-07-20T02:36:06.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 -- 172.21.15.26:0/321608907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc0b3380 msgr2=0x7f90dc065440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.904 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.790+0000 7f90e26b9700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc0b3380 0x7f90dc065440 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f90d40032f0 tx=0x7f90d4009290).stop 2022-07-20T02:36:06.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- 172.21.15.26:0/321608907 shutdown_connections 2022-07-20T02:36:06.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc0b3380 0x7f90dc065440 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- 172.21.15.26:0/321608907 >> v2:172.21.15.120:3300/0 conn(0x7f90dc064450 0x7f90dc064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- 172.21.15.26:0/321608907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- 172.21.15.26:0/321608907 >> 172.21.15.26:0/321608907 conn(0x7f90dc05f840 msgr2=0x7f90dc05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:06.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- 172.21.15.26:0/321608907 shutdown_connections 2022-07-20T02:36:06.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- 172.21.15.26:0/321608907 wait complete. 2022-07-20T02:36:06.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 Processor -- start 2022-07-20T02:36:06.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- start start 2022-07-20T02:36:06.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc1400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 0x7f90dc0afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc13ed70 con 0x7f90dc13f610 2022-07-20T02:36:06.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0af650 con 0x7f90dc1412b0 2022-07-20T02:36:06.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e26b9700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc13f3a0 con 0x7f90dc0670c0 2022-07-20T02:36:06.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e0eb6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e0eb6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44712/0 (socket says 172.21.15.26:44712) 2022-07-20T02:36:06.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e16b7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc1400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e0eb6700 1 -- 172.21.15.26:0/3397028959 learned_addr learned my addr 172.21.15.26:0/3397028959 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.794+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 0x7f90dc0afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 -- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 msgr2=0x7f90dc1400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc1400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 -- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 msgr2=0x7f90dc143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 -- 172.21.15.26:0/3397028959 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d800b180 con 0x7f90dc1412b0 2022-07-20T02:36:06.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e16b7700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc1400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:36:06.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e0eb6700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:36:06.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e1eb8700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 0x7f90dc0afab0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f90d40032f0 tx=0x7f90d4017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d400b680 con 0x7f90dc1412b0 2022-07-20T02:36:06.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d400bea0 con 0x7f90dc1412b0 2022-07-20T02:36:06.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4009530 con 0x7f90dc1412b0 2022-07-20T02:36:06.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d401b440 con 0x7f90dc1412b0 2022-07-20T02:36:06.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc13efa0 con 0x7f90dc1412b0 2022-07-20T02:36:06.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc064ab0 con 0x7f90dc1412b0 2022-07-20T02:36:06.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90d4021030 con 0x7f90dc1412b0 2022-07-20T02:36:06.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 0x7f90bc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:06.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.798+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f90d406c850 con 0x7f90dc1412b0 2022-07-20T02:36:06.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.802+0000 7f90e16b7700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 0x7f90bc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:06.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.802+0000 7f90e16b7700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 0x7f90bc055e40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f90d800b000 tx=0x7f90d8006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:06.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.810+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90d4043670 con 0x7f90dc1412b0 2022-07-20T02:36:06.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.978+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90dc0af790 con 0x7f90dc1412b0 2022-07-20T02:36:06.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.978+0000 7f90ca7fc700 1 -- 172.21.15.26:0/3397028959 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f90d4046c90 con 0x7f90dc1412b0 2022-07-20T02:36:06.987 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T02:36:06.987 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":90,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2498908683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2498908683},{"type":"v1","addr":"172.21.15.120:6833","nonce":2498908683}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":84},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.26:6836/2661342408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":2661342408},{"type":"v1","addr":"172.21.15.26:6836","nonce":2661342408}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T02:35:44.142374+0000","modified":"2022-07-20T02:36:06.621625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":163,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T02:36:06.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 msgr2=0x7f90bc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 0x7f90bc055e40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f90d800b000 tx=0x7f90d8006df0).stop 2022-07-20T02:36:06.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 msgr2=0x7f90dc0afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:06.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 0x7f90dc0afab0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f90d40032f0 tx=0x7f90d4017000).stop 2022-07-20T02:36:06.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 shutdown_connections 2022-07-20T02:36:06.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90dc13f610 0x7f90dc143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f90bc053980 0x7f90bc055e40 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90dc1412b0 0x7f90dc0afab0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 --2- 172.21.15.26:0/3397028959 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90dc0670c0 0x7f90dc1400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:06.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 >> 172.21.15.26:0/3397028959 conn(0x7f90dc05f840 msgr2=0x7f90dc145560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:06.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 shutdown_connections 2022-07-20T02:36:06.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:06.982+0000 7f90e26b9700 1 -- 172.21.15.26:0/3397028959 wait complete. 2022-07-20T02:36:06.992 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 90 2022-07-20T02:36:07.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T02:36:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 Processor -- start 2022-07-20T02:36:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 -- start start 2022-07-20T02:36:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5880b3260 0x7fb588060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb5880a95a0 0x7fb5880676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 0x7fb5880b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880b3640 con 0x7fb5880b3260 2022-07-20T02:36:07.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880615c0 con 0x7fb5880a95a0 2022-07-20T02:36:07.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58eefb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb588061860 con 0x7fb5880b1720 2022-07-20T02:36:07.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58def9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5880b3260 0x7fb588060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58def9700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5880b3260 0x7fb588060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44718/0 (socket says 172.21.15.26:44718) 2022-07-20T02:36:07.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58def9700 1 -- 172.21.15.26:0/2303001052 learned_addr learned my addr 172.21.15.26:0/2303001052 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:07.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58e6fa700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 0x7fb5880b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.110+0000 7fb58d6f8700 1 --2- 172.21.15.26:0/2303001052 >> v2:172.21.15.120:3300/0 conn(0x7fb5880a95a0 0x7fb5880676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 -- 172.21.15.26:0/2303001052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 msgr2=0x7fb5880b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:07.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 0x7fb5880b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 -- 172.21.15.26:0/2303001052 >> v2:172.21.15.120:3300/0 conn(0x7fb5880a95a0 msgr2=0x7fb5880676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:07.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 --2- 172.21.15.26:0/2303001052 >> v2:172.21.15.120:3300/0 conn(0x7fb5880a95a0 0x7fb5880676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 -- 172.21.15.26:0/2303001052 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588061b00 con 0x7fb5880b3260 2022-07-20T02:36:07.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58e6fa700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 0x7fb5880b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:36:07.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58def9700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb588060950 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fb584002f70 tx=0x7fb584006eb0).ready entity=mon.0 client_cookie=8a2ac8fcd539a7c1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58cef7700 1 -- 172.21.15.26:0/2303001052 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58400f970 con 0x7fb5880b3260 2022-07-20T02:36:07.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58cef7700 1 -- 172.21.15.26:0/2303001052 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584000e90 con 0x7fb5880b3260 2022-07-20T02:36:07.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58cef7700 1 -- 172.21.15.26:0/2303001052 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58400fd00 con 0x7fb5880b3260 2022-07-20T02:36:07.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- 172.21.15.26:0/2303001052 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 msgr2=0x7fb588060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.148 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb588060950 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fb584002f70 tx=0x7fb584006eb0).stop 2022-07-20T02:36:07.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- 172.21.15.26:0/2303001052 shutdown_connections 2022-07-20T02:36:07.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880b1720 0x7fb5880b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 --2- 172.21.15.26:0/2303001052 >> v2:172.21.15.120:3300/0 conn(0x7fb5880a95a0 0x7fb5880676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.149 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 --2- 172.21.15.26:0/2303001052 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb588060950 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- 172.21.15.26:0/2303001052 >> 172.21.15.26:0/2303001052 conn(0x7fb5880b67f0 msgr2=0x7fb5880b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:07.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- 172.21.15.26:0/2303001052 shutdown_connections 2022-07-20T02:36:07.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- 172.21.15.26:0/2303001052 wait complete. 2022-07-20T02:36:07.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 Processor -- start 2022-07-20T02:36:07.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.114+0000 7fb58eefb700 1 -- start start 2022-07-20T02:36:07.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880a95a0 0x7fb588142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5880b1720 0x7fb5880adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880ad8b0 con 0x7fb5880b3260 2022-07-20T02:36:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb588141960 con 0x7fb5880b1720 2022-07-20T02:36:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880adae0 con 0x7fb5880a95a0 2022-07-20T02:36:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44726/0 (socket says 172.21.15.26:44726) 2022-07-20T02:36:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 -- 172.21.15.26:0/3532920924 learned_addr learned my addr 172.21.15.26:0/3532920924 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58d6f8700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5880b1720 0x7fb5880adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58def9700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880a95a0 0x7fb588142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 -- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880a95a0 msgr2=0x7fb588142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880a95a0 0x7fb588142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 -- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5880b1720 msgr2=0x7fb5880adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5880b1720 0x7fb5880adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 -- 172.21.15.26:0/3532920924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58400b1e0 con 0x7fb5880b3260 2022-07-20T02:36:07.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58e6fa700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7fb580006cc0 tx=0x7fb580008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb580011780 con 0x7fb5880b3260 2022-07-20T02:36:07.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5800118e0 con 0x7fb5880b3260 2022-07-20T02:36:07.157 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5880afd30 con 0x7fb5880b3260 2022-07-20T02:36:07.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb580011c40 con 0x7fb5880b3260 2022-07-20T02:36:07.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb588141c00 con 0x7fb5880b3260 2022-07-20T02:36:07.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb588067210 con 0x7fb5880b3260 2022-07-20T02:36:07.158 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.118+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb58001c460 con 0x7fb5880b3260 2022-07-20T02:36:07.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.122+0000 7fb576ffd700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 0x7fb568055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.122+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb58006c610 con 0x7fb5880b3260 2022-07-20T02:36:07.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.122+0000 7fb58def9700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 0x7fb568055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.159 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.122+0000 7fb58def9700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 0x7fb568055f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fb58401c000 tx=0x7fb584017000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.130+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb580043320 con 0x7fb5880b3260 2022-07-20T02:36:07.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.290+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb5880574e0 con 0x7fb5880b3260 2022-07-20T02:36:07.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.642+0000 7fb576ffd700 1 -- 172.21.15.26:0/3532920924 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb580046940 con 0x7fb5880b3260 2022-07-20T02:36:07.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 msgr2=0x7fb568055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 0x7fb568055f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fb58401c000 tx=0x7fb584017000).stop 2022-07-20T02:36:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 msgr2=0x7fb5880ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7fb580006cc0 tx=0x7fb580008040).stop 2022-07-20T02:36:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 shutdown_connections 2022-07-20T02:36:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5880b3260 0x7fb5880ae470 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb568053a70 0x7fb568055f30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5880b1720 0x7fb5880adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 --2- 172.21.15.26:0/3532920924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5880a95a0 0x7fb588142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 >> 172.21.15.26:0/3532920924 conn(0x7fb5880b67f0 msgr2=0x7fb5880b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 shutdown_connections 2022-07-20T02:36:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.646+0000 7fb58eefb700 1 -- 172.21.15.26:0/3532920924 wait complete. 2022-07-20T02:36:07.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T02:36:07.795 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 Processor -- start 2022-07-20T02:36:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 -- start start 2022-07-20T02:36:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 0x7f66a80676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66a80b1720 0x7f66a80b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a80b3640 con 0x7f66a80b1720 2022-07-20T02:36:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a80615c0 con 0x7f66a80a95a0 2022-07-20T02:36:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66ace23700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a8061860 con 0x7f66a80b3260 2022-07-20T02:36:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66a80b1720 0x7f66a80b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f66a80b1720 0x7f66a80b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44736/0 (socket says 172.21.15.26:44736) 2022-07-20T02:36:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 -- 172.21.15.26:0/2561941548 learned_addr learned my addr 172.21.15.26:0/2561941548 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a77fe700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a6ffd700 1 --2- 172.21.15.26:0/2561941548 >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 0x7f66a80676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 -- 172.21.15.26:0/2561941548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 msgr2=0x7f66a8060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 -- 172.21.15.26:0/2561941548 >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 msgr2=0x7f66a80676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T02:36:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 --2- 172.21.15.26:0/2561941548 >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 0x7f66a80676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 -- 172.21.15.26:0/2561941548 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8061b00 con 0x7f66a80b1720 2022-07-20T02:36:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a6ffd700 1 --2- 172.21.15.26:0/2561941548 >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 0x7f66a80676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:36:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a7fff700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80b1720 0x7f66a80b1b40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f66a00032f0 tx=0x7f66a0009290).ready entity=mon.0 client_cookie=bb417cbd23c5d48c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a67fc700 1 -- 172.21.15.26:0/2561941548 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a000ba60 con 0x7f66a80b1720 2022-07-20T02:36:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a67fc700 1 -- 172.21.15.26:0/2561941548 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a000bbc0 con 0x7f66a80b1720 2022-07-20T02:36:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.794+0000 7f66a67fc700 1 -- 172.21.15.26:0/2561941548 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a0010470 con 0x7f66a80b1720 2022-07-20T02:36:07.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- 172.21.15.26:0/2561941548 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80b1720 msgr2=0x7f66a80b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80b1720 0x7f66a80b1b40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f66a00032f0 tx=0x7f66a0009290).stop 2022-07-20T02:36:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- 172.21.15.26:0/2561941548 shutdown_connections 2022-07-20T02:36:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80b1720 0x7f66a80b1b40 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- 172.21.15.26:0/2561941548 >> v2:172.21.15.120:3300/0 conn(0x7f66a80a95a0 0x7f66a80676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- 172.21.15.26:0/2561941548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- 172.21.15.26:0/2561941548 >> 172.21.15.26:0/2561941548 conn(0x7f66a80b67f0 msgr2=0x7f66a80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- 172.21.15.26:0/2561941548 shutdown_connections 2022-07-20T02:36:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- 172.21.15.26:0/2561941548 wait complete. 2022-07-20T02:36:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 Processor -- start 2022-07-20T02:36:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- start start 2022-07-20T02:36:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 0x7f66a813fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80aa8a0 con 0x7f66a80a95a0 2022-07-20T02:36:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a813fab0 con 0x7f66a80b1720 2022-07-20T02:36:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.798+0000 7f66ace23700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a8140750 con 0x7f66a80b3260 2022-07-20T02:36:07.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44738/0 (socket says 172.21.15.26:44738) 2022-07-20T02:36:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 0x7f66a813fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a77fe700 1 -- 172.21.15.26:0/1190921890 learned_addr learned my addr 172.21.15.26:0/1190921890 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a7fff700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 -- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 msgr2=0x7f66a8140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.844 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 -- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 msgr2=0x7f66a80b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:07.845 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:07.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 -- 172.21.15.26:0/1190921890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0009530 con 0x7f66a80b1720 2022-07-20T02:36:07.846 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a77fe700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T02:36:07.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a6ffd700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 0x7f66a813fd70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f669c00a000 tx=0x7f669c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f669c012610 con 0x7f66a80b1720 2022-07-20T02:36:07.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c012770 con 0x7f66a80b1720 2022-07-20T02:36:07.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a813f810 con 0x7f66a80b1720 2022-07-20T02:36:07.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f669c012ad0 con 0x7f66a80b1720 2022-07-20T02:36:07.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a80abb60 con 0x7f66a80b1720 2022-07-20T02:36:07.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a813f670 con 0x7f66a80b1720 2022-07-20T02:36:07.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.802+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f669c012c30 con 0x7f66a80b1720 2022-07-20T02:36:07.870 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.806+0000 7f66a4ff9700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 0x7f6688055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:07.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.806+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f669c06c120 con 0x7f66a80b1720 2022-07-20T02:36:07.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.806+0000 7f66a77fe700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 0x7f6688055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:07.871 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.806+0000 7f66a77fe700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 0x7f6688055ea0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f669800b000 tx=0x7f6698006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:07.872 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.814+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66a813f670 con 0x7f66a80b1720 2022-07-20T02:36:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:07.946+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66a813f670 con 0x7f66a80b1720 2022-07-20T02:36:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.650+0000 7f66a4ff9700 1 -- 172.21.15.26:0/1190921890 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v164) v1 ==== 185+0+0 (secure 0 0 0) 0x7f66a813f670 con 0x7f66a80b1720 2022-07-20T02:36:08.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 msgr2=0x7f6688055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:08.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 0x7f6688055ea0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f669800b000 tx=0x7f6698006df0).stop 2022-07-20T02:36:08.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 msgr2=0x7f66a813fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:08.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 0x7f66a813fd70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f669c00a000 tx=0x7f669c008040).stop 2022-07-20T02:36:08.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 shutdown_connections 2022-07-20T02:36:08.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f66880539e0 0x7f6688055ea0 secure :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0x7f669800b000 tx=0x7f6698006df0).stop 2022-07-20T02:36:08.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a80b3260 0x7f66a8140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:08.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a80b1720 0x7f66a813fd70 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:08.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 --2- 172.21.15.26:0/1190921890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f66a80a95a0 0x7f66a80b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:08.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 >> 172.21.15.26:0/1190921890 conn(0x7f66a80b67f0 msgr2=0x7f66a809a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:08.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 shutdown_connections 2022-07-20T02:36:08.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.654+0000 7f66ace23700 1 -- 172.21.15.26:0/1190921890 wait complete. 2022-07-20T02:36:08.661 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2022-07-20T02:36:08.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T02:36:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 Processor -- start 2022-07-20T02:36:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 -- start start 2022-07-20T02:36:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4b2c0b14b0 0x7f4b2c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:08.894 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0ae370 con 0x7f4b2c0adf90 2022-07-20T02:36:08.900 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0b28b0 con 0x7f4b2c0614b0 2022-07-20T02:36:08.902 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.846+0000 7f4b32dd3700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2c0b0610 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b30b6f700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4b2c0adf90 0x7f4b2c060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44026/0 (socket says 172.21.15.26:44026) 2022-07-20T02:36:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b30b6f700 1 -- 172.21.15.26:0/2185501797 learned_addr learned my addr 172.21.15.26:0/2185501797 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T02:36:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2bfff700 1 --2- 172.21.15.26:0/2185501797 >> v2:172.21.15.120:3300/0 conn(0x7f4b2c0b14b0 0x7f4b2c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927811805 0 0) 0x7f4b2c0b28b0 con 0x7f4b2c0614b0 2022-07-20T02:36:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c003260 con 0x7f4b2c0614b0 2022-07-20T02:36:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744810323 0 0) 0x7f4b2c0ae370 con 0x7f4b2c0adf90 2022-07-20T02:36:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2bfff700 1 -- 172.21.15.26:0/2185501797 >> v1:172.21.15.120:6789/0 conn(0x7f4b2c0614b0 legacy=0x7f4b2c061910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2bfff700 1 -- 172.21.15.26:0/2185501797 >> v1:172.21.15.26:6789/0 conn(0x7f4b2c0adf90 legacy=0x7f4b2c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2bfff700 1 -- 172.21.15.26:0/2185501797 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c0b08b0 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2bfff700 1 --2- 172.21.15.26:0/2185501797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0b2c60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f4b2c0b1900 tx=0x7f4b1c006df0).ready entity=mon.1 client_cookie=ef75a3ccead70805 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b1c00d610 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c000e90 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b2b7fe700 1 -- 172.21.15.26:0/2185501797 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b1c00d960 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 -- 172.21.15.26:0/2185501797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 msgr2=0x7f4b2c0b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:08.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/2185501797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0b2c60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f4b2c0b1900 tx=0x7f4b1c006df0).stop 2022-07-20T02:36:08.960 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 -- 172.21.15.26:0/2185501797 shutdown_connections 2022-07-20T02:36:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/2185501797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0b2c60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 -- 172.21.15.26:0/2185501797 >> 172.21.15.26:0/2185501797 conn(0x7f4b2c0b66e0 msgr2=0x7f4b2c0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 -- 172.21.15.26:0/2185501797 shutdown_connections 2022-07-20T02:36:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.850+0000 7f4b32dd3700 1 -- 172.21.15.26:0/2185501797 wait complete. 2022-07-20T02:36:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 Processor -- start 2022-07-20T02:36:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- start start 2022-07-20T02:36:08.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b2c0614b0 0x7f4b2c09f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:08.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:08.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:08.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c065d50 con 0x7f4b2c0adf90 2022-07-20T02:36:08.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c0a9d80 con 0x7f4b2c0b14b0 2022-07-20T02:36:08.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c0a9fe0 con 0x7f4b2c0614b0 2022-07-20T02:36:08.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:08.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44754/0 (socket says 172.21.15.26:44754) 2022-07-20T02:36:08.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 -- 172.21.15.26:0/1182013747 learned_addr learned my addr 172.21.15.26:0/1182013747 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T02:36:09.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b30b6f700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b2c0614b0 0x7f4b2c09f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:09.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b31370700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:09.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 -- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b2c0614b0 msgr2=0x7f4b2c09f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:09.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b2c0614b0 0x7f4b2c09f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 -- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 msgr2=0x7f4b2c0aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:09.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 -- 172.21.15.26:0/1182013747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c00b180 con 0x7f4b2c0adf90 2022-07-20T02:36:09.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b31370700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T02:36:09.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b2bfff700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f4b2c0b1900 tx=0x7f4b1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:09.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b1c013e10 con 0x7f4b2c0adf90 2022-07-20T02:36:09.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c00d290 con 0x7f4b2c0adf90 2022-07-20T02:36:09.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c0aa280 con 0x7f4b2c0adf90 2022-07-20T02:36:09.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b1c01e5f0 con 0x7f4b2c0adf90 2022-07-20T02:36:09.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.854+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c0656b0 con 0x7f4b2c0adf90 2022-07-20T02:36:09.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2c0574e0 con 0x7f4b2c0adf90 2022-07-20T02:36:09.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b1c025030 con 0x7f4b2c0adf90 2022-07-20T02:36:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b29ffb700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 0x7f4b10055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T02:36:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f4b1c06cbe0 con 0x7f4b2c0adf90 2022-07-20T02:36:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b30b6f700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 0x7f4b10055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T02:36:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.858+0000 7f4b30b6f700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 0x7f4b10055fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f4b1400d000 tx=0x7f4b14009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T02:36:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.862+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b1c043b00 con 0x7f4b2c0adf90 2022-07-20T02:36:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:08.990+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4b2c0683d0 con 0x7f4b2c0adf90 2022-07-20T02:36:09.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.658+0000 7f4b29ffb700 1 -- 172.21.15.26:0/1182013747 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v165) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4b1c023020 con 0x7f4b2c0adf90 2022-07-20T02:36:09.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 msgr2=0x7f4b10055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:09.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 0x7f4b10055fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f4b1400d000 tx=0x7f4b14009040).stop 2022-07-20T02:36:09.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 msgr2=0x7f4b2c0aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T02:36:09.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f4b2c0b1900 tx=0x7f4b1c014000).stop 2022-07-20T02:36:09.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 shutdown_connections 2022-07-20T02:36:09.686 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b2c0adf90 0x7f4b2c0aaab0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4b10053b00 0x7f4b10055fc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b2c0b14b0 0x7f4b2c0aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 --2- 172.21.15.26:0/1182013747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b2c0614b0 0x7f4b2c09f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T02:36:09.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 >> 172.21.15.26:0/1182013747 conn(0x7f4b2c0b66e0 msgr2=0x7f4b2c1447c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T02:36:09.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 shutdown_connections 2022-07-20T02:36:09.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T02:36:09.662+0000 7f4b32dd3700 1 -- 172.21.15.26:0/1182013747 wait complete. 2022-07-20T02:36:09.696 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2022-07-20T02:36:09.711 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-20T02:36:09.713 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-20T02:36:18.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:18.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:18.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:18.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:39.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:36:39.840 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:36:39.840 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:36:48.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:48.975 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:49.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:36:49.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:09.733 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T02:37:09.753 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T02:37:19.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:19.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:19.086 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:19.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:49.189 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:49.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:49.222 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:37:49.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:19.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:19.311 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:19.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:19.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:49.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:49.406 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:49.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:38:49.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:19.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:19.508 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:19.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:19.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:49.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:49.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:49.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:39:49.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:19.789 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:19.794 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:19.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:19.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:49.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:49.868 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:49.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:40:49.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:20.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:20.013 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:20.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:20.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:50.111 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:50.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:50.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:41:50.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:20.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:20.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:20.197 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:20.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:50.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:50.293 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:50.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:42:50.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:20.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:20.410 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:20.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:20.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:50.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:50.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:50.478 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:43:50.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:20.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:20.555 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:20.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:20.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:50.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:50.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:50.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:44:50.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:20.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:20.712 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:20.715 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:20.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:50.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:50.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:50.781 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:45:50.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:20.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:20.846 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:20.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:20.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:50.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:50.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:50.924 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:46:50.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:21.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:21.017 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:21.020 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:21.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:51.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:51.095 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:51.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:47:51.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:21.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:21.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:21.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:21.244 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:51.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:51.307 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:51.311 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:48:51.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:21.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:21.398 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:21.403 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:21.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:51.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:51.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:51.478 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:49:51.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:21.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:21.556 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:21.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:21.561 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:51.639 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:51.649 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:51.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:50:51.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:21.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:21.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:21.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:21.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:39.855 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T02:51:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:51:39.860 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T02:51:39.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:51:39.862 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T02:51:39.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:51:39.909 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:51:39.934 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T02:51:39.962 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T02:51:39.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T02:51:39.964 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T02:51:39.965 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T02:51:51.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:51.807 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:51.811 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:51:51.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:21.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:21.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:21.891 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:21.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:51.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:51.992 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:52.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:52:52.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:22.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:22.109 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:22.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:22.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:52.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:52.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:52.384 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:53:52.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:22.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:22.466 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:22.469 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:22.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:52.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:52.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:52.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:54:52.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:22.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:22.666 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:22.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:22.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:52.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:52.812 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:52.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:55:52.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:22.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:22.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:22.959 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:22.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:53.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:53.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:53.024 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:56:53.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:23.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:23.095 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:23.102 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:23.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:53.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:53.181 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:53.184 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:57:53.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:23.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:23.301 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:23.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:23.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:53.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:53.381 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:53.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:58:53.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:23.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:23.508 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:23.511 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:23.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:53.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:53.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:53.587 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T02:59:53.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:23.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:23.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:23.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:23.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:53.737 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:53.742 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:53.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:00:53.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:23.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:23.856 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:23.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:23.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:53.973 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:53.979 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:53.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:01:53.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:24.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:24.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:24.102 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:24.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:54.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:54.172 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:54.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:02:54.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:24.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:24.257 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:24.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:24.267 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:54.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:54.417 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:54.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:03:54.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:24.498 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:24.502 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:24.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:24.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:54.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:54.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:54.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:04:54.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:24.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:24.859 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:24.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:24.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:55.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:55.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:55.095 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:05:55.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:25.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:25.174 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:25.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:25.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:40.007 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:06:40.009 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T03:06:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:40.033 INFO:tasks.cephfs_test_runner:test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) ... ERROR 2022-07-20T03:06:40.052 INFO:tasks.cephfs_test_runner:ERROR 2022-07-20T03:06:40.053 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.057 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:06:40.058 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:40.059 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:06:40.059 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.060 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:06:40.060 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:06:40.061 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:06:40.062 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:06:40.113 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:06:40.114 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.115 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:06:40.116 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.116 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.117 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-20T03:06:40.119 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-20T03:06:40.120 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-20T03:06:40.120 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-20T03:06:40.121 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:06:40.122 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:06:40.122 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:06:40.123 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:06:40.124 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:06:40.124 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:06:40.125 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:06:40.125 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:06:40.126 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:06:40.127 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:06:40.127 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:06:40.128 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:06:40.129 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:06:40.129 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:06:40.132 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:06:40.132 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:06:40.133 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:06:40.133 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:06:40.134 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:06:40.134 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:06:40.135 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:06:40.144 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:06:40.145 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.146 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:06:40.163 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:40.164 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:06:40.166 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.166 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:06:40.167 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:06:40.167 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:06:40.168 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:06:40.169 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:06:40.169 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.170 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:06:40.179 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.180 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.181 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-20T03:06:40.181 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-20T03:06:40.183 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-20T03:06:40.183 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-20T03:06:40.184 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-20T03:06:40.184 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-20T03:06:40.185 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:06:40.185 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:06:40.187 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:06:40.187 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:06:40.188 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:06:40.188 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:06:40.189 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:06:40.189 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:06:40.190 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:06:40.190 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:06:40.191 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:06:40.192 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:06:40.193 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:06:40.193 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:06:40.194 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:06:40.195 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:06:40.195 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:06:40.196 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:06:40.197 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:06:40.198 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:06:40.199 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:06:40.208 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:06:40.209 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.226 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:06:40.227 INFO:tasks.cephfs_test_runner:Ran 12 tests in 2439.880s 2022-07-20T03:06:40.227 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.228 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2022-07-20T03:06:40.229 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.229 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:06:40.230 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:40.231 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:06:40.232 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.232 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:06:40.233 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:06:40.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:06:40.234 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:06:40.235 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:06:40.235 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.236 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:06:40.236 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.237 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.237 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-20T03:06:40.249 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-20T03:06:40.250 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-20T03:06:40.250 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-20T03:06:40.263 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:06:40.263 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:06:40.264 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:06:40.265 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:06:40.265 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:06:40.270 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:06:40.271 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:06:40.271 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:06:40.271 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:06:40.272 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:06:40.272 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:06:40.272 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:06:40.273 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:06:40.273 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:06:40.273 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:06:40.273 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:06:40.274 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:06:40.274 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:06:40.274 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:06:40.275 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:06:40.275 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:06:40.275 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:06:40.276 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.276 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:06:40.276 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:40.277 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:06:40.277 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.277 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:06:40.277 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:06:40.278 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:06:40.278 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:06:40.278 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:06:40.279 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.279 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:06:40.279 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.279 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:06:40.280 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-20T03:06:40.280 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-20T03:06:40.280 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-20T03:06:40.281 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-20T03:06:40.281 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-20T03:06:40.281 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-20T03:06:40.281 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:06:40.305 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:06:40.306 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:06:40.306 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:06:40.306 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:06:40.307 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:06:40.307 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:06:40.307 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:06:40.307 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:06:40.308 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:06:40.308 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:06:40.308 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:06:40.309 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:06:40.309 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:06:40.309 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:06:40.309 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:06:40.312 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:06:40.312 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:06:40.312 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:06:40.313 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:06:40.313 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:06:40.313 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:06:40.314 INFO:tasks.cephfs_test_runner: 2022-07-20T03:06:40.314 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:41.004 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3b6deb9b55ca4d3abd7a981e69e018b4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:06:41.010 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2022-07-20T03:06:41.068 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2022-07-20T03:06:41.103 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2022-07-20T03:06:41.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:06:41.106 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:06:41.107 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:06:55.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:55.358 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:55.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:55.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:25.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:25.550 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:25.564 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:25.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:55.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:55.732 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:55.736 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:55.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:25.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:25.872 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:25.983 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:26.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:56.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:56.093 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:56.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:56.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:26.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:26.260 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:26.265 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:26.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:56.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:56.417 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:56.434 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:56.444 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:26.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:26.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:26.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:26.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:56.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:56.747 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:56.785 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:56.944 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:27.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:27.081 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:27.084 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:27.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:57.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:57.197 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:57.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:57.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:27.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:27.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:27.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:27.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:57.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:57.547 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:57.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:57.553 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:27.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:27.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:27.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:27.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:57.739 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:57.752 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:57.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:57.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:27.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:27.897 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:27.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:27.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:57.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:57.970 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:57.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:57.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:28.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:28.060 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:28.065 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:28.080 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:58.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:58.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:58.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:58.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:28.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:28.344 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:28.350 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:28.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:58.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:58.493 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:58.496 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:58.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:28.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:28.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:28.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:28.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:58.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:58.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:58.772 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:58.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:29.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:29.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:29.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:29.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:59.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:59.280 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:59.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:59.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:29.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:29.410 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:29.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:29.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:59.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:59.555 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:59.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:59.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:29.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:29.689 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:29.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:29.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:59.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:59.833 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:59.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:59.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:29.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:29.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:29.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:29.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:21:41.172 INFO:teuthology.orchestra.run.smithi159.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Connection timed out 2022-07-20T03:21:41.175 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T03:21:41.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:21:41.179 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:21:41.180 DEBUG:teuthology.orchestra.run.smithi159:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:22:00.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:00.097 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:00.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:00.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:30.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:30.235 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:30.239 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:30.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:00.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:00.396 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:00.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:00.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:30.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:30.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:30.564 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:30.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:00.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:00.714 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:00.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:00.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:30.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:30.871 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:30.895 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:30.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:01.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:01.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:01.074 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:01.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:31.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:31.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:31.221 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:31.226 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:01.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:01.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:01.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:01.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:31.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:31.573 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:31.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:31.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:01.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:01.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:01.788 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:01.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:31.930 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:31.939 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:31.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:31.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:02.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:02.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:02.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:02.121 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:32.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:32.297 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:32.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:32.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:02.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:02.441 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:02.458 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:02.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:32.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:32.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:32.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:32.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:02.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:02.911 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:02.915 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:02.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:33.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:33.056 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:33.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:33.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:03.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:03.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:03.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:03.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:33.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:33.413 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:33.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:33.424 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:03.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:03.557 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:03.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:03.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:33.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:33.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:33.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:33.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:03.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:03.961 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:03.967 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:03.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:34.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:34.213 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:34.216 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:34.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:04.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:04.365 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:04.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:04.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:34.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:34.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:34.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:34.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:04.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:04.719 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:04.730 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:04.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:34.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:34.892 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:34.896 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:34.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:05.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:05.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:05.105 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:05.131 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:35.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:35.263 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:35.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:35.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:41.193 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:36:41.243 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T03:36:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:36:41.312 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:36:41.344 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-07-20T03:36:41.361 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-07-20T03:36:41.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:36:41.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 Processor -- start 2022-07-20T03:36:41.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- start start 2022-07-20T03:36:41.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f95801484a0 0x7f9580066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9580061050 0x7f9580066f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95800b1bc0 con 0x7f95800b2920 2022-07-20T03:36:41.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958014b040 con 0x7f95801484a0 2022-07-20T03:36:41.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95800a8860 con 0x7f9580061050 2022-07-20T03:36:41.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f95801484a0 0x7f9580066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f95801484a0 0x7f9580066a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44850/0 (socket says 172.21.15.26:44850) 2022-07-20T03:36:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 -- 172.21.15.26:0/685645786 learned_addr learned my addr 172.21.15.26:0/685645786 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9584e06700 1 --2- 172.21.15.26:0/685645786 >> v2:172.21.15.120:3300/0 conn(0x7f9580061050 0x7f9580066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.516 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 -- 172.21.15.26:0/685645786 >> v2:172.21.15.120:3300/0 conn(0x7f9580061050 msgr2=0x7f9580066f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:41.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 --2- 172.21.15.26:0/685645786 >> v2:172.21.15.120:3300/0 conn(0x7f9580061050 0x7f9580066f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.517 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 -- 172.21.15.26:0/685645786 >> v1:172.21.15.120:6789/0 conn(0x7f95800b2920 legacy=0x7f9580147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.519 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 -- 172.21.15.26:0/685645786 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580147700 con 0x7f95801484a0 2022-07-20T03:36:41.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f95777fe700 1 --2- 172.21.15.26:0/685645786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95801484a0 0x7f9580066a80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f9568002f70 tx=0x7f9568006eb0).ready entity=mon.0 client_cookie=97295a545b2fc086 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f957f7fe700 1 -- 172.21.15.26:0/685645786 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f956800d610 con 0x7f95801484a0 2022-07-20T03:36:41.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f957f7fe700 1 -- 172.21.15.26:0/685645786 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9568000e90 con 0x7f95801484a0 2022-07-20T03:36:41.525 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f957f7fe700 1 -- 172.21.15.26:0/685645786 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f956800d9a0 con 0x7f95801484a0 2022-07-20T03:36:41.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- 172.21.15.26:0/685645786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95801484a0 msgr2=0x7f9580066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 --2- 172.21.15.26:0/685645786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95801484a0 0x7f9580066a80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f9568002f70 tx=0x7f9568006eb0).stop 2022-07-20T03:36:41.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- 172.21.15.26:0/685645786 shutdown_connections 2022-07-20T03:36:41.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 --2- 172.21.15.26:0/685645786 >> v2:172.21.15.120:3300/0 conn(0x7f9580061050 0x7f9580066f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 --2- 172.21.15.26:0/685645786 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95801484a0 0x7f9580066a80 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- 172.21.15.26:0/685645786 >> 172.21.15.26:0/685645786 conn(0x7f95800b60a0 msgr2=0x7f95800b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:41.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- 172.21.15.26:0/685645786 shutdown_connections 2022-07-20T03:36:41.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.507+0000 7f9586869700 1 -- 172.21.15.26:0/685645786 wait complete. 2022-07-20T03:36:41.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 Processor -- start 2022-07-20T03:36:41.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- start start 2022-07-20T03:36:41.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9580061050 0x7f958014a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95801484a0 0x7f9580157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95800ab3f0 con 0x7f95800b2920 2022-07-20T03:36:41.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95800ab690 con 0x7f9580061050 2022-07-20T03:36:41.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9580156be0 con 0x7f95801484a0 2022-07-20T03:36:41.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44856/0 (socket says 172.21.15.26:44856) 2022-07-20T03:36:41.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 -- 172.21.15.26:0/567301168 learned_addr learned my addr 172.21.15.26:0/567301168 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:41.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f957ffff700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9580061050 0x7f958014a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9584e06700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95801484a0 0x7f9580157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 -- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95801484a0 msgr2=0x7f9580157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95801484a0 0x7f9580157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 -- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9580061050 msgr2=0x7f958014a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9580061050 0x7f958014a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 -- 172.21.15.26:0/567301168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956800b1e0 con 0x7f95800b2920 2022-07-20T03:36:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f95777fe700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f9568002f70 tx=0x7f9568015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9568011760 con 0x7f95800b2920 2022-07-20T03:36:41.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95680118c0 con 0x7f95800b2920 2022-07-20T03:36:41.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9568011c20 con 0x7f95800b2920 2022-07-20T03:36:41.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9580062db0 con 0x7f95800b2920 2022-07-20T03:36:41.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.511+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9580065dd0 con 0x7f95800b2920 2022-07-20T03:36:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958014ada0 con 0x7f95800b2920 2022-07-20T03:36:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f956801a040 con 0x7f95800b2920 2022-07-20T03:36:41.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f957dffb700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 0x7f9564055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f956806cb40 con 0x7f95800b2920 2022-07-20T03:36:41.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f957ffff700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 0x7f9564055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.515+0000 7f957ffff700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 0x7f9564055e40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f957000d000 tx=0x7f9570007b90).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.523+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9568043b50 con 0x7f95800b2920 2022-07-20T03:36:41.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.643+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95800574e0 con 0x7f95800b2920 2022-07-20T03:36:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f957dffb700 1 -- 172.21.15.26:0/567301168 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+13601 (secure 0 0 0) 0x7f95680470f0 con 0x7f95800b2920 2022-07-20T03:36:41.652 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T03:36:41.652 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":165,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:36:09.646654+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.26:6837/1434658358":"2022-07-21T02:36:06.314800+0000","172.21.15.26:6835/1434658358":"2022-07-21T02:36:06.314800+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:36:41.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 msgr2=0x7f9564055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 0x7f9564055e40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f957000d000 tx=0x7f9570007b90).stop 2022-07-20T03:36:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 msgr2=0x7f9580157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f9568002f70 tx=0x7f9568015000).stop 2022-07-20T03:36:41.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 shutdown_connections 2022-07-20T03:36:41.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f95800b2920 0x7f9580157080 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9564053980 0x7f9564055e40 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95801484a0 0x7f9580157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 --2- 172.21.15.26:0/567301168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9580061050 0x7f958014a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 >> 172.21.15.26:0/567301168 conn(0x7f95800b60a0 msgr2=0x7f9580146dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:41.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.647+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 shutdown_connections 2022-07-20T03:36:41.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.651+0000 7f9586869700 1 -- 172.21.15.26:0/567301168 wait complete. 2022-07-20T03:36:41.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T03:36:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 Processor -- start 2022-07-20T03:36:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 -- start start 2022-07-20T03:36:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5c40b5ef0 0x7fb5c41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb5c40a8c40 0x7fb5c40af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c40b62d0 con 0x7fb5c40a7fc0 2022-07-20T03:36:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c414aac0 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5cb2c4700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c414ad60 con 0x7fb5c40a8c40 2022-07-20T03:36:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5c40b5ef0 0x7fb5c41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fb5c40b5ef0 0x7fb5c41474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44862/0 (socket says 172.21.15.26:44862) 2022-07-20T03:36:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 -- 172.21.15.26:0/4138711825 learned_addr learned my addr 172.21.15.26:0/4138711825 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5caac3700 1 --2- 172.21.15.26:0/4138711825 >> v2:172.21.15.120:3300/0 conn(0x7fb5c40a8c40 0x7fb5c40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5c92c0700 1 -- 172.21.15.26:0/4138711825 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198835519 0 0) 0x7fb5c40b62d0 con 0x7fb5c40a7fc0 2022-07-20T03:36:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5c92c0700 1 -- 172.21.15.26:0/4138711825 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ac003260 con 0x7fb5c40a7fc0 2022-07-20T03:36:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 -- 172.21.15.26:0/4138711825 >> v2:172.21.15.120:3300/0 conn(0x7fb5c40a8c40 msgr2=0x7fb5c40af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 --2- 172.21.15.26:0/4138711825 >> v2:172.21.15.120:3300/0 conn(0x7fb5c40a8c40 0x7fb5c40af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 -- 172.21.15.26:0/4138711825 >> v1:172.21.15.120:6789/0 conn(0x7fb5c40a7fc0 legacy=0x7fb5c40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 -- 172.21.15.26:0/4138711825 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c414b000 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5ca2c2700 1 --2- 172.21.15.26:0/4138711825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c41474d0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7fb5b4002f70 tx=0x7fb5b4006eb0).ready entity=mon.0 client_cookie=f01ae158a9193581 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5c92c0700 1 -- 172.21.15.26:0/4138711825 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b400d610 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5c92c0700 1 -- 172.21.15.26:0/4138711825 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b4000e90 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.779+0000 7fb5c92c0700 1 -- 172.21.15.26:0/4138711825 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b400d9a0 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/4138711825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40b5ef0 msgr2=0x7fb5c41474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/4138711825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c41474d0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7fb5b4002f70 tx=0x7fb5b4006eb0).stop 2022-07-20T03:36:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/4138711825 shutdown_connections 2022-07-20T03:36:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/4138711825 >> v2:172.21.15.120:3300/0 conn(0x7fb5c40a8c40 0x7fb5c40af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/4138711825 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c41474d0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/4138711825 >> 172.21.15.26:0/4138711825 conn(0x7fb5c4060310 msgr2=0x7fb5c40b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/4138711825 shutdown_connections 2022-07-20T03:36:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/4138711825 wait complete. 2022-07-20T03:36:41.798 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 Processor -- start 2022-07-20T03:36:41.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- start start 2022-07-20T03:36:41.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5c40a7fc0 0x7fb5c40aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.799 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c4065040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4159ab0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40ac080 con 0x7fb5c40b5ef0 2022-07-20T03:36:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5cb2c4700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40aac40 con 0x7fb5c40a7fc0 2022-07-20T03:36:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5c9ac1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5c9ac1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44870/0 (socket says 172.21.15.26:44870) 2022-07-20T03:36:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5caac3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c4065040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5ca2c2700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5c40a7fc0 0x7fb5c40aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.783+0000 7fb5c9ac1700 1 -- 172.21.15.26:0/1481463997 learned_addr learned my addr 172.21.15.26:0/1481463997 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 -- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5c40a7fc0 msgr2=0x7fb5c40aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5c40a7fc0 0x7fb5c40aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 -- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5c40b5ef0 msgr2=0x7fb5c4065040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c4065040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 -- 172.21.15.26:0/1481463997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b400b1e0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5c9ac1700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fb5c0001490 tx=0x7fb5c0007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c000e4a0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c000e600 con 0x7fb5c40a8c40 2022-07-20T03:36:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c40aaee0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c000e960 con 0x7fb5c40a8c40 2022-07-20T03:36:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c40630a0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c414b2a0 con 0x7fb5c40a8c40 2022-07-20T03:36:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb5c0019030 con 0x7fb5c40a8c40 2022-07-20T03:36:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 0x7fb5a4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb5c006b670 con 0x7fb5c40a8c40 2022-07-20T03:36:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.787+0000 7fb5ca2c2700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 0x7fb5a4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.791+0000 7fb5ca2c2700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 0x7fb5a4055fc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fb5b401c000 tx=0x7fb5b4015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.795+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5c0042610 con 0x7fb5c40a8c40 2022-07-20T03:36:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.919+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5c40aaac0 con 0x7fb5a4053b00 2022-07-20T03:36:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5b37fe700 1 -- 172.21.15.26:0/1481463997 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67695 (secure 0 0 0) 0x7fb5b40069f0 con 0x7fb5a4053b00 2022-07-20T03:36:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T03:36:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 msgr2=0x7fb5a4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 0x7fb5a4055fc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fb5b401c000 tx=0x7fb5b4015000).stop 2022-07-20T03:36:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 msgr2=0x7fb5c4064b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fb5c0001490 tx=0x7fb5c0007e90).stop 2022-07-20T03:36:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 shutdown_connections 2022-07-20T03:36:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb5c40a8c40 0x7fb5c4064b50 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fb5a4053b00 0x7fb5a4055fc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5c40b5ef0 0x7fb5c4065040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 --2- 172.21.15.26:0/1481463997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5c40a7fc0 0x7fb5c40aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.927+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 >> 172.21.15.26:0/1481463997 conn(0x7fb5c4060310 msgr2=0x7fb5c4067950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:41.940 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.931+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 shutdown_connections 2022-07-20T03:36:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:41.931+0000 7fb5cb2c4700 1 -- 172.21.15.26:0/1481463997 wait complete. 2022-07-20T03:36:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2022-07-20T03:36:41.945 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2299,"stamp":"2022-07-20T03:36:40.579013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8405088,"kb_used_data":16416,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569632,"statfs":{"total":773094113280,"available":764487303168,"internally_reserved":8589934592,"allocated":16809984,"data_stored":8211272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"1.0","version":"14'2","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264259+0000","last_change":"2022-07-20T02:25:25.353187+0000","last_active":"2022-07-20T02:36:09.264259+0000","last_peered":"2022-07-20T02:36:09.264259+0000","last_clean":"2022-07-20T02:36:09.264259+0000","last_became_active":"2022-07-20T02:25:25.351806+0000","last_became_peered":"2022-07-20T02:25:25.351806+0000","last_unstale":"2022-07-20T02:36:09.264259+0000","last_undegraded":"2022-07-20T02:36:09.264259+0000","last_fullsized":"2022-07-20T02:36:09.264259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_clean_scrub_stamp":"2022-07-20T02:25:19.583498+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.262881+0000","last_change":"2022-07-20T02:25:29.458996+0000","last_active":"2022-07-20T02:36:09.262881+0000","last_peered":"2022-07-20T02:36:09.262881+0000","last_clean":"2022-07-20T02:36:09.262881+0000","last_became_active":"2022-07-20T02:25:29.458580+0000","last_became_peered":"2022-07-20T02:25:29.458580+0000","last_unstale":"2022-07-20T02:36:09.262881+0000","last_undegraded":"2022-07-20T02:36:09.262881+0000","last_fullsized":"2022-07-20T02:36:09.262881+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.263966+0000","last_change":"2022-07-20T02:25:29.455281+0000","last_active":"2022-07-20T02:36:09.263966+0000","last_peered":"2022-07-20T02:36:09.263966+0000","last_clean":"2022-07-20T02:36:09.263966+0000","last_became_active":"2022-07-20T02:25:29.454947+0000","last_became_peered":"2022-07-20T02:25:29.454947+0000","last_unstale":"2022-07-20T02:36:09.263966+0000","last_undegraded":"2022-07-20T02:36:09.263966+0000","last_fullsized":"2022-07-20T02:36:09.263966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":142,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.263414+0000","last_change":"2022-07-20T02:25:29.454317+0000","last_active":"2022-07-20T02:36:09.263414+0000","last_peered":"2022-07-20T02:36:09.263414+0000","last_clean":"2022-07-20T02:36:09.263414+0000","last_became_active":"2022-07-20T02:25:29.453725+0000","last_became_peered":"2022-07-20T02:25:29.453725+0000","last_unstale":"2022-07-20T02:36:09.263414+0000","last_undegraded":"2022-07-20T02:36:09.263414+0000","last_fullsized":"2022-07-20T02:36:09.263414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.262821+0000","last_change":"2022-07-20T02:25:29.457758+0000","last_active":"2022-07-20T02:36:09.262821+0000","last_peered":"2022-07-20T02:36:09.262821+0000","last_clean":"2022-07-20T02:36:09.262821+0000","last_became_active":"2022-07-20T02:25:29.457326+0000","last_became_peered":"2022-07-20T02:25:29.457326+0000","last_unstale":"2022-07-20T02:36:09.262821+0000","last_undegraded":"2022-07-20T02:36:09.262821+0000","last_fullsized":"2022-07-20T02:36:09.262821+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.263961+0000","last_change":"2022-07-20T02:25:29.902408+0000","last_active":"2022-07-20T02:36:09.263961+0000","last_peered":"2022-07-20T02:36:09.263961+0000","last_clean":"2022-07-20T02:36:09.263961+0000","last_became_active":"2022-07-20T02:25:29.901376+0000","last_became_peered":"2022-07-20T02:25:29.901376+0000","last_unstale":"2022-07-20T02:36:09.263961+0000","last_undegraded":"2022-07-20T02:36:09.263961+0000","last_fullsized":"2022-07-20T02:36:09.263961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264800+0000","last_change":"2022-07-20T02:25:29.909282+0000","last_active":"2022-07-20T02:36:09.264800+0000","last_peered":"2022-07-20T02:36:09.264800+0000","last_clean":"2022-07-20T02:36:09.264800+0000","last_became_active":"2022-07-20T02:25:29.908425+0000","last_became_peered":"2022-07-20T02:25:29.908425+0000","last_unstale":"2022-07-20T02:36:09.264800+0000","last_undegraded":"2022-07-20T02:36:09.264800+0000","last_fullsized":"2022-07-20T02:36:09.264800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264964+0000","last_change":"2022-07-20T02:25:29.452052+0000","last_active":"2022-07-20T02:36:09.264964+0000","last_peered":"2022-07-20T02:36:09.264964+0000","last_clean":"2022-07-20T02:36:09.264964+0000","last_became_active":"2022-07-20T02:25:29.451566+0000","last_became_peered":"2022-07-20T02:25:29.451566+0000","last_unstale":"2022-07-20T02:36:09.264964+0000","last_undegraded":"2022-07-20T02:36:09.264964+0000","last_fullsized":"2022-07-20T02:36:09.264964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:08.652855+0000","last_change":"2022-07-20T02:25:29.458394+0000","last_active":"2022-07-20T02:36:08.652855+0000","last_peered":"2022-07-20T02:36:08.652855+0000","last_clean":"2022-07-20T02:36:08.652855+0000","last_became_active":"2022-07-20T02:25:29.458000+0000","last_became_peered":"2022-07-20T02:25:29.458000+0000","last_unstale":"2022-07-20T02:36:08.652855+0000","last_undegraded":"2022-07-20T02:36:08.652855+0000","last_fullsized":"2022-07-20T02:36:08.652855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575705,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.52700000000000002,"5min":0.47099999999999997,"15min":0.43099999999999999},"max":{"1min":1.0329999999999999,"5min":1.089,"15min":1.1739999999999999},"last":0.87},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.753,"5min":0.56899999999999995,"15min":0.52400000000000002},"max":{"1min":1.099,"5min":1.115,"15min":1.2430000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.57899999999999996,"5min":0.54500000000000004,"15min":0.52000000000000002},"max":{"1min":0.96199999999999997,"5min":1.208,"15min":1.2749999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81499999999999995,"15min":0.81999999999999995},"min":{"1min":0.61599999999999999,"5min":0.60799999999999998,"15min":0.56899999999999995},"max":{"1min":0.997,"5min":1.042,"15min":1.1830000000000001},"last":0.81999999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.86499999999999999,"15min":0.872},"min":{"1min":0.63,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.0349999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.91700000000000004,"15min":0.92900000000000005},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.04,"5min":1.135,"15min":1.24},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81100000000000005,"15min":0.82699999999999996},"min":{"1min":0.63700000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.99299999999999999,"5min":1.093,"15min":1.159},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.875,"15min":0.879},"min":{"1min":0.61499999999999999,"5min":0.61399999999999999,"15min":0.54700000000000004},"max":{"1min":1.0600000000000001,"5min":1.244,"15min":1.244},"last":1.032}]},{"osd":4,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.50700000000000001},"max":{"1min":1.0029999999999999,"5min":1.111,"15min":1.163},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92900000000000005,"15min":0.92500000000000004},"min":{"1min":0.69099999999999995,"5min":0.67800000000000005,"15min":0.56799999999999995},"max":{"1min":1.129,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.1240000000000001}]},{"osd":5,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.871,"15min":0.874},"min":{"1min":0.57499999999999996,"5min":0.53500000000000003,"15min":0.47299999999999998},"max":{"1min":1.044,"5min":1.1719999999999999,"15min":1.355},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.94999999999999996,"15min":0.94599999999999995},"min":{"1min":0.61499999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1479999999999999,"5min":1.2629999999999999,"15min":1.266},"last":1.083}]},{"osd":6,"last update":"Wed Jul 20 03:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90100000000000002},"min":{"1min":0.67300000000000004,"5min":0.62,"15min":0.62},"max":{"1min":1.1930000000000001,"5min":1.208,"15min":1.2669999999999999},"last":1.008},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.91300000000000003,"15min":0.90500000000000003},"min":{"1min":0.59899999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":1.165}]}]},{"osd":6,"up_from":13,"seq":55834575705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":1.163,"15min":1.1739999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51200000000000001},"max":{"1min":1.0509999999999999,"5min":1.119,"15min":1.165},"last":0.91700000000000004}]},{"osd":1,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84699999999999998,"15min":0.82399999999999995},"min":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.036,"5min":1.149,"15min":1.2609999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.84199999999999997,"15min":0.85799999999999998},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60099999999999998},"max":{"1min":1.111,"5min":1.1200000000000001,"15min":1.1970000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81000000000000005,"15min":0.80700000000000005},"min":{"1min":0.624,"5min":0.59199999999999997,"15min":0.58299999999999996},"max":{"1min":1.004,"5min":1.085,"15min":1.272},"last":0.748},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.86499999999999999,"15min":0.86699999999999999},"min":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.0269999999999999,"5min":1.121,"15min":1.2989999999999999},"last":0.878}]},{"osd":3,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.874,"15min":0.86599999999999999},"min":{"1min":0.74299999999999999,"5min":0.626,"15min":0.54400000000000004},"max":{"1min":1.097,"5min":1.1639999999999999,"15min":1.2090000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.62,"15min":0.61799999999999999},"max":{"1min":0.98099999999999998,"5min":1.069,"15min":1.179},"last":0.84099999999999997}]},{"osd":4,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76400000000000001,"15min":0.77400000000000002},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51500000000000001},"max":{"1min":1.121,"5min":1.1839999999999999,"15min":1.2609999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84999999999999998,"15min":0.83699999999999997},"min":{"1min":0.60099999999999998,"5min":0.56399999999999995,"15min":0.38700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.86599999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81399999999999995,"15min":0.82099999999999995},"min":{"1min":0.60799999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.038,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":1.038},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.56100000000000005},"max":{"1min":1.077,"5min":1.327,"15min":1.3360000000000001},"last":1.0129999999999999}]},{"osd":7,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93500000000000005,"15min":0.94399999999999995},"min":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.436},"max":{"1min":1.403,"5min":1.403,"15min":1.4379999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.80900000000000005,"5min":0.63800000000000001,"15min":0.56000000000000005},"max":{"1min":1.083,"5min":1.2050000000000001,"15min":1.5389999999999999},"last":0.96499999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.81799999999999995,"15min":0.80900000000000005},"min":{"1min":0.59299999999999997,"5min":0.57299999999999995,"15min":0.441},"max":{"1min":1.125,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.61499999999999999,"15min":0.59899999999999998},"max":{"1min":1.1200000000000001,"5min":1.1220000000000001,"15min":1.1779999999999999},"last":0.77600000000000002}]},{"osd":1,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.83499999999999996},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.84999999999999998,"15min":0.84599999999999997},"min":{"1min":0.76800000000000002,"5min":0.625,"15min":0.60399999999999998},"max":{"1min":0.99399999999999999,"5min":1.073,"15min":1.149},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85699999999999998,"15min":0.86899999999999999},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.621},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86599999999999999,"15min":0.877},"min":{"1min":0.65700000000000003,"5min":0.61399999999999999,"15min":0.61199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.2210000000000001},"last":0.85199999999999998}]},{"osd":3,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"min":{"1min":0.70199999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.0109999999999999,"5min":1.232,"15min":1.232},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91300000000000003,"15min":0.91000000000000003},"min":{"1min":0.78700000000000003,"5min":0.65900000000000003,"15min":0.63800000000000001},"max":{"1min":1.111,"5min":1.3,"15min":1.3},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.94299999999999995,"15min":0.92000000000000004},"min":{"1min":0.68200000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1950000000000001,"5min":1.242,"15min":1.268},"last":0.83699999999999997}]},{"osd":6,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92900000000000005,"15min":0.90800000000000003},"min":{"1min":0.65300000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2549999999999999,"5min":1.357,"15min":1.357},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.67900000000000005,"5min":0.63200000000000001,"15min":0.59399999999999997},"max":{"1min":1.21,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.91200000000000003}]},{"osd":7,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.995,"15min":0.98999999999999999},"min":{"1min":0.82899999999999996,"5min":0.66600000000000004,"15min":0.63800000000000001},"max":{"1min":1.216,"5min":1.2929999999999999,"15min":1.4350000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.042,"15min":1.0229999999999999},"min":{"1min":0.82499999999999996,"5min":0.70299999999999996,"15min":0.60099999999999998},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.014}]}]},{"osd":4,"up_from":13,"seq":55834575705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.53000000000000003},"max":{"1min":1.081,"5min":1.101,"15min":1.149},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.60699999999999998,"5min":0.59199999999999997,"15min":0.56599999999999995},"max":{"1min":1.0980000000000001,"5min":1.1339999999999999,"15min":1.2150000000000001},"last":1.028}]},{"osd":1,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.83299999999999996},"min":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.59799999999999998},"max":{"1min":1.0620000000000001,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.878,"15min":0.86299999999999999},"min":{"1min":0.63900000000000001,"5min":0.61399999999999999,"15min":0.59399999999999997},"max":{"1min":1.0760000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.90400000000000003,"15min":0.88400000000000001},"min":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.58899999999999997},"max":{"1min":1.1779999999999999,"5min":1.1919999999999999,"15min":1.2390000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89200000000000002,"15min":0.90000000000000002},"min":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.59799999999999998},"max":{"1min":1.171,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.1220000000000001}]},{"osd":3,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.84399999999999997,"15min":0.85099999999999998},"min":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.60599999999999998},"max":{"1min":1.198,"5min":1.198,"15min":1.248},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92900000000000005,"15min":0.92400000000000004},"min":{"1min":0.68000000000000005,"5min":0.67800000000000005,"15min":0.65500000000000003},"max":{"1min":1.181,"5min":1.236,"15min":1.25},"last":0.95399999999999996}]},{"osd":5,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.86699999999999999,"15min":0.85499999999999998},"min":{"1min":0.58799999999999997,"5min":0.56699999999999995,"15min":0.54400000000000004},"max":{"1min":1.2030000000000001,"5min":1.268,"15min":1.268},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.58899999999999997,"15min":0.55300000000000005},"max":{"1min":1.272,"5min":1.2729999999999999,"15min":1.413},"last":1.149}]},{"osd":6,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.89800000000000002,"15min":0.91300000000000003},"min":{"1min":0.68100000000000005,"5min":0.623,"15min":0.60399999999999998},"max":{"1min":1.125,"5min":1.2390000000000001,"15min":1.2769999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.98299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.64400000000000002},"max":{"1min":1.2250000000000001,"5min":1.339,"15min":1.339},"last":1.006}]},{"osd":7,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.79300000000000004,"5min":0.65800000000000003,"15min":0.55600000000000005},"max":{"1min":1.3149999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.107,"5min":1.069,"15min":1.044},"min":{"1min":0.90300000000000002,"5min":0.70599999999999996,"15min":0.67300000000000004},"max":{"1min":1.444,"5min":1.4510000000000001,"15min":1.482},"last":1.258}]}]},{"osd":3,"up_from":13,"seq":55834575705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61199999999999999,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":0.98699999999999999,"5min":0.995,"15min":1.0620000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.46000000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.1299999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":0.79600000000000004,"5min":1.1180000000000001,"15min":1.236},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.99299999999999999,"5min":1.0820000000000001,"15min":1.2150000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82699999999999996,"15min":0.83999999999999997},"min":{"1min":0.59999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0680000000000001,"5min":1.171,"15min":1.2330000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.89200000000000002,"15min":0.89900000000000002},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.55500000000000005},"max":{"1min":1.143,"5min":1.143,"15min":1.206},"last":0.99199999999999999}]},{"osd":4,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.57899999999999996},"max":{"1min":1.0980000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.93799999999999994,"15min":0.95399999999999996},"min":{"1min":0.748,"5min":0.68899999999999995,"15min":0.63200000000000001},"max":{"1min":1.1870000000000001,"5min":1.579,"15min":1.579},"last":0.79000000000000004}]},{"osd":5,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.95999999999999996,"15min":0.98499999999999999},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.68600000000000005},"max":{"1min":1.198,"5min":1.4590000000000001,"15min":1.4690000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.016,"15min":1.036},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.69799999999999995},"max":{"1min":1.3979999999999999,"5min":1.601,"15min":1.6379999999999999},"last":0.92000000000000004}]},{"osd":6,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.014,"15min":1.012},"min":{"1min":0.77900000000000003,"5min":0.73899999999999999,"15min":0.69499999999999995},"max":{"1min":1.3620000000000001,"5min":1.534,"15min":1.534},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.052,"15min":1.0629999999999999},"min":{"1min":0.82399999999999995,"5min":0.73599999999999999,"15min":0.72799999999999998},"max":{"1min":1.2390000000000001,"5min":1.661,"15min":1.661},"last":1.075}]},{"osd":7,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.077,"15min":1.105},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.74299999999999999},"max":{"1min":1.4510000000000001,"5min":1.5209999999999999,"15min":1.681},"last":1.032},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.88200000000000001,"5min":0.879,"15min":0.76000000000000001},"max":{"1min":1.294,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.198}]}]},{"osd":2,"up_from":13,"seq":55834575705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.1080000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.71499999999999997,"15min":0.72699999999999998},"min":{"1min":0.52400000000000002,"5min":0.48099999999999998,"15min":0.44800000000000001},"max":{"1min":1.121,"5min":1.1479999999999999,"15min":1.2270000000000001},"last":0.77100000000000002}]},{"osd":1,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.46200000000000002},"max":{"1min":0.76900000000000002,"5min":1.1359999999999999,"15min":1.238},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.50900000000000001},"max":{"1min":0.873,"5min":1.151,"15min":1.194},"last":0.71399999999999997}]},{"osd":3,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.69699999999999995,"5min":0.66200000000000003,"15min":0.628},"max":{"1min":1.1379999999999999,"5min":1.1910000000000001,"15min":1.2470000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78500000000000003,"15min":0.79200000000000004},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52400000000000002},"max":{"1min":0.90400000000000003,"5min":1.2529999999999999,"15min":1.351},"last":0.58499999999999996}]},{"osd":4,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88200000000000001},"min":{"1min":0.72999999999999998,"5min":0.65100000000000002,"15min":0.60499999999999998},"max":{"1min":1.0920000000000001,"5min":1.373,"15min":1.373},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.63},"max":{"1min":1.2929999999999999,"5min":1.385,"15min":1.4179999999999999},"last":0.83599999999999997}]},{"osd":5,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96699999999999997,"15min":0.97399999999999998},"min":{"1min":0.79100000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2729999999999999,"5min":1.546,"15min":1.546},"last":1.038},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.996,"15min":0.997},"min":{"1min":0.80600000000000005,"5min":0.70099999999999996,"15min":0.68300000000000005},"max":{"1min":1.181,"5min":1.4830000000000001,"15min":1.645},"last":0.84899999999999998}]},{"osd":6,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.0429999999999999},"min":{"1min":0.75700000000000001,"5min":0.75,"15min":0.66700000000000004},"max":{"1min":1.232,"5min":1.514,"15min":1.595},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0880000000000001,"15min":1.073},"min":{"1min":0.80500000000000005,"5min":0.77300000000000002,"15min":0.69699999999999995},"max":{"1min":1.3049999999999999,"5min":1.611,"15min":1.611},"last":0.95999999999999996}]},{"osd":7,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1140000000000001,"15min":1.1020000000000001},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.73599999999999999},"max":{"1min":1.339,"5min":1.4990000000000001,"15min":2.0960000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.121,"15min":1.1259999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.359,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.097}]}]},{"osd":1,"up_from":13,"seq":55834575705,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.45100000000000001,"15min":0.42599999999999999},"max":{"1min":0.94099999999999995,"5min":0.99199999999999999,"15min":1.046},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":0.997,"5min":0.997,"15min":1.181},"last":0.64900000000000002}]},{"osd":2,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.71499999999999997},"min":{"1min":0.52400000000000002,"5min":0.51600000000000001,"15min":0.496},"max":{"1min":1.085,"5min":1.085,"15min":1.147},"last":0.498},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71699999999999997,"15min":0.70499999999999996},"min":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.47399999999999998},"max":{"1min":0.96599999999999997,"5min":1.139,"15min":1.139},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.002,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.70999999999999996,"15min":0.72399999999999998},"min":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"max":{"1min":1.123,"5min":1.123,"15min":1.171},"last":0.56599999999999995}]},{"osd":4,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.80300000000000005,"5min":0.70999999999999996,"15min":0.65900000000000003},"max":{"1min":1.159,"5min":1.169,"15min":1.3},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91500000000000004,"15min":0.91400000000000003},"min":{"1min":0.71899999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.089,"5min":1.2190000000000001,"15min":1.3169999999999999},"last":0.81000000000000005}]},{"osd":5,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94499999999999995,"15min":0.96399999999999997},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.1200000000000001,"5min":1.2749999999999999,"15min":1.395},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.052,"5min":1.038,"15min":1.0529999999999999},"min":{"1min":0.88200000000000001,"5min":0.84499999999999997,"15min":0.64200000000000002},"max":{"1min":1.3080000000000001,"5min":1.3799999999999999,"15min":1.486},"last":0.95799999999999996}]},{"osd":6,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.018,"15min":1.046},"min":{"1min":0.76400000000000001,"5min":0.69099999999999995,"15min":0.65800000000000003},"max":{"1min":1.444,"5min":1.444,"15min":1.589},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.014,"15min":1.016},"min":{"1min":0.78800000000000003,"5min":0.77200000000000002,"15min":0.69299999999999995},"max":{"1min":1.3380000000000001,"5min":1.653,"15min":1.653},"last":0.88200000000000001}]},{"osd":7,"last update":"Wed Jul 20 03:35:38 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0109999999999999,"15min":1.04},"min":{"1min":0.754,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3080000000000001,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.107,"15min":1.127},"min":{"1min":1.0049999999999999,"5min":0.77700000000000002,"15min":0.71099999999999997},"max":{"1min":1.391,"5min":1.411,"15min":1.544},"last":0.996}]}]},{"osd":0,"up_from":13,"seq":55834575705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.51500000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.80800000000000005,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.52800000000000002},"max":{"1min":0.91800000000000004,"5min":1.0820000000000001,"15min":1.1779999999999999},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64900000000000002,"15min":0.63700000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.48699999999999999},"max":{"1min":0.88500000000000001,"5min":0.96999999999999997,"15min":1.089},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.53200000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.92700000000000005,"5min":1.161,"15min":1.1639999999999999},"last":0.62}]},{"osd":3,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.55200000000000005,"5min":0.53100000000000003,"15min":0.504},"max":{"1min":1.085,"5min":1.1559999999999999,"15min":1.1719999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77300000000000002,"15min":0.76200000000000001},"min":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":1.0549999999999999,"5min":1.177,"15min":1.177},"last":0.86299999999999999}]},{"osd":4,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.93799999999999994,"15min":0.93500000000000005},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.96199999999999997,"15min":0.95999999999999996},"min":{"1min":0.749,"5min":0.71099999999999997,"15min":0.69199999999999995},"max":{"1min":1.29,"5min":1.29,"15min":1.3400000000000001},"last":0.78000000000000003}]},{"osd":5,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96699999999999997,"15min":0.98499999999999999},"min":{"1min":0.78700000000000003,"5min":0.69499999999999995,"15min":0.66800000000000004},"max":{"1min":1.2390000000000001,"5min":1.3220000000000001,"15min":1.3560000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97299999999999998,"15min":0.96199999999999997},"min":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.65800000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.52},"last":1.2629999999999999}]},{"osd":6,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0269999999999999,"15min":1.0129999999999999},"min":{"1min":0.80700000000000005,"5min":0.72799999999999998,"15min":0.624},"max":{"1min":1.329,"5min":1.3420000000000001,"15min":1.4690000000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.075,"5min":1.0369999999999999,"15min":1.056},"min":{"1min":0.93799999999999994,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.615},"last":1.0549999999999999}]},{"osd":7,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.069,"15min":1.077},"min":{"1min":0.95299999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.268,"5min":1.401,"15min":1.7050000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.091,"15min":1.1020000000000001},"min":{"1min":0.89800000000000002,"5min":0.82399999999999995,"15min":0.78600000000000003},"max":{"1min":1.379,"5min":1.419,"15min":1.6399999999999999},"last":0.995}]}]}]}} 2022-07-20T03:36:41.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:36:42.056 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 Processor -- start 2022-07-20T03:36:42.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 -- start start 2022-07-20T03:36:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f201474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f1f200a8c40 0x7f1f200af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b62d0 con 0x7f1f200a7fc0 2022-07-20T03:36:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2014aac0 con 0x7f1f200a8c40 2022-07-20T03:36:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f25daf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2014ad60 con 0x7f1f200b5ef0 2022-07-20T03:36:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f1ffff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1f200a7fc0 0x7f1f200aee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44156/0 (socket says 172.21.15.26:44156) 2022-07-20T03:36:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f1ffff700 1 -- 172.21.15.26:0/160005396 learned_addr learned my addr 172.21.15.26:0/160005396 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f255ae700 1 --2- 172.21.15.26:0/160005396 >> v2:172.21.15.26:3300/0 conn(0x7f1f200a8c40 0x7f1f200af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f24dad700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f201474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711304723 0 0) 0x7f1f200b62d0 con 0x7f1f200a7fc0 2022-07-20T03:36:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.055+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0c003260 con 0x7f1f200a7fc0 2022-07-20T03:36:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703219530 0 0) 0x7f1f0c003260 con 0x7f1f200a7fc0 2022-07-20T03:36:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 msgr2=0x7f1f201474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:42.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f201474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.080 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 >> v2:172.21.15.26:3300/0 conn(0x7f1f200a8c40 msgr2=0x7f1f200af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/160005396 >> v2:172.21.15.26:3300/0 conn(0x7f1f200a8c40 0x7f1f200af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2014b000 con 0x7f1f200a7fc0 2022-07-20T03:36:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f1f100019e0 con 0x7f1f200a7fc0 2022-07-20T03:36:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 >> v1:172.21.15.26:6789/0 conn(0x7f1f200a7fc0 legacy=0x7f1f200aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f0c003750 0x7f1f0c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.084 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c004500 0x7f1f0c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f0c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2014aac0 con 0x7f1f0c003750 2022-07-20T03:36:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2014ad60 con 0x7f1f0c004500 2022-07-20T03:36:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c00b750 con 0x7f1f200b5ef0 2022-07-20T03:36:42.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f0c003750 0x7f1f0c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.087 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f255ae700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f0c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1ffff700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c004500 0x7f1f0c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 -- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 msgr2=0x7f1f0c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f200b5ef0 0x7f1f0c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 -- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c004500 msgr2=0x7f1f0c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c004500 0x7f1f0c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 -- 172.21.15.26:0/160005396 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c003260 con 0x7f1f0c003750 2022-07-20T03:36:42.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f24dad700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f0c003750 0x7f1f0c003b70 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f1f1400c000 tx=0x7f1f14006df0).ready entity=mon.0 client_cookie=d1a923f7a84f7397 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:42.090 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1400e830 con 0x7f1f0c003750 2022-07-20T03:36:42.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1400e990 con 0x7f1f0c003750 2022-07-20T03:36:42.091 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f1f7fe700 1 -- 172.21.15.26:0/160005396 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f1400ecf0 con 0x7f1f0c003750 2022-07-20T03:36:42.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 -- 172.21.15.26:0/160005396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f0c003750 msgr2=0x7f1f0c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.092 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 --2- 172.21.15.26:0/160005396 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f0c003750 0x7f1f0c003b70 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f1f1400c000 tx=0x7f1f14006df0).stop 2022-07-20T03:36:42.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f255ae700 1 -- 172.21.15.26:0/160005396 reap_dead start 2022-07-20T03:36:42.093 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 -- 172.21.15.26:0/160005396 shutdown_connections 2022-07-20T03:36:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 -- 172.21.15.26:0/160005396 >> 172.21.15.26:0/160005396 conn(0x7f1f20060310 msgr2=0x7f1f200b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 -- 172.21.15.26:0/160005396 shutdown_connections 2022-07-20T03:36:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 -- 172.21.15.26:0/160005396 wait complete. 2022-07-20T03:36:42.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.059+0000 7f1f25daf700 1 Processor -- start 2022-07-20T03:36:42.095 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- start start 2022-07-20T03:36:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c003750 0x7f1f20146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f2014a260 0x7f1f200b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200b48e0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200a9630 con 0x7f1f0c003750 2022-07-20T03:36:42.097 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20147cc0 con 0x7f1f2014a260 2022-07-20T03:36:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f24dad700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c003750 0x7f1f20146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44892/0 (socket says 172.21.15.26:44892) 2022-07-20T03:36:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 -- 172.21.15.26:0/2617888589 learned_addr learned my addr 172.21.15.26:0/2617888589 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f1ffff700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f2014a260 0x7f1f200b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 -- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f2014a260 msgr2=0x7f1f200b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f2014a260 0x7f1f200b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 -- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c003750 msgr2=0x7f1f20146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c003750 0x7f1f20146460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 -- 172.21.15.26:0/2617888589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20140700 con 0x7f1f2013c6b0 2022-07-20T03:36:42.106 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f255ae700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f1f1800b000 tx=0x7f1f18006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:42.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f180135b0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18013710 con 0x7f1f2013c6b0 2022-07-20T03:36:42.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f201409a0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f18013a70 con 0x7f1f2013c6b0 2022-07-20T03:36:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f20145370 con 0x7f1f2013c6b0 2022-07-20T03:36:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.063+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200b4df0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.067+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f18013bd0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.067+0000 7f1f1dffb700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 0x7f1f00055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.067+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1f1806ccf0 con 0x7f1f2013c6b0 2022-07-20T03:36:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.067+0000 7f1f24dad700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 0x7f1f00055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.067+0000 7f1f24dad700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 0x7f1f00055f60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f1f1401e000 tx=0x7f1f14019000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.071+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f18043d90 con 0x7f1f2013c6b0 2022-07-20T03:36:42.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.199+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f2014a640 con 0x7f1f2013c6b0 2022-07-20T03:36:42.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.199+0000 7f1f1dffb700 1 -- 172.21.15.26:0/2617888589 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+13601 (secure 0 0 0) 0x7f1f18013e60 con 0x7f1f2013c6b0 2022-07-20T03:36:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T03:36:42.202 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":165,"fsid":"36c84aa7-9150-4850-bbd5-e28fb0ff0eb0","created":"2022-07-20T02:25:16.528661+0000","modified":"2022-07-20T02:36:09.646654+0000","last_up_change":"2022-07-20T02:25:24.287145+0000","last_in_change":"2022-07-20T02:25:24.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T02:25:19.431119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T02:25:28.375708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9b99f86d-740c-4e86-aa14-2c202b408772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":23049},{"type":"v1","addr":"172.21.15.26:6803","nonce":23049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":23049},{"type":"v1","addr":"172.21.15.26:6805","nonce":23049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":23049},{"type":"v1","addr":"172.21.15.26:6809","nonce":23049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":23049},{"type":"v1","addr":"172.21.15.26:6807","nonce":23049}]},"public_addr":"172.21.15.26:6803/23049","cluster_addr":"172.21.15.26:6805/23049","heartbeat_back_addr":"172.21.15.26:6809/23049","heartbeat_front_addr":"172.21.15.26:6807/23049","state":["exists","up"]},{"osd":1,"uuid":"0256d8ae-ca5e-445d-8c9e-25203aeea92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":23050},{"type":"v1","addr":"172.21.15.26:6811","nonce":23050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":23050},{"type":"v1","addr":"172.21.15.26:6813","nonce":23050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":23050},{"type":"v1","addr":"172.21.15.26:6817","nonce":23050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":23050},{"type":"v1","addr":"172.21.15.26:6815","nonce":23050}]},"public_addr":"172.21.15.26:6811/23050","cluster_addr":"172.21.15.26:6813/23050","heartbeat_back_addr":"172.21.15.26:6817/23050","heartbeat_front_addr":"172.21.15.26:6815/23050","state":["exists","up"]},{"osd":2,"uuid":"e7012e19-7787-4117-be57-55c92bd2e289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":23051},{"type":"v1","addr":"172.21.15.26:6819","nonce":23051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":23051},{"type":"v1","addr":"172.21.15.26:6821","nonce":23051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":23051},{"type":"v1","addr":"172.21.15.26:6825","nonce":23051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":23051},{"type":"v1","addr":"172.21.15.26:6823","nonce":23051}]},"public_addr":"172.21.15.26:6819/23051","cluster_addr":"172.21.15.26:6821/23051","heartbeat_back_addr":"172.21.15.26:6825/23051","heartbeat_front_addr":"172.21.15.26:6823/23051","state":["exists","up"]},{"osd":3,"uuid":"8abca3ca-a3f0-44e2-80e9-a3d7ff287442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":23052},{"type":"v1","addr":"172.21.15.26:6827","nonce":23052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":23052},{"type":"v1","addr":"172.21.15.26:6829","nonce":23052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":23052},{"type":"v1","addr":"172.21.15.26:6833","nonce":23052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":23052},{"type":"v1","addr":"172.21.15.26:6831","nonce":23052}]},"public_addr":"172.21.15.26:6827/23052","cluster_addr":"172.21.15.26:6829/23052","heartbeat_back_addr":"172.21.15.26:6833/23052","heartbeat_front_addr":"172.21.15.26:6831/23052","state":["exists","up"]},{"osd":4,"uuid":"9a0750c5-1042-4cfe-bb8e-4b8ee006a6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":23448},{"type":"v1","addr":"172.21.15.120:6817","nonce":23448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":23448},{"type":"v1","addr":"172.21.15.120:6819","nonce":23448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":23448},{"type":"v1","addr":"172.21.15.120:6823","nonce":23448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":23448},{"type":"v1","addr":"172.21.15.120:6821","nonce":23448}]},"public_addr":"172.21.15.120:6817/23448","cluster_addr":"172.21.15.120:6819/23448","heartbeat_back_addr":"172.21.15.120:6823/23448","heartbeat_front_addr":"172.21.15.120:6821/23448","state":["exists","up"]},{"osd":5,"uuid":"4d5a42ac-83f0-4bf0-8aa9-4411bf3821f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":23447},{"type":"v1","addr":"172.21.15.120:6801","nonce":23447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":23447},{"type":"v1","addr":"172.21.15.120:6803","nonce":23447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":23447},{"type":"v1","addr":"172.21.15.120:6807","nonce":23447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":23447},{"type":"v1","addr":"172.21.15.120:6805","nonce":23447}]},"public_addr":"172.21.15.120:6801/23447","cluster_addr":"172.21.15.120:6803/23447","heartbeat_back_addr":"172.21.15.120:6807/23447","heartbeat_front_addr":"172.21.15.120:6805/23447","state":["exists","up"]},{"osd":6,"uuid":"87a7cb28-e512-4d57-910f-5b4b8bc96eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":23449},{"type":"v1","addr":"172.21.15.120:6809","nonce":23449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":23449},{"type":"v1","addr":"172.21.15.120:6811","nonce":23449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":23449},{"type":"v1","addr":"172.21.15.120:6815","nonce":23449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":23449},{"type":"v1","addr":"172.21.15.120:6813","nonce":23449}]},"public_addr":"172.21.15.120:6809/23449","cluster_addr":"172.21.15.120:6811/23449","heartbeat_back_addr":"172.21.15.120:6815/23449","heartbeat_front_addr":"172.21.15.120:6813/23449","state":["exists","up"]},{"osd":7,"uuid":"b1607f07-e4fb-42df-84d9-b5c855babf2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":23460},{"type":"v1","addr":"172.21.15.120:6825","nonce":23460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":23460},{"type":"v1","addr":"172.21.15.120:6827","nonce":23460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":23460},{"type":"v1","addr":"172.21.15.120:6831","nonce":23460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":23460},{"type":"v1","addr":"172.21.15.120:6829","nonce":23460}]},"public_addr":"172.21.15.120:6825/23460","cluster_addr":"172.21.15.120:6827/23460","heartbeat_back_addr":"172.21.15.120:6831/23460","heartbeat_front_addr":"172.21.15.120:6829/23460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.444208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.626271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.733012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.679666+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.704323+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.588747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.961520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T02:25:22.657760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.26:6837/1434658358":"2022-07-21T02:36:06.314800+0000","172.21.15.26:6835/1434658358":"2022-07-21T02:36:06.314800+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:36:42.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 msgr2=0x7f1f00055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 0x7f1f00055f60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f1f1401e000 tx=0x7f1f14019000).stop 2022-07-20T03:36:42.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 msgr2=0x7f1f2013cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:42.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f1f1800b000 tx=0x7f1f18006df0).stop 2022-07-20T03:36:42.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 shutdown_connections 2022-07-20T03:36:42.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2013c6b0 0x7f1f2013cb30 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1f00053aa0 0x7f1f00055f60 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f2014a260 0x7f1f200b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 --2- 172.21.15.26:0/2617888589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f0c003750 0x7f1f20146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:42.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 >> 172.21.15.26:0/2617888589 conn(0x7f1f20060310 msgr2=0x7f1f201441f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:42.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 shutdown_connections 2022-07-20T03:36:42.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:42.203+0000 7f1f25daf700 1 -- 172.21.15.26:0/2617888589 wait complete. 2022-07-20T03:36:43.220 INFO:tasks.ceph:Scrubbing osd.0 2022-07-20T03:36:43.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:43.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.343+0000 7fc79163c700 1 Processor -- start 2022-07-20T03:36:43.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.343+0000 7fc79163c700 1 -- start start 2022-07-20T03:36:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fc78c064220 0x7fc78c1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0b2e40 con 0x7fc78c0b3060 2022-07-20T03:36:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c0b4c20 con 0x7fc78c064220 2022-07-20T03:36:43.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0617a0 con 0x7fc78c0b7d80 2022-07-20T03:36:43.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78b7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc78c0b3060 0x7fc78c0b4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44176/0 (socket says 172.21.15.26:44176) 2022-07-20T03:36:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78b7fe700 1 -- 172.21.15.26:0/36747977 learned_addr learned my addr 172.21.15.26:0/36747977 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 --2- 172.21.15.26:0/36747977 >> v2:172.21.15.26:3300/0 conn(0x7fc78c064220 0x7fc78c1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78bfff700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236532559 0 0) 0x7fc78c0b2e40 con 0x7fc78c0b3060 2022-07-20T03:36:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76c003260 con 0x7fc78c0b3060 2022-07-20T03:36:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 -- 172.21.15.26:0/36747977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 msgr2=0x7fc78c0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119797643 0 0) 0x7fc76c003260 con 0x7fc78c0b3060 2022-07-20T03:36:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 -- 172.21.15.26:0/36747977 >> v1:172.21.15.26:6789/0 conn(0x7fc78c0b3060 legacy=0x7fc78c0b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 -- 172.21.15.26:0/36747977 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c060b20 con 0x7fc78c064220 2022-07-20T03:36:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78bfff700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc790e3b700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c064220 0x7fc78c1434f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7fc774001b80 tx=0x7fc784006df0).ready entity=mon.0 client_cookie=9aa99a87ff1cad80 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7740019e0 con 0x7fc78c064220 2022-07-20T03:36:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784000e90 con 0x7fc78c064220 2022-07-20T03:36:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc78affd700 1 -- 172.21.15.26:0/36747977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc784012460 con 0x7fc78c064220 2022-07-20T03:36:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- 172.21.15.26:0/36747977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c064220 msgr2=0x7fc78c1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c064220 0x7fc78c1434f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7fc774001b80 tx=0x7fc784006df0).stop 2022-07-20T03:36:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- 172.21.15.26:0/36747977 shutdown_connections 2022-07-20T03:36:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c064220 0x7fc78c1434f0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 --2- 172.21.15.26:0/36747977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- 172.21.15.26:0/36747977 >> 172.21.15.26:0/36747977 conn(0x7fc78c05f840 msgr2=0x7fc78c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- 172.21.15.26:0/36747977 shutdown_connections 2022-07-20T03:36:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.347+0000 7fc79163c700 1 -- 172.21.15.26:0/36747977 wait complete. 2022-07-20T03:36:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 Processor -- start 2022-07-20T03:36:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- start start 2022-07-20T03:36:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc78c064220 0x7fc78c0ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 0x7fc78c155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0abe20 con 0x7fc78c0b3060 2022-07-20T03:36:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c1494b0 con 0x7fc78c064220 2022-07-20T03:36:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c149100 con 0x7fc78c0b7d80 2022-07-20T03:36:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 0x7fc78c155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 0x7fc78c155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44904/0 (socket says 172.21.15.26:44904) 2022-07-20T03:36:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 -- 172.21.15.26:0/514109056 learned_addr learned my addr 172.21.15.26:0/514109056 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78bfff700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc78c064220 0x7fc78c0ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc790e3b700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 -- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 msgr2=0x7fc78c0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc78c0b7d80 0x7fc78c0ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 -- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc78c064220 msgr2=0x7fc78c0ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc78c064220 0x7fc78c0ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c00b180 con 0x7fc78c0b3060 2022-07-20T03:36:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78bfff700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc78c064220 0x7fc78c0ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:36:43.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc78b7fe700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 0x7fc78c155cc0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7fc774001c40 tx=0x7fc774002210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc774005ba0 con 0x7fc78c0b3060 2022-07-20T03:36:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78400b180 con 0x7fc78c0b3060 2022-07-20T03:36:43.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774005d00 con 0x7fc78c0b3060 2022-07-20T03:36:43.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc774007300 con 0x7fc78c0b3060 2022-07-20T03:36:43.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c149980 con 0x7fc78c0b3060 2022-07-20T03:36:43.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.351+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc78c149b80 con 0x7fc78c0b3060 2022-07-20T03:36:43.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc774010030 con 0x7fc78c0b3060 2022-07-20T03:36:43.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc770053b00 0x7fc770055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fc77406c8b0 con 0x7fc78c0b3060 2022-07-20T03:36:43.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fc77005a790 0x7fc77005cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc77005d210 con 0x7fc77005a790 2022-07-20T03:36:43.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc77406cc70 con 0x7fc78c0b3060 2022-07-20T03:36:43.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc78bfff700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc770053b00 0x7fc770055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc790e3b700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fc77005a790 0x7fc77005cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc78bfff700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc770053b00 0x7fc770055fc0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc77c00b000 tx=0x7fc77c008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.355+0000 7fc790e3b700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fc77005a790 0x7fc77005cc30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.359+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== osd.0 v2:172.21.15.26:6802/23049 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fc77005d210 con 0x7fc77005a790 2022-07-20T03:36:43.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.371+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 --> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc78c060dc0 con 0x7fc77005a790 2022-07-20T03:36:43.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc7897fa700 1 -- 172.21.15.26:0/514109056 <== osd.0 v2:172.21.15.26:6802/23049 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc78c060dc0 con 0x7fc77005a790 2022-07-20T03:36:43.399 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:43.400 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:43.400 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:43.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fc77005a790 msgr2=0x7fc77005cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.404 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6802/23049,v1:172.21.15.26:6803/23049] conn(0x7fc77005a790 0x7fc77005cc30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.405 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc770053b00 msgr2=0x7fc770055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fc770053b00 0x7fc770055fc0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc77c00b000 tx=0x7fc77c008040).stop 2022-07-20T03:36:43.406 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 msgr2=0x7fc78c155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 --2- 172.21.15.26:0/514109056 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc78c0b3060 0x7fc78c155cc0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7fc774001c40 tx=0x7fc774002210).stop 2022-07-20T03:36:43.407 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc790e3b700 1 -- 172.21.15.26:0/514109056 reap_dead start 2022-07-20T03:36:43.408 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 shutdown_connections 2022-07-20T03:36:43.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 >> 172.21.15.26:0/514109056 conn(0x7fc78c05f840 msgr2=0x7fc78c05cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.410 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 shutdown_connections 2022-07-20T03:36:43.411 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.375+0000 7fc79163c700 1 -- 172.21.15.26:0/514109056 wait complete. 2022-07-20T03:36:43.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2022-07-20T03:36:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 Processor -- start 2022-07-20T03:36:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- start start 2022-07-20T03:36:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 0x7ff3d40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3d4060950 0x7ff3d4060dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d40aa1e0 con 0x7ff3d40612b0 2022-07-20T03:36:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d40ae110 con 0x7ff3d4060950 2022-07-20T03:36:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d40a9f40 con 0x7ff3d40add30 2022-07-20T03:36:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3d4060950 0x7ff3d4060dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7ff3d4060950 0x7ff3d4060dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44912/0 (socket says 172.21.15.26:44912) 2022-07-20T03:36:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 -- 172.21.15.26:0/3524147588 learned_addr learned my addr 172.21.15.26:0/3524147588 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3da133700 1 --2- 172.21.15.26:0/3524147588 >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 0x7ff3d40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 -- 172.21.15.26:0/3524147588 >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 msgr2=0x7ff3d40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 --2- 172.21.15.26:0/3524147588 >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 0x7ff3d40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 -- 172.21.15.26:0/3524147588 >> v1:172.21.15.120:6789/0 conn(0x7ff3d40612b0 legacy=0x7ff3d4143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 -- 172.21.15.26:0/3524147588 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d40b1110 con 0x7ff3d4060950 2022-07-20T03:36:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3da133700 1 --2- 172.21.15.26:0/3524147588 >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 0x7ff3d40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:36:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9932700 1 --2- 172.21.15.26:0/3524147588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d4060950 0x7ff3d4060dc0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7ff3d40630a0 tx=0x7ff3d0006eb0).ready entity=mon.0 client_cookie=7b71eb617a62b58d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9131700 1 -- 172.21.15.26:0/3524147588 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d000d5c0 con 0x7ff3d4060950 2022-07-20T03:36:43.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9131700 1 -- 172.21.15.26:0/3524147588 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0000e90 con 0x7ff3d4060950 2022-07-20T03:36:43.539 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3d9131700 1 -- 172.21.15.26:0/3524147588 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d000d910 con 0x7ff3d4060950 2022-07-20T03:36:43.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- 172.21.15.26:0/3524147588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d4060950 msgr2=0x7ff3d4060dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 --2- 172.21.15.26:0/3524147588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d4060950 0x7ff3d4060dc0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7ff3d40630a0 tx=0x7ff3d0006eb0).stop 2022-07-20T03:36:43.540 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- 172.21.15.26:0/3524147588 shutdown_connections 2022-07-20T03:36:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 --2- 172.21.15.26:0/3524147588 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d4060950 0x7ff3d4060dc0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 --2- 172.21.15.26:0/3524147588 >> v2:172.21.15.120:3300/0 conn(0x7ff3d40add30 0x7ff3d40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- 172.21.15.26:0/3524147588 >> 172.21.15.26:0/3524147588 conn(0x7ff3d40b3050 msgr2=0x7ff3d40b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- 172.21.15.26:0/3524147588 shutdown_connections 2022-07-20T03:36:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.527+0000 7ff3db135700 1 -- 172.21.15.26:0/3524147588 wait complete. 2022-07-20T03:36:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 Processor -- start 2022-07-20T03:36:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- start start 2022-07-20T03:36:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d4060950 0x7ff3d4141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d40add30 0x7ff3d413af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4146b10 con 0x7ff3d40612b0 2022-07-20T03:36:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d413ed90 con 0x7ff3d4060950 2022-07-20T03:36:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4065010 con 0x7ff3d40add30 2022-07-20T03:36:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44922/0 (socket says 172.21.15.26:44922) 2022-07-20T03:36:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 -- 172.21.15.26:0/3015822871 learned_addr learned my addr 172.21.15.26:0/3015822871 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3da934700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d40add30 0x7ff3d413af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3da133700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d4060950 0x7ff3d4141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 -- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d40add30 msgr2=0x7ff3d413af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.547 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d40add30 0x7ff3d413af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 -- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d4060950 msgr2=0x7ff3d4141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d4060950 0x7ff3d4141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.549 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 -- 172.21.15.26:0/3015822871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d000b1e0 con 0x7ff3d40612b0 2022-07-20T03:36:43.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3d9932700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7ff3d4147150 tx=0x7ff3d0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.550 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d00068b0 con 0x7ff3d40612b0 2022-07-20T03:36:43.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d000d940 con 0x7ff3d40612b0 2022-07-20T03:36:43.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d0011d30 con 0x7ff3d40612b0 2022-07-20T03:36:43.551 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d4064a10 con 0x7ff3d40612b0 2022-07-20T03:36:43.552 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.531+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d40654e0 con 0x7ff3d40612b0 2022-07-20T03:36:43.554 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff3d0025030 con 0x7ff3d40612b0 2022-07-20T03:36:43.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d40656e0 con 0x7ff3d40612b0 2022-07-20T03:36:43.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3c37fe700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 0x7ff3b4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.555 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7ff3d006d060 con 0x7ff3d40612b0 2022-07-20T03:36:43.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3da133700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 0x7ff3b4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.556 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.535+0000 7ff3da133700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 0x7ff3b4055e10 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7ff3c400d000 tx=0x7ff3c4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.557 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.543+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3d0043ff0 con 0x7ff3d40612b0 2022-07-20T03:36:43.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.662+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3d40b13b0 con 0x7ff3b4053950 2022-07-20T03:36:43.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.662+0000 7ff3c37fe700 1 -- 172.21.15.26:0/3015822871 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff3d40b13b0 con 0x7ff3b4053950 2022-07-20T03:36:43.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 msgr2=0x7ff3b4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 0x7ff3b4055e10 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7ff3c400d000 tx=0x7ff3c4008040).stop 2022-07-20T03:36:43.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 msgr2=0x7ff3d413aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7ff3d4147150 tx=0x7ff3d0015000).stop 2022-07-20T03:36:43.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 shutdown_connections 2022-07-20T03:36:43.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d40612b0 0x7ff3d413aa50 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7ff3b4053950 0x7ff3b4055e10 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3d40add30 0x7ff3d413af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 --2- 172.21.15.26:0/3015822871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3d4060950 0x7ff3d4141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 >> 172.21.15.26:0/3015822871 conn(0x7ff3d40b3050 msgr2=0x7ff3d4065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 shutdown_connections 2022-07-20T03:36:43.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.666+0000 7ff3db135700 1 -- 172.21.15.26:0/3015822871 wait complete. 2022-07-20T03:36:43.677 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2022-07-20T03:36:43.682 INFO:tasks.ceph:Scrubbing osd.1 2022-07-20T03:36:43.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:43.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.794+0000 7f5a9f1b0700 1 Processor -- start 2022-07-20T03:36:43.800 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.794+0000 7f5a9f1b0700 1 -- start start 2022-07-20T03:36:43.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a98143090 0x7f5a98098490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.801 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5a980b0810 0x7f5a980950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5a980b0c50 0x7f5a980955d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98143470 con 0x7f5a980b0c50 2022-07-20T03:36:43.802 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9805fd90 con 0x7f5a980b0810 2022-07-20T03:36:43.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98097d50 con 0x7f5a98143090 2022-07-20T03:36:43.803 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5a980b0c50 0x7f5a980955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f5a980b0c50 0x7f5a980955d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44930/0 (socket says 172.21.15.26:44930) 2022-07-20T03:36:43.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 -- 172.21.15.26:0/1727298023 learned_addr learned my addr 172.21.15.26:0/1727298023 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.804 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9cf4c700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a98143090 0x7f5a98098490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a97fff700 1 --2- 172.21.15.26:0/1727298023 >> v2:172.21.15.120:3300/0 conn(0x7f5a980b0810 0x7f5a980950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 -- 172.21.15.26:0/1727298023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a98143090 msgr2=0x7f5a98098490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.805 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a98143090 0x7f5a98098490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 -- 172.21.15.26:0/1727298023 >> v2:172.21.15.120:3300/0 conn(0x7f5a980b0810 msgr2=0x7f5a980950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 --2- 172.21.15.26:0/1727298023 >> v2:172.21.15.120:3300/0 conn(0x7f5a980b0810 0x7f5a980950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.806 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 -- 172.21.15.26:0/1727298023 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98097ff0 con 0x7f5a980b0c50 2022-07-20T03:36:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9d74d700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980955d0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f5a98098c00 tx=0x7f5a90009290).ready entity=mon.0 client_cookie=85fb640bdedf22cc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.807 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a977fe700 1 -- 172.21.15.26:0/1727298023 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a9000b680 con 0x7f5a980b0c50 2022-07-20T03:36:43.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a977fe700 1 -- 172.21.15.26:0/1727298023 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9000b7e0 con 0x7f5a980b0c50 2022-07-20T03:36:43.808 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a977fe700 1 -- 172.21.15.26:0/1727298023 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a9000b950 con 0x7f5a980b0c50 2022-07-20T03:36:43.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/1727298023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 msgr2=0x7f5a980955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.819 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980955d0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f5a98098c00 tx=0x7f5a90009290).stop 2022-07-20T03:36:43.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/1727298023 shutdown_connections 2022-07-20T03:36:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980955d0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/1727298023 >> v2:172.21.15.120:3300/0 conn(0x7f5a980b0810 0x7f5a980950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/1727298023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a98143090 0x7f5a98098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/1727298023 >> 172.21.15.26:0/1727298023 conn(0x7f5a980b5e70 msgr2=0x7f5a980b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.823 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/1727298023 shutdown_connections 2022-07-20T03:36:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.798+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/1727298023 wait complete. 2022-07-20T03:36:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 Processor -- start 2022-07-20T03:36:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 -- start start 2022-07-20T03:36:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a980b0810 0x7f5a980aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a98143090 0x7f5a980aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980ae6e0 con 0x7f5a980b0c50 2022-07-20T03:36:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980ad520 con 0x7f5a98143090 2022-07-20T03:36:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9f1b0700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980ad280 con 0x7f5a980b0810 2022-07-20T03:36:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9d74d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a98143090 0x7f5a980aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9cf4c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a980b0810 0x7f5a980aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9d74d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a98143090 0x7f5a980aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:36394/0 (socket says 172.21.15.26:36394) 2022-07-20T03:36:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a97fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44936/0 (socket says 172.21.15.26:44936) 2022-07-20T03:36:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.802+0000 7f5a9d74d700 1 -- 172.21.15.26:0/2996407593 learned_addr learned my addr 172.21.15.26:0/2996407593 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 -- 172.21.15.26:0/2996407593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a980b0810 msgr2=0x7f5a980aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a980b0810 0x7f5a980aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 -- 172.21.15.26:0/2996407593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a98143090 msgr2=0x7f5a980aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a98143090 0x7f5a980aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90009530 con 0x7f5a980b0c50 2022-07-20T03:36:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a9cf4c700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a980b0810 0x7f5a980aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a97fff700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980adf00 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f5a8400c000 tx=0x7f5a84006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a84010780 con 0x7f5a980b0c50 2022-07-20T03:36:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a840108e0 con 0x7f5a980b0c50 2022-07-20T03:36:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a980ae940 con 0x7f5a980b0c50 2022-07-20T03:36:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a84010c40 con 0x7f5a980b0c50 2022-07-20T03:36:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a980af2d0 con 0x7f5a980b0c50 2022-07-20T03:36:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a980af580 con 0x7f5a980b0c50 2022-07-20T03:36:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5a8401b410 con 0x7f5a980b0c50 2022-07-20T03:36:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.806+0000 7f5a95ffb700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a78053980 0x7f5a78055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5a8406bdb0 con 0x7f5a980b0c50 2022-07-20T03:36:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a9cf4c700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a78053980 0x7f5a78055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a95ffb700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f5a7805a6b0 0x7f5a7805cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a7805d1d0 con 0x7f5a7805a6b0 2022-07-20T03:36:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a8406e440 con 0x7f5a980b0c50 2022-07-20T03:36:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a9d74d700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f5a7805a6b0 0x7f5a7805cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a9cf4c700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a78053980 0x7f5a78055e40 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f5a8c00b000 tx=0x7f5a8c006df0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a9d74d700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f5a7805a6b0 0x7f5a7805cb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.810+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== osd.1 v2:172.21.15.26:6810/23050 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f5a7805d1d0 con 0x7f5a7805a6b0 2022-07-20T03:36:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.826+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 --> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5a980aead0 con 0x7f5a7805a6b0 2022-07-20T03:36:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a95ffb700 1 -- 172.21.15.26:0/2996407593 <== osd.1 v2:172.21.15.26:6810/23050 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5a980aead0 con 0x7f5a7805a6b0 2022-07-20T03:36:43.844 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:43.845 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:43.845 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f5a7805a6b0 msgr2=0x7f5a7805cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6810/23050,v1:172.21.15.26:6811/23050] conn(0x7f5a7805a6b0 0x7f5a7805cb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a78053980 msgr2=0x7f5a78055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f5a78053980 0x7f5a78055e40 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f5a8c00b000 tx=0x7f5a8c006df0).stop 2022-07-20T03:36:43.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 msgr2=0x7f5a980adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 --2- 172.21.15.26:0/2996407593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a980b0c50 0x7f5a980adf00 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f5a8400c000 tx=0x7f5a84006df0).stop 2022-07-20T03:36:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9d74d700 1 -- 172.21.15.26:0/2996407593 reap_dead start 2022-07-20T03:36:43.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 shutdown_connections 2022-07-20T03:36:43.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 >> 172.21.15.26:0/2996407593 conn(0x7f5a980b5e70 msgr2=0x7f5a9805c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 shutdown_connections 2022-07-20T03:36:43.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.830+0000 7f5a9f1b0700 1 -- 172.21.15.26:0/2996407593 wait complete. 2022-07-20T03:36:43.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2022-07-20T03:36:43.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 Processor -- start 2022-07-20T03:36:43.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 -- start start 2022-07-20T03:36:43.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52140b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5214064450 0x7f5214064f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52140b85f0 con 0x7f52140b3380 2022-07-20T03:36:43.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5214064880 con 0x7f5214064450 2022-07-20T03:36:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52140b80b0 con 0x7f52140670c0 2022-07-20T03:36:43.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f5212d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f5212d9d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44944/0 (socket says 172.21.15.26:44944) 2022-07-20T03:36:43.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52140b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f5212d9d700 1 -- 172.21.15.26:0/2288504372 learned_addr learned my addr 172.21.15.26:0/2288504372 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f5211d9b700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.120:3300/0 conn(0x7f5214064450 0x7f5214064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 -- 172.21.15.26:0/2288504372 >> v2:172.21.15.120:3300/0 conn(0x7f5214064450 msgr2=0x7f5214064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.973 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.120:3300/0 conn(0x7f5214064450 0x7f5214064f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.974 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 -- 172.21.15.26:0/2288504372 >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 msgr2=0x7f5214065440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.980 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 -- 172.21.15.26:0/2288504372 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140b8350 con 0x7f52140670c0 2022-07-20T03:36:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f5212d9d700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:43.981 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521259c700 1 --2- 172.21.15.26:0/2288504372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52140b2e90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f52040032f0 tx=0x7f52040091d0).ready entity=mon.2 client_cookie=bfd24f729175a232 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521159a700 1 -- 172.21.15.26:0/2288504372 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f520400b600 con 0x7f52140670c0 2022-07-20T03:36:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521159a700 1 -- 172.21.15.26:0/2288504372 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520400b760 con 0x7f52140670c0 2022-07-20T03:36:43.982 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.962+0000 7f521159a700 1 -- 172.21.15.26:0/2288504372 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f520400b8d0 con 0x7f52140670c0 2022-07-20T03:36:43.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- 172.21.15.26:0/2288504372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 msgr2=0x7f52140b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- 172.21.15.26:0/2288504372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52140b2e90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f52040032f0 tx=0x7f52040091d0).stop 2022-07-20T03:36:43.983 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- 172.21.15.26:0/2288504372 shutdown_connections 2022-07-20T03:36:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.26:3300/0 conn(0x7f52140b3380 0x7f5214065440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- 172.21.15.26:0/2288504372 >> v2:172.21.15.120:3300/0 conn(0x7f5214064450 0x7f5214064f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.984 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- 172.21.15.26:0/2288504372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52140b2e90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- 172.21.15.26:0/2288504372 >> 172.21.15.26:0/2288504372 conn(0x7f521405f840 msgr2=0x7f521405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:43.985 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- 172.21.15.26:0/2288504372 shutdown_connections 2022-07-20T03:36:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- 172.21.15.26:0/2288504372 wait complete. 2022-07-20T03:36:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 Processor -- start 2022-07-20T03:36:43.986 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- start start 2022-07-20T03:36:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52141434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5214144c70 0x7f5214143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.987 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214152150 con 0x7f521413b8d0 2022-07-20T03:36:43.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140b13f0 con 0x7f5214144c70 2022-07-20T03:36:43.988 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521359e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52141441a0 con 0x7f52140670c0 2022-07-20T03:36:43.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f5212d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f5212d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44952/0 (socket says 172.21.15.26:44952) 2022-07-20T03:36:43.989 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f5211d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5214144c70 0x7f5214143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f5212d9d700 1 -- 172.21.15.26:0/2277545832 learned_addr learned my addr 172.21.15.26:0/2277545832 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.966+0000 7f521259c700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52141434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 -- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 msgr2=0x7f52141434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52141434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.991 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 -- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5214144c70 msgr2=0x7f5214143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:43.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5214144c70 0x7f5214143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:43.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 -- 172.21.15.26:0/2277545832 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c00b1e0 con 0x7f521413b8d0 2022-07-20T03:36:43.992 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f5212d9d700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f5214144330 tx=0x7f520c006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f520c012600 con 0x7f521413b8d0 2022-07-20T03:36:43.993 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c012760 con 0x7f521413b8d0 2022-07-20T03:36:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f520c012ac0 con 0x7f521413b8d0 2022-07-20T03:36:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5204009470 con 0x7f521413b8d0 2022-07-20T03:36:43.994 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52140b18f0 con 0x7f521413b8d0 2022-07-20T03:36:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214064ab0 con 0x7f521413b8d0 2022-07-20T03:36:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f520c022070 con 0x7f521413b8d0 2022-07-20T03:36:43.995 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 0x7f51ec055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.970+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f520c06c190 con 0x7f521413b8d0 2022-07-20T03:36:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.974+0000 7f521259c700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 0x7f51ec055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:43.996 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.974+0000 7f521259c700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 0x7f51ec055e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f520401b000 tx=0x7f5204016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:43.997 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:43.978+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f520c043230 con 0x7f521413b8d0 2022-07-20T03:36:44.104 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.102+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f52140b1af0 con 0x7f51ec053980 2022-07-20T03:36:44.105 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.102+0000 7f520b7fe700 1 -- 172.21.15.26:0/2277545832 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f520400b8d0 con 0x7f51ec053980 2022-07-20T03:36:44.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 msgr2=0x7f51ec055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 0x7f51ec055e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f520401b000 tx=0x7f5204016000).stop 2022-07-20T03:36:44.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 msgr2=0x7f521413bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f5214144330 tx=0x7f520c006eb0).stop 2022-07-20T03:36:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 shutdown_connections 2022-07-20T03:36:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f521413b8d0 0x7f521413bd50 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f51ec053980 0x7f51ec055e40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5214144c70 0x7f5214143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 --2- 172.21.15.26:0/2277545832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52140670c0 0x7f52141434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 >> 172.21.15.26:0/2277545832 conn(0x7f521405f840 msgr2=0x7f521405cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.114 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 shutdown_connections 2022-07-20T03:36:44.115 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.106+0000 7f521359e700 1 -- 172.21.15.26:0/2277545832 wait complete. 2022-07-20T03:36:44.115 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2022-07-20T03:36:44.118 INFO:tasks.ceph:Scrubbing osd.2 2022-07-20T03:36:44.120 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.230+0000 7f07e2268700 1 Processor -- start 2022-07-20T03:36:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.230+0000 7f07e2268700 1 -- start start 2022-07-20T03:36:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e2268700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07dc0b06e0 0x7f07dc14a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e2268700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07dc0aa1e0 con 0x7f07dc0add30 2022-07-20T03:36:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e2268700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07dc0ae110 con 0x7f07dc0b2ab0 2022-07-20T03:36:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e2268700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07dc0a9f40 con 0x7f07dc0b06e0 2022-07-20T03:36:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1266700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f07dc0add30 0x7f07dc060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44236/0 (socket says 172.21.15.26:44236) 2022-07-20T03:36:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1266700 1 -- 172.21.15.26:0/3721531663 learned_addr learned my addr 172.21.15.26:0/3721531663 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> v2:172.21.15.120:3300/0 conn(0x7f07dc0b06e0 0x7f07dc14a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169251288 0 0) 0x7f07dc0aa1e0 con 0x7f07dc0add30 2022-07-20T03:36:44.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c0003260 con 0x7f07dc0add30 2022-07-20T03:36:44.259 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231342354 0 0) 0x7f07dc0ae110 con 0x7f07dc0b2ab0 2022-07-20T03:36:44.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc0aa1e0 con 0x7f07dc0b2ab0 2022-07-20T03:36:44.260 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254639541 0 0) 0x7f07c0003260 con 0x7f07dc0add30 2022-07-20T03:36:44.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 >> v2:172.21.15.120:3300/0 conn(0x7f07dc0b06e0 msgr2=0x7f07dc14a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.261 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 --2- 172.21.15.26:0/3721531663 >> v2:172.21.15.120:3300/0 conn(0x7f07dc0b06e0 0x7f07dc14a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 >> v1:172.21.15.120:6789/0 conn(0x7f07dc0b2ab0 legacy=0x7f07dc061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.262 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc0b1110 con 0x7f07dc0add30 2022-07-20T03:36:44.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f07d8001660 con 0x7f07dc0add30 2022-07-20T03:36:44.263 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 >> v1:172.21.15.26:6789/0 conn(0x7f07dc0add30 legacy=0x7f07dc060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07c00038e0 0x7f07c0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07c0004500 0x7f07c0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.264 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07c000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc0a9f40 con 0x7f07c0004e70 2022-07-20T03:36:44.265 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c000bcf0 con 0x7f07c00038e0 2022-07-20T03:36:44.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c000beb0 con 0x7f07c0004500 2022-07-20T03:36:44.266 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> v2:172.21.15.120:3300/0 conn(0x7f07dc0b06e0 0x7f07dc14a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:44.267 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07c000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1266700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07c00038e0 0x7f07c0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.268 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e0a65700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07c0004500 0x7f07c0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.269 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 -- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07c0004500 msgr2=0x7f07c0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07c0004500 0x7f07c0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.270 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 -- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07c00038e0 msgr2=0x7f07c0003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.271 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07c00038e0 0x7f07c0003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 -- 172.21.15.26:0/3721531663 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0003260 con 0x7f07c0004e70 2022-07-20T03:36:44.272 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 -- 172.21.15.26:0/3721531663 reap_dead start 2022-07-20T03:36:44.273 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07e1a67700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07c000b750 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f07d400c000 tx=0x7f07d40091d0).ready entity=mon.0 client_cookie=cc7bd943fbfdfbe5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.274 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d400e8f0 con 0x7f07c0004e70 2022-07-20T03:36:44.276 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d400ea50 con 0x7f07c0004e70 2022-07-20T03:36:44.277 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.234+0000 7f07cffff700 1 -- 172.21.15.26:0/3721531663 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d400edb0 con 0x7f07c0004e70 2022-07-20T03:36:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 msgr2=0x7f07c000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07c000b750 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f07d400c000 tx=0x7f07d40091d0).stop 2022-07-20T03:36:44.279 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- 172.21.15.26:0/3721531663 shutdown_connections 2022-07-20T03:36:44.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 --2- 172.21.15.26:0/3721531663 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07c000b750 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.280 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- 172.21.15.26:0/3721531663 >> 172.21.15.26:0/3721531663 conn(0x7f07dc0b3050 msgr2=0x7f07dc0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- 172.21.15.26:0/3721531663 shutdown_connections 2022-07-20T03:36:44.281 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- 172.21.15.26:0/3721531663 wait complete. 2022-07-20T03:36:44.282 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 Processor -- start 2022-07-20T03:36:44.283 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- start start 2022-07-20T03:36:44.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07dc0b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.284 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07dc1400f0 0x7f07dc0aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.285 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07dc13f9d0 0x7f07dc0aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.286 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc14fff0 con 0x7f07c0004e70 2022-07-20T03:36:44.287 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc150290 con 0x7f07dc13f9d0 2022-07-20T03:36:44.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e2268700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc150530 con 0x7f07dc1400f0 2022-07-20T03:36:44.288 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e1266700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07dc0b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.289 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e1266700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07dc0b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44968/0 (socket says 172.21.15.26:44968) 2022-07-20T03:36:44.290 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.238+0000 7f07e1266700 1 -- 172.21.15.26:0/2652765590 learned_addr learned my addr 172.21.15.26:0/2652765590 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:44.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e0a65700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07dc1400f0 0x7f07dc0aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.291 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1a67700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07dc13f9d0 0x7f07dc0aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 -- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07dc1400f0 msgr2=0x7f07dc0aa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07dc1400f0 0x7f07dc0aa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.293 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 -- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07dc13f9d0 msgr2=0x7f07dc0aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07dc13f9d0 0x7f07dc0aa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.294 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc1507d0 con 0x7f07c0004e70 2022-07-20T03:36:44.295 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1a67700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07dc13f9d0 0x7f07dc0aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:36:44.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1266700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07dc0b5a80 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f07d8001040 tx=0x7f07dc0af5f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.296 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d8011780 con 0x7f07c0004e70 2022-07-20T03:36:44.297 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d80118e0 con 0x7f07c0004e70 2022-07-20T03:36:44.298 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc13f5d0 con 0x7f07c0004e70 2022-07-20T03:36:44.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d8011c40 con 0x7f07c0004e70 2022-07-20T03:36:44.299 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc150e90 con 0x7f07c0004e70 2022-07-20T03:36:44.300 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07dc150960 con 0x7f07c0004e70 2022-07-20T03:36:44.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f07d801b030 con 0x7f07c0004e70 2022-07-20T03:36:44.301 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f07c4053950 0x7f07c4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.302 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f07d806b930 con 0x7f07c0004e70 2022-07-20T03:36:44.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f07c405a680 0x7f07c405cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.303 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e0a65700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f07c4053950 0x7f07c4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.304 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07c405d1a0 con 0x7f07c405a680 2022-07-20T03:36:44.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07d8070240 con 0x7f07c0004e70 2022-07-20T03:36:44.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e1a67700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f07c405a680 0x7f07c405cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.242+0000 7f07e0a65700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f07c4053950 0x7f07c4055e10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f07d0003000 tx=0x7f07d0009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.246+0000 7f07e1a67700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f07c405a680 0x7f07c405cb20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.246+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== osd.2 v2:172.21.15.26:6818/23051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f07c405d1a0 con 0x7f07c405a680 2022-07-20T03:36:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 --> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f07dc05c0e0 con 0x7f07c405a680 2022-07-20T03:36:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07ce7fc700 1 -- 172.21.15.26:0/2652765590 <== osd.2 v2:172.21.15.26:6818/23051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f07dc05c0e0 con 0x7f07c405a680 2022-07-20T03:36:44.311 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:44.312 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:44.313 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f07c405a680 msgr2=0x7f07c405cb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6818/23051,v1:172.21.15.26:6819/23051] conn(0x7f07c405a680 0x7f07c405cb20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f07c4053950 msgr2=0x7f07c4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f07c4053950 0x7f07c4055e10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f07d0003000 tx=0x7f07d0009040).stop 2022-07-20T03:36:44.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 msgr2=0x7f07dc0b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 --2- 172.21.15.26:0/2652765590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c0004e70 0x7f07dc0b5a80 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f07d8001040 tx=0x7f07dc0af5f0).stop 2022-07-20T03:36:44.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e1a67700 1 -- 172.21.15.26:0/2652765590 reap_dead start 2022-07-20T03:36:44.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 shutdown_connections 2022-07-20T03:36:44.321 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 >> 172.21.15.26:0/2652765590 conn(0x7f07dc0b3050 msgr2=0x7f07dc05c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 shutdown_connections 2022-07-20T03:36:44.322 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.262+0000 7f07e2268700 1 -- 172.21.15.26:0/2652765590 wait complete. 2022-07-20T03:36:44.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2022-07-20T03:36:44.428 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 Processor -- start 2022-07-20T03:36:44.429 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 -- start start 2022-07-20T03:36:44.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.432 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efed4067310 0x7efed4143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40610e0 con 0x7efed40b2ac0 2022-07-20T03:36:44.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.426+0000 7efed9cb5700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40b2ea0 con 0x7efed4067310 2022-07-20T03:36:44.433 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40a8e20 con 0x7efed4060850 2022-07-20T03:36:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efed4067310 0x7efed4143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed2ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed37fe700 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7efed40b2ac0 0x7efed40ab880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:58470/0 (socket says 172.21.15.26:58470) 2022-07-20T03:36:44.436 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efed4067310 0x7efed4143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:36442/0 (socket says 172.21.15.26:36442) 2022-07-20T03:36:44.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed37fe700 1 -- 172.21.15.26:0/3137149357 learned_addr learned my addr 172.21.15.26:0/3137149357 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:44.437 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed2ffd700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:38264/0 (socket says 172.21.15.26:38264) 2022-07-20T03:36:44.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 -- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 msgr2=0x7efed40abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.438 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.439 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 -- 172.21.15.26:0/3137149357 >> v1:172.21.15.120:6789/0 conn(0x7efed40b2ac0 legacy=0x7efed40ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 -- 172.21.15.26:0/3137149357 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40a95e0 con 0x7efed4067310 2022-07-20T03:36:44.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed2ffd700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:44.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed3fff700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed4143480 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7efecc002f70 tx=0x7efecc006df0).ready entity=mon.1 client_cookie=dd19ad2c8d0d894b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed27fc700 1 -- 172.21.15.26:0/3137149357 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc00d570 con 0x7efed4067310 2022-07-20T03:36:44.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed27fc700 1 -- 172.21.15.26:0/3137149357 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc000e90 con 0x7efed4067310 2022-07-20T03:36:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed27fc700 1 -- 172.21.15.26:0/3137149357 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc00d8c0 con 0x7efed4067310 2022-07-20T03:36:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 msgr2=0x7efed4143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed4143480 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7efecc002f70 tx=0x7efecc006df0).stop 2022-07-20T03:36:44.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- 172.21.15.26:0/3137149357 shutdown_connections 2022-07-20T03:36:44.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed4143480 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 --2- 172.21.15.26:0/3137149357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed40abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- 172.21.15.26:0/3137149357 >> 172.21.15.26:0/3137149357 conn(0x7efed40b60a0 msgr2=0x7efed40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- 172.21.15.26:0/3137149357 shutdown_connections 2022-07-20T03:36:44.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.430+0000 7efed9cb5700 1 -- 172.21.15.26:0/3137149357 wait complete. 2022-07-20T03:36:44.461 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 Processor -- start 2022-07-20T03:36:44.462 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- start start 2022-07-20T03:36:44.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed4143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.463 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed413e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.465 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40af320 con 0x7efed40b2ac0 2022-07-20T03:36:44.466 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed4142a00 con 0x7efed4067310 2022-07-20T03:36:44.467 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40af080 con 0x7efed4060850 2022-07-20T03:36:44.468 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44988/0 (socket says 172.21.15.26:44988) 2022-07-20T03:36:44.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 -- 172.21.15.26:0/721224843 learned_addr learned my addr 172.21.15.26:0/721224843 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:44.470 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed37fe700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed4143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed2ffd700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed413e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.471 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 -- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 msgr2=0x7efed4143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed4143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.472 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 -- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 msgr2=0x7efed413e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed413e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 -- 172.21.15.26:0/721224843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4009470 con 0x7efed40b2ac0 2022-07-20T03:36:44.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed3fff700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7efecc002f70 tx=0x7efecc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.474 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc016070 con 0x7efed40b2ac0 2022-07-20T03:36:44.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc00d8f0 con 0x7efed40b2ac0 2022-07-20T03:36:44.476 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc00b180 con 0x7efed40b2ac0 2022-07-20T03:36:44.477 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc013de0 con 0x7efed40b2ac0 2022-07-20T03:36:44.478 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.434+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed413f1e0 con 0x7efed40b2ac0 2022-07-20T03:36:44.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed40ac480 con 0x7efed40b2ac0 2022-07-20T03:36:44.479 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efecc025030 con 0x7efed40b2ac0 2022-07-20T03:36:44.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed0ff9700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 0x7efeb4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.480 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed37fe700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 0x7efeb4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.481 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7efecc06cdb0 con 0x7efed40b2ac0 2022-07-20T03:36:44.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.438+0000 7efed37fe700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 0x7efeb4055f30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7efec800d000 tx=0x7efec8000bc0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.482 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.442+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efecc071070 con 0x7efed40b2ac0 2022-07-20T03:36:44.570 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.566+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7efed4142880 con 0x7efeb4053a70 2022-07-20T03:36:44.572 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed0ff9700 1 -- 172.21.15.26:0/721224843 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efec8015410 con 0x7efeb4053a70 2022-07-20T03:36:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 msgr2=0x7efeb4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.588 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 0x7efeb4055f30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7efec800d000 tx=0x7efec8000bc0).stop 2022-07-20T03:36:44.589 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 msgr2=0x7efed413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7efecc002f70 tx=0x7efecc014000).stop 2022-07-20T03:36:44.590 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 shutdown_connections 2022-07-20T03:36:44.591 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efed40b2ac0 0x7efed413ecf0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7efeb4053a70 0x7efeb4055f30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.592 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed4067310 0x7efed413e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 --2- 172.21.15.26:0/721224843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed4060850 0x7efed4143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.593 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 >> 172.21.15.26:0/721224843 conn(0x7efed40b60a0 msgr2=0x7efed40aac40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 shutdown_connections 2022-07-20T03:36:44.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.570+0000 7efed9cb5700 1 -- 172.21.15.26:0/721224843 wait complete. 2022-07-20T03:36:44.595 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2022-07-20T03:36:44.596 INFO:tasks.ceph:Scrubbing osd.3 2022-07-20T03:36:44.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:44.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 Processor -- start 2022-07-20T03:36:44.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 -- start start 2022-07-20T03:36:44.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4b0005cfd0 0x7f4b00143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b00060f00 0x7f4b0013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b001474e0 con 0x7f4b00147000 2022-07-20T03:36:44.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b000b4e40 con 0x7f4b00060f00 2022-07-20T03:36:44.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afffff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0005d3b0 con 0x7f4b0005cfd0 2022-07-20T03:36:44.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4aff7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b00060f00 0x7f4b0013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4aff7fe700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4b00060f00 0x7f4b0013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44996/0 (socket says 172.21.15.26:44996) 2022-07-20T03:36:44.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4aff7fe700 1 -- 172.21.15.26:0/491311005 learned_addr learned my addr 172.21.15.26:0/491311005 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:44.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.714+0000 7f4afeffd700 1 --2- 172.21.15.26:0/491311005 >> v2:172.21.15.120:3300/0 conn(0x7f4b0005cfd0 0x7f4b00143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4aff7fe700 1 -- 172.21.15.26:0/491311005 >> v2:172.21.15.120:3300/0 conn(0x7f4b0005cfd0 msgr2=0x7f4b00143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4aff7fe700 1 --2- 172.21.15.26:0/491311005 >> v2:172.21.15.120:3300/0 conn(0x7f4b0005cfd0 0x7f4b00143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4aff7fe700 1 -- 172.21.15.26:0/491311005 >> v1:172.21.15.120:6789/0 conn(0x7f4b00147000 legacy=0x7f4b000b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.738 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4aff7fe700 1 -- 172.21.15.26:0/491311005 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b001431e0 con 0x7f4b00060f00 2022-07-20T03:36:44.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4aff7fe700 1 --2- 172.21.15.26:0/491311005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b00060f00 0x7f4b0013c9b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f4af80032f0 tx=0x7f4af8009290).ready entity=mon.0 client_cookie=2ac3b4c10211b1f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afdffb700 1 -- 172.21.15.26:0/491311005 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af800b870 con 0x7f4b00060f00 2022-07-20T03:36:44.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afdffb700 1 -- 172.21.15.26:0/491311005 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af800b9d0 con 0x7f4b00060f00 2022-07-20T03:36:44.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afdffb700 1 -- 172.21.15.26:0/491311005 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af800bb40 con 0x7f4b00060f00 2022-07-20T03:36:44.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- 172.21.15.26:0/491311005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b00060f00 msgr2=0x7f4b0013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.741 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 --2- 172.21.15.26:0/491311005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b00060f00 0x7f4b0013c9b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f4af80032f0 tx=0x7f4af8009290).stop 2022-07-20T03:36:44.742 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- 172.21.15.26:0/491311005 shutdown_connections 2022-07-20T03:36:44.743 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 --2- 172.21.15.26:0/491311005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b00060f00 0x7f4b0013c9b0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 --2- 172.21.15.26:0/491311005 >> v2:172.21.15.120:3300/0 conn(0x7f4b0005cfd0 0x7f4b00143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.744 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- 172.21.15.26:0/491311005 >> 172.21.15.26:0/491311005 conn(0x7f4b000620c0 msgr2=0x7f4b0005c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.745 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- 172.21.15.26:0/491311005 shutdown_connections 2022-07-20T03:36:44.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- 172.21.15.26:0/491311005 wait complete. 2022-07-20T03:36:44.746 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 Processor -- start 2022-07-20T03:36:44.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.718+0000 7f4afffff700 1 -- start start 2022-07-20T03:36:44.747 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b00060f00 0x7f4b000b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.748 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b00147000 0x7f4b000adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.749 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 0x7f4b000ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.750 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b001429e0 con 0x7f4b000b0420 2022-07-20T03:36:44.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00153200 con 0x7f4b00060f00 2022-07-20T03:36:44.751 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00142c10 con 0x7f4b00147000 2022-07-20T03:36:44.752 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 0x7f4b000ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 0x7f4b000ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45002/0 (socket says 172.21.15.26:45002) 2022-07-20T03:36:44.753 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 -- 172.21.15.26:0/3596900595 learned_addr learned my addr 172.21.15.26:0/3596900595 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:44.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afe7fc700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b00147000 0x7f4b000adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.754 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afeffd700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b00060f00 0x7f4b000b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.755 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 -- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b00147000 msgr2=0x7f4b000adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b00147000 0x7f4b000adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.756 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 -- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b00060f00 msgr2=0x7f4b000b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.757 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b00060f00 0x7f4b000b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af8009530 con 0x7f4b000b0420 2022-07-20T03:36:44.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afeffd700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b00060f00 0x7f4b000b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:36:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aff7fe700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 0x7f4b000ae460 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f4af80032f0 tx=0x7f4af8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af800b870 con 0x7f4b000b0420 2022-07-20T03:36:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00142eb0 con 0x7f4b000b0420 2022-07-20T03:36:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8021630 con 0x7f4b000b0420 2022-07-20T03:36:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b000ae950 con 0x7f4b000b0420 2022-07-20T03:36:44.763 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af8021990 con 0x7f4b000b0420 2022-07-20T03:36:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.722+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b00142700 con 0x7f4b000b0420 2022-07-20T03:36:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4af800f070 con 0x7f4b000b0420 2022-07-20T03:36:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4adc053a70 0x7f4adc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f4af806d3f0 con 0x7f4b000b0420 2022-07-20T03:36:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f4adc05a700 0x7f4adc05cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4afeffd700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4adc053a70 0x7f4adc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc05d180 con 0x7f4adc05a700 2022-07-20T03:36:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4af806d7b0 con 0x7f4b000b0420 2022-07-20T03:36:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4afe7fc700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f4adc05a700 0x7f4adc05cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4afeffd700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4adc053a70 0x7f4adc055f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f4af400b000 tx=0x7f4af4008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4afe7fc700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f4adc05a700 0x7f4adc05cba0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.726+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== osd.3 v2:172.21.15.26:6826/23052 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f4adc05d180 con 0x7f4adc05a700 2022-07-20T03:36:44.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 --> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4b00062300 con 0x7f4adc05a700 2022-07-20T03:36:44.770 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4aeffff700 1 -- 172.21.15.26:0/3596900595 <== osd.3 v2:172.21.15.26:6826/23052 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4b00062300 con 0x7f4adc05a700 2022-07-20T03:36:44.771 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:44.771 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:44.772 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:44.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f4adc05a700 msgr2=0x7f4adc05cba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6826/23052,v1:172.21.15.26:6827/23052] conn(0x7f4adc05a700 0x7f4adc05cba0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.774 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4adc053a70 msgr2=0x7f4adc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4adc053a70 0x7f4adc055f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f4af400b000 tx=0x7f4af4008040).stop 2022-07-20T03:36:44.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 msgr2=0x7f4b000ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.775 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 --2- 172.21.15.26:0/3596900595 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4b000b0420 0x7f4b000ae460 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f4af80032f0 tx=0x7f4af8016000).stop 2022-07-20T03:36:44.776 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4aff7fe700 1 -- 172.21.15.26:0/3596900595 reap_dead start 2022-07-20T03:36:44.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 shutdown_connections 2022-07-20T03:36:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 >> 172.21.15.26:0/3596900595 conn(0x7f4b0005cfd0 msgr2=0x7f4b000b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 shutdown_connections 2022-07-20T03:36:44.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.746+0000 7f4afffff700 1 -- 172.21.15.26:0/3596900595 wait complete. 2022-07-20T03:36:44.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2022-07-20T03:36:44.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 Processor -- start 2022-07-20T03:36:44.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 -- start start 2022-07-20T03:36:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0de80670c0 0x7f0de80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80b3380 0x7f0de814b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de8064880 con 0x7f0de8064450 2022-07-20T03:36:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de80b85f0 con 0x7f0de80670c0 2022-07-20T03:36:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b80b0 con 0x7f0de80b3380 2022-07-20T03:36:44.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de5d9b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0de8064450 0x7f0de8064f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44292/0 (socket says 172.21.15.26:44292) 2022-07-20T03:36:44.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de5d9b700 1 -- 172.21.15.26:0/4233735892 learned_addr learned my addr 172.21.15.26:0/4233735892 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:44.914 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de659c700 1 --2- 172.21.15.26:0/4233735892 >> v2:172.21.15.120:3300/0 conn(0x7f0de80670c0 0x7f0de80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.906+0000 7f0de6d9d700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80b3380 0x7f0de814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.915 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de559a700 1 -- 172.21.15.26:0/4233735892 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864096406 0 0) 0x7f0de8064880 con 0x7f0de8064450 2022-07-20T03:36:44.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de559a700 1 -- 172.21.15.26:0/4233735892 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dcc003260 con 0x7f0de8064450 2022-07-20T03:36:44.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de659c700 1 -- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80b3380 msgr2=0x7f0de814b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:44.916 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de659c700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80b3380 0x7f0de814b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de659c700 1 -- 172.21.15.26:0/4233735892 >> v1:172.21.15.26:6789/0 conn(0x7f0de8064450 legacy=0x7f0de8064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de659c700 1 -- 172.21.15.26:0/4233735892 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de80b8350 con 0x7f0de80670c0 2022-07-20T03:36:44.917 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de659c700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de80b2e90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f0dd80032f0 tx=0x7f0dd80091d0).ready entity=mon.1 client_cookie=2389e39dd5560bc7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de559a700 1 -- 172.21.15.26:0/4233735892 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd800b600 con 0x7f0de80670c0 2022-07-20T03:36:44.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de559a700 1 -- 172.21.15.26:0/4233735892 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800b760 con 0x7f0de80670c0 2022-07-20T03:36:44.918 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de559a700 1 -- 172.21.15.26:0/4233735892 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd800b8d0 con 0x7f0de80670c0 2022-07-20T03:36:44.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 msgr2=0x7f0de80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de80b2e90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f0dd80032f0 tx=0x7f0dd80091d0).stop 2022-07-20T03:36:44.919 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- 172.21.15.26:0/4233735892 shutdown_connections 2022-07-20T03:36:44.920 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80b3380 0x7f0de814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 --2- 172.21.15.26:0/4233735892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de80b2e90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- 172.21.15.26:0/4233735892 >> 172.21.15.26:0/4233735892 conn(0x7f0de805f840 msgr2=0x7f0de805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:44.921 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- 172.21.15.26:0/4233735892 shutdown_connections 2022-07-20T03:36:44.922 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- 172.21.15.26:0/4233735892 wait complete. 2022-07-20T03:36:44.926 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 Processor -- start 2022-07-20T03:36:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.910+0000 7f0de759e700 1 -- start start 2022-07-20T03:36:44.927 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de8142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80ac970 0x7f0de80aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.928 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b1490 con 0x7f0de80b1860 2022-07-20T03:36:44.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b0480 con 0x7f0de80670c0 2022-07-20T03:36:44.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b11f0 con 0x7f0de80ac970 2022-07-20T03:36:44.929 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45016/0 (socket says 172.21.15.26:45016) 2022-07-20T03:36:44.930 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 -- 172.21.15.26:0/921821463 learned_addr learned my addr 172.21.15.26:0/921821463 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:44.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de659c700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de8142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de6d9d700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80ac970 0x7f0de80aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.931 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 -- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80ac970 msgr2=0x7f0de80aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80ac970 0x7f0de80aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.932 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 -- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 msgr2=0x7f0de8142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:44.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de8142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:44.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 -- 172.21.15.26:0/921821463 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8009470 con 0x7f0de80b1860 2022-07-20T03:36:44.933 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de5d9b700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f0dd0001110 tx=0x7f0dd0007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.934 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd00074d0 con 0x7f0de80b1860 2022-07-20T03:36:44.935 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de8150a00 con 0x7f0de80b1860 2022-07-20T03:36:44.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd0005610 con 0x7f0de80b1860 2022-07-20T03:36:44.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd0005970 con 0x7f0de80b1860 2022-07-20T03:36:44.936 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de814da20 con 0x7f0de80b1860 2022-07-20T03:36:44.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.914+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de8064ab0 con 0x7f0de80b1860 2022-07-20T03:36:44.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.918+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0dd0021070 con 0x7f0de80b1860 2022-07-20T03:36:44.937 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.918+0000 7f0ddf7fe700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 0x7f0dc0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:44.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.918+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0dd006fc90 con 0x7f0de80b1860 2022-07-20T03:36:44.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.918+0000 7f0de659c700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 0x7f0dc0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:44.938 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.918+0000 7f0de659c700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 0x7f0dc0055e10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f0dd8019000 tx=0x7f0dd8016000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:44.939 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:44.922+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0dd0046c90 con 0x7f0de80b1860 2022-07-20T03:36:45.051 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0de80574e0 con 0x7f0dc0053950 2022-07-20T03:36:45.052 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0ddf7fe700 1 -- 172.21.15.26:0/921821463 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0de80574e0 con 0x7f0dc0053950 2022-07-20T03:36:45.057 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 msgr2=0x7f0dc0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 0x7f0dc0055e10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f0dd8019000 tx=0x7f0dd8016000).stop 2022-07-20T03:36:45.058 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 msgr2=0x7f0de80ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f0dd0001110 tx=0x7f0dd0007b40).stop 2022-07-20T03:36:45.059 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 shutdown_connections 2022-07-20T03:36:45.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0de80b1860 0x7f0de80ac480 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f0dc0053950 0x7f0dc0055e10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.060 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0de80ac970 0x7f0de80aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 --2- 172.21.15.26:0/921821463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0de80670c0 0x7f0de8142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 >> 172.21.15.26:0/921821463 conn(0x7f0de805f840 msgr2=0x7f0de813d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.061 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 shutdown_connections 2022-07-20T03:36:45.062 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.050+0000 7f0de759e700 1 -- 172.21.15.26:0/921821463 wait complete. 2022-07-20T03:36:45.062 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 3 to deep-scrub 2022-07-20T03:36:45.066 INFO:tasks.ceph:Scrubbing osd.4 2022-07-20T03:36:45.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:45.192 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.178+0000 7f4f65d1d700 1 Processor -- start 2022-07-20T03:36:45.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.178+0000 7f4f65d1d700 1 -- start start 2022-07-20T03:36:45.193 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4f600b2ab0 0x7f4f60061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f600af6d0 0x7f4f600afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.194 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f600aa1e0 con 0x7f4f600add30 2022-07-20T03:36:45.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f600ae110 con 0x7f4f600af6d0 2022-07-20T03:36:45.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f600a9f40 con 0x7f4f600b2ab0 2022-07-20T03:36:45.195 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f600af6d0 0x7f4f600afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f4f600af6d0 0x7f4f600afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45026/0 (socket says 172.21.15.26:45026) 2022-07-20T03:36:45.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f64d1b700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4f600add30 0x7f4f60060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44302/0 (socket says 172.21.15.26:44302) 2022-07-20T03:36:45.196 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 -- 172.21.15.26:0/1840962282 learned_addr learned my addr 172.21.15.26:0/1840962282 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f5ffff700 1 --2- 172.21.15.26:0/1840962282 >> v2:172.21.15.120:3300/0 conn(0x7f4f600b2ab0 0x7f4f60061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.197 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 -- 172.21.15.26:0/1840962282 >> v2:172.21.15.120:3300/0 conn(0x7f4f600b2ab0 msgr2=0x7f4f60061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:45.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 --2- 172.21.15.26:0/1840962282 >> v2:172.21.15.120:3300/0 conn(0x7f4f600b2ab0 0x7f4f60061060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 -- 172.21.15.26:0/1840962282 >> v1:172.21.15.26:6789/0 conn(0x7f4f600add30 legacy=0x7f4f60060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.198 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 -- 172.21.15.26:0/1840962282 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f600b1110 con 0x7f4f600af6d0 2022-07-20T03:36:45.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f6551c700 1 --2- 172.21.15.26:0/1840962282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600af6d0 0x7f4f600afb60 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7f4f580032f0 tx=0x7f4f58009290).ready entity=mon.0 client_cookie=43743419764167a9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.199 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f5f7fe700 1 -- 172.21.15.26:0/1840962282 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f5800b870 con 0x7f4f600af6d0 2022-07-20T03:36:45.200 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f5f7fe700 1 -- 172.21.15.26:0/1840962282 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5800b9d0 con 0x7f4f600af6d0 2022-07-20T03:36:45.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f5f7fe700 1 -- 172.21.15.26:0/1840962282 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f5800bb40 con 0x7f4f600af6d0 2022-07-20T03:36:45.201 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- 172.21.15.26:0/1840962282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600af6d0 msgr2=0x7f4f600afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/1840962282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600af6d0 0x7f4f600afb60 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7f4f580032f0 tx=0x7f4f58009290).stop 2022-07-20T03:36:45.202 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- 172.21.15.26:0/1840962282 shutdown_connections 2022-07-20T03:36:45.203 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/1840962282 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600af6d0 0x7f4f600afb60 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/1840962282 >> v2:172.21.15.120:3300/0 conn(0x7f4f600b2ab0 0x7f4f60061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.204 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- 172.21.15.26:0/1840962282 >> 172.21.15.26:0/1840962282 conn(0x7f4f600b3050 msgr2=0x7f4f600b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.205 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- 172.21.15.26:0/1840962282 shutdown_connections 2022-07-20T03:36:45.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.182+0000 7f4f65d1d700 1 -- 172.21.15.26:0/1840962282 wait complete. 2022-07-20T03:36:45.206 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 Processor -- start 2022-07-20T03:36:45.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- start start 2022-07-20T03:36:45.207 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 0x7f4f600b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.208 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f600af6d0 0x7f4f601371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f600b2ab0 0x7f4f60065020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60065860 con 0x7f4f600add30 2022-07-20T03:36:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60065ab0 con 0x7f4f600af6d0 2022-07-20T03:36:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60136480 con 0x7f4f600b2ab0 2022-07-20T03:36:45.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 0x7f4f600b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.211 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 0x7f4f600b5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45028/0 (socket says 172.21.15.26:45028) 2022-07-20T03:36:45.212 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f5ffff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f600af6d0 0x7f4f601371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.213 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 -- 172.21.15.26:0/2088852312 learned_addr learned my addr 172.21.15.26:0/2088852312 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f6551c700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f600b2ab0 0x7f4f60065020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.214 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 -- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f600b2ab0 msgr2=0x7f4f60065020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.215 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f600b2ab0 0x7f4f60065020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 -- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f600af6d0 msgr2=0x7f4f601371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.216 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f600af6d0 0x7f4f601371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58009530 con 0x7f4f600add30 2022-07-20T03:36:45.217 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f64d1b700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 0x7f4f600b5760 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f4f600630a0 tx=0x7f4f54007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.218 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f540070d0 con 0x7f4f600add30 2022-07-20T03:36:45.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f54007230 con 0x7f4f600add30 2022-07-20T03:36:45.219 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f60136720 con 0x7f4f600add30 2022-07-20T03:36:45.220 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f54005740 con 0x7f4f600add30 2022-07-20T03:36:45.221 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.186+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f60131210 con 0x7f4f600add30 2022-07-20T03:36:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f600656b0 con 0x7f4f600add30 2022-07-20T03:36:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f54010070 con 0x7f4f600add30 2022-07-20T03:36:45.224 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f400539e0 0x7f4f40055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f4f5406b1d0 con 0x7f4f600add30 2022-07-20T03:36:45.225 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5ffff700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f400539e0 0x7f4f40055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.226 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f4f4005a670 0x7f4f4005cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f4005d190 con 0x7f4f4005a670 2022-07-20T03:36:45.227 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5406d860 con 0x7f4f600add30 2022-07-20T03:36:45.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f6551c700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f4f4005a670 0x7f4f4005cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.230 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f5ffff700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f400539e0 0x7f4f40055ea0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f4f5000c000 tx=0x7f4f50008040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.190+0000 7f4f6551c700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f4f4005a670 0x7f4f4005cb10 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.231 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.194+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== osd.4 v2:172.21.15.120:6816/23448 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f4f4005d190 con 0x7f4f4005a670 2022-07-20T03:36:45.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 --> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f60064d80 con 0x7f4f4005a670 2022-07-20T03:36:45.232 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f5dffb700 1 -- 172.21.15.26:0/2088852312 <== osd.4 v2:172.21.15.120:6816/23448 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4f5800bdb0 con 0x7f4f4005a670 2022-07-20T03:36:45.233 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:45.233 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:45.233 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:45.237 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f4f4005a670 msgr2=0x7f4f4005cb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.238 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.120:6816/23448,v1:172.21.15.120:6817/23448] conn(0x7f4f4005a670 0x7f4f4005cb10 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f400539e0 msgr2=0x7f4f40055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4f400539e0 0x7f4f40055ea0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f4f5000c000 tx=0x7f4f50008040).stop 2022-07-20T03:36:45.240 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 msgr2=0x7f4f600b5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.241 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f65d1d700 1 --2- 172.21.15.26:0/2088852312 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4f600add30 0x7f4f600b5760 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f4f600630a0 tx=0x7f4f54007c00).stop 2022-07-20T03:36:45.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.210+0000 7f4f6551c700 1 -- 172.21.15.26:0/2088852312 reap_dead start 2022-07-20T03:36:45.242 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.214+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 shutdown_connections 2022-07-20T03:36:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.214+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 >> 172.21.15.26:0/2088852312 conn(0x7f4f600b3050 msgr2=0x7f4f6013be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.214+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 shutdown_connections 2022-07-20T03:36:45.244 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.214+0000 7f4f65d1d700 1 -- 172.21.15.26:0/2088852312 wait complete. 2022-07-20T03:36:45.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-07-20T03:36:45.376 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 Processor -- start 2022-07-20T03:36:45.377 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 -- start start 2022-07-20T03:36:45.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6b180b0070 0x7f6b18060210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.379 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b18141560 0x7f6b181419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b180aa340 con 0x7f6b180b2c70 2022-07-20T03:36:45.380 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b18067270 con 0x7f6b180b0070 2022-07-20T03:36:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b1e789700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18062590 con 0x7f6b18141560 2022-07-20T03:36:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b17fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6b180b0070 0x7f6b18060210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.381 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b17fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f6b180b0070 0x7f6b18060210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45038/0 (socket says 172.21.15.26:45038) 2022-07-20T03:36:45.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.362+0000 7f6b17fff700 1 -- 172.21.15.26:0/592053570 learned_addr learned my addr 172.21.15.26:0/592053570 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1cd26700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b18141560 0x7f6b181419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.382 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b17fff700 1 -- 172.21.15.26:0/592053570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b18141560 msgr2=0x7f6b181419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b17fff700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b18141560 0x7f6b181419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b17fff700 1 -- 172.21.15.26:0/592053570 >> v1:172.21.15.120:6789/0 conn(0x7f6b180b2c70 legacy=0x7f6b18060700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b17fff700 1 -- 172.21.15.26:0/592053570 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180b0450 con 0x7f6b180b0070 2022-07-20T03:36:45.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b17fff700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b180b0070 0x7f6b18060210 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7f6b180623f0 tx=0x7f6b08006eb0).ready entity=mon.0 client_cookie=3c992b3165619244 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b16ffd700 1 -- 172.21.15.26:0/592053570 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b0800d5c0 con 0x7f6b180b0070 2022-07-20T03:36:45.384 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b16ffd700 1 -- 172.21.15.26:0/592053570 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08000e90 con 0x7f6b180b0070 2022-07-20T03:36:45.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b16ffd700 1 -- 172.21.15.26:0/592053570 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b0800d950 con 0x7f6b180b0070 2022-07-20T03:36:45.385 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 -- 172.21.15.26:0/592053570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b180b0070 msgr2=0x7f6b18060210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b180b0070 0x7f6b18060210 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7f6b180623f0 tx=0x7f6b08006eb0).stop 2022-07-20T03:36:45.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 -- 172.21.15.26:0/592053570 shutdown_connections 2022-07-20T03:36:45.386 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b18141560 0x7f6b181419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 --2- 172.21.15.26:0/592053570 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b180b0070 0x7f6b18060210 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 -- 172.21.15.26:0/592053570 >> 172.21.15.26:0/592053570 conn(0x7f6b180b66e0 msgr2=0x7f6b180b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.387 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 -- 172.21.15.26:0/592053570 shutdown_connections 2022-07-20T03:36:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 -- 172.21.15.26:0/592053570 wait complete. 2022-07-20T03:36:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.366+0000 7f6b1e789700 1 Processor -- start 2022-07-20T03:36:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- start start 2022-07-20T03:36:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b180b0070 0x7f6b18143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b180b2c70 0x7f6b18152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18156380 con 0x7f6b18141560 2022-07-20T03:36:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18156570 con 0x7f6b180b2c70 2022-07-20T03:36:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18156810 con 0x7f6b180b0070 2022-07-20T03:36:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45048/0 (socket says 172.21.15.26:45048) 2022-07-20T03:36:45.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 -- 172.21.15.26:0/440043269 learned_addr learned my addr 172.21.15.26:0/440043269 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b17fff700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b180b0070 0x7f6b18143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.393 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b177fe700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b180b2c70 0x7f6b18152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 -- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b180b0070 msgr2=0x7f6b18143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.394 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b180b0070 0x7f6b18143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 -- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b180b2c70 msgr2=0x7f6b18152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b180b2c70 0x7f6b18152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.395 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 -- 172.21.15.26:0/440043269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0800b1e0 con 0x7f6b18141560 2022-07-20T03:36:45.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1cd26700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f6b1000a000 tx=0x7f6b10008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.396 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b10011780 con 0x7f6b18141560 2022-07-20T03:36:45.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b100118e0 con 0x7f6b18141560 2022-07-20T03:36:45.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b18141a90 con 0x7f6b18141560 2022-07-20T03:36:45.397 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b10011c40 con 0x7f6b18141560 2022-07-20T03:36:45.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b180ab540 con 0x7f6b18141560 2022-07-20T03:36:45.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.370+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b18141c20 con 0x7f6b18141560 2022-07-20T03:36:45.398 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.374+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6b10012420 con 0x7f6b18141560 2022-07-20T03:36:45.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.374+0000 7f6b157fa700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 0x7f6afc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.374+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6b1006ba60 con 0x7f6b18141560 2022-07-20T03:36:45.399 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.374+0000 7f6b17fff700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 0x7f6afc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.400 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.374+0000 7f6b17fff700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 0x7f6afc055e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f6b0801c000 tx=0x7f6b08015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.401 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.378+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6b10042aa0 con 0x7f6b18141560 2022-07-20T03:36:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.506+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b181420d0 con 0x7f6afc053950 2022-07-20T03:36:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.506+0000 7f6b157fa700 1 -- 172.21.15.26:0/440043269 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b181420d0 con 0x7f6afc053950 2022-07-20T03:36:45.511 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 msgr2=0x7f6afc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 0x7f6afc055e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f6b0801c000 tx=0x7f6b08015000).stop 2022-07-20T03:36:45.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 msgr2=0x7f6b181528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.512 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f6b1000a000 tx=0x7f6b10008040).stop 2022-07-20T03:36:45.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 shutdown_connections 2022-07-20T03:36:45.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6b18141560 0x7f6b181528f0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.513 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f6afc053950 0x7f6afc055e10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b180b2c70 0x7f6b18152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 --2- 172.21.15.26:0/440043269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b180b0070 0x7f6b18143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.514 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 >> 172.21.15.26:0/440043269 conn(0x7f6b180b66e0 msgr2=0x7f6b180b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 shutdown_connections 2022-07-20T03:36:45.515 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.510+0000 7f6b1e789700 1 -- 172.21.15.26:0/440043269 wait complete. 2022-07-20T03:36:45.516 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 4 to deep-scrub 2022-07-20T03:36:45.520 INFO:tasks.ceph:Scrubbing osd.5 2022-07-20T03:36:45.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 Processor -- start 2022-07-20T03:36:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 -- start start 2022-07-20T03:36:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa974061050 0x7fa974067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740b1bc0 con 0x7fa9740b2920 2022-07-20T03:36:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa974147700 con 0x7fa9741484a0 2022-07-20T03:36:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa97a757700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97414b040 con 0x7fa974061050 2022-07-20T03:36:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa974061050 0x7fa974067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fa974061050 0x7fa974067190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45056/0 (socket says 172.21.15.26:45056) 2022-07-20T03:36:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa973fff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9740b2920 0x7fa974147d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44332/0 (socket says 172.21.15.26:44332) 2022-07-20T03:36:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 -- 172.21.15.26:0/3767497903 learned_addr learned my addr 172.21.15.26:0/3767497903 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 -- 172.21.15.26:0/3767497903 >> v1:172.21.15.120:6789/0 conn(0x7fa9741484a0 legacy=0x7fa974066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 -- 172.21.15.26:0/3767497903 >> v1:172.21.15.26:6789/0 conn(0x7fa9740b2920 legacy=0x7fa974147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 -- 172.21.15.26:0/3767497903 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974147930 con 0x7fa974061050 2022-07-20T03:36:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa972ffd700 1 -- 172.21.15.26:0/3767497903 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806981020 0 0) 0x7fa9740b1bc0 con 0x7fa9740b2920 2022-07-20T03:36:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.626+0000 7fa978cf4700 1 --2- 172.21.15.26:0/3767497903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa974061050 0x7fa974067190 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c006eb0).ready entity=mon.0 client_cookie=b425ed6e7b78f36b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa972ffd700 1 -- 172.21.15.26:0/3767497903 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c00d660 con 0x7fa974061050 2022-07-20T03:36:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa972ffd700 1 -- 172.21.15.26:0/3767497903 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c000e90 con 0x7fa974061050 2022-07-20T03:36:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa972ffd700 1 -- 172.21.15.26:0/3767497903 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c00d9b0 con 0x7fa974061050 2022-07-20T03:36:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- 172.21.15.26:0/3767497903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa974061050 msgr2=0x7fa974067190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 --2- 172.21.15.26:0/3767497903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa974061050 0x7fa974067190 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c006eb0).stop 2022-07-20T03:36:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- 172.21.15.26:0/3767497903 shutdown_connections 2022-07-20T03:36:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 --2- 172.21.15.26:0/3767497903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa974061050 0x7fa974067190 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- 172.21.15.26:0/3767497903 >> 172.21.15.26:0/3767497903 conn(0x7fa9740b60a0 msgr2=0x7fa9740b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- 172.21.15.26:0/3767497903 shutdown_connections 2022-07-20T03:36:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- 172.21.15.26:0/3767497903 wait complete. 2022-07-20T03:36:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 Processor -- start 2022-07-20T03:36:45.663 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- start start 2022-07-20T03:36:45.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974061050 0x7fa9740a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.664 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9740b2920 0x7fa9740aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 0x7fa9740af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.665 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740a9fd0 con 0x7fa9741484a0 2022-07-20T03:36:45.666 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974066740 con 0x7fa974061050 2022-07-20T03:36:45.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa97a757700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740669e0 con 0x7fa9740b2920 2022-07-20T03:36:45.667 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.630+0000 7fa978cf4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 0x7fa9740af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 0x7fa9740af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45062/0 (socket says 172.21.15.26:45062) 2022-07-20T03:36:45.668 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 -- 172.21.15.26:0/2908157577 learned_addr learned my addr 172.21.15.26:0/2908157577 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa973fff700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974061050 0x7fa9740a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.669 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa9737fe700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9740b2920 0x7fa9740aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 -- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9740b2920 msgr2=0x7fa9740aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.670 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9740b2920 0x7fa9740aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.671 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 -- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974061050 msgr2=0x7fa9740a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974061050 0x7fa9740a99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c00b1e0 con 0x7fa9741484a0 2022-07-20T03:36:45.673 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa978cf4700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 0x7fa9740af480 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c006b60 con 0x7fa9741484a0 2022-07-20T03:36:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c00d9e0 con 0x7fa9741484a0 2022-07-20T03:36:45.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa974064ee0 con 0x7fa9741484a0 2022-07-20T03:36:45.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c011d70 con 0x7fa9741484a0 2022-07-20T03:36:45.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9740641a0 con 0x7fa9741484a0 2022-07-20T03:36:45.677 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa974064d60 con 0x7fa9741484a0 2022-07-20T03:36:45.678 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.634+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa96c025030 con 0x7fa9741484a0 2022-07-20T03:36:45.679 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa954053b00 0x7fa954055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa96c06d610 con 0x7fa9741484a0 2022-07-20T03:36:45.680 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa973fff700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa954053b00 0x7fa954055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.681 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7fa95405a790 0x7fa95405cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa95405d210 con 0x7fa95405a790 2022-07-20T03:36:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa96c06d9f0 con 0x7fa9741484a0 2022-07-20T03:36:45.683 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9737fe700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7fa95405a790 0x7fa95405cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa973fff700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa954053b00 0x7fa954055fc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fa96800d000 tx=0x7fa968007a70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.684 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9737fe700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7fa95405a790 0x7fa95405cc30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.638+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== osd.5 v2:172.21.15.120:6800/23447 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fa95405d210 con 0x7fa95405a790 2022-07-20T03:36:45.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 --> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa974065b40 con 0x7fa95405a790 2022-07-20T03:36:45.687 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa9717fa700 1 -- 172.21.15.26:0/2908157577 <== osd.5 v2:172.21.15.120:6800/23447 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa974065b40 con 0x7fa95405a790 2022-07-20T03:36:45.688 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:45.689 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:45.690 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:45.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7fa95405a790 msgr2=0x7fa95405cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.120:6800/23447,v1:172.21.15.120:6801/23447] conn(0x7fa95405a790 0x7fa95405cc30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa954053b00 msgr2=0x7fa954055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.696 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fa954053b00 0x7fa954055fc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fa96800d000 tx=0x7fa968007a70).stop 2022-07-20T03:36:45.697 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 msgr2=0x7fa9740af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 --2- 172.21.15.26:0/2908157577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9741484a0 0x7fa9740af480 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c015000).stop 2022-07-20T03:36:45.698 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa978cf4700 1 -- 172.21.15.26:0/2908157577 reap_dead start 2022-07-20T03:36:45.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 shutdown_connections 2022-07-20T03:36:45.699 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 >> 172.21.15.26:0/2908157577 conn(0x7fa9740b60a0 msgr2=0x7fa9740b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.700 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 shutdown_connections 2022-07-20T03:36:45.701 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.658+0000 7fa97a757700 1 -- 172.21.15.26:0/2908157577 wait complete. 2022-07-20T03:36:45.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2022-07-20T03:36:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 Processor -- start 2022-07-20T03:36:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 -- start start 2022-07-20T03:36:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f735c0add30 0x7f735c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f735c060950 0x7f735c060dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 0x7f735c143450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c0aa1e0 con 0x7f735c060950 2022-07-20T03:36:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c0ae110 con 0x7f735c0add30 2022-07-20T03:36:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7361a6d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0a9f40 con 0x7f735c0612b0 2022-07-20T03:36:45.827 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f735bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f735c060950 0x7f735c060dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f735bfff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f735c060950 0x7f735c060dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45070/0 (socket says 172.21.15.26:45070) 2022-07-20T03:36:45.828 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f735bfff700 1 -- 172.21.15.26:0/2802656842 learned_addr learned my addr 172.21.15.26:0/2802656842 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.829 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f7360a6b700 1 --2- 172.21.15.26:0/2802656842 >> v2:172.21.15.120:3300/0 conn(0x7f735c0add30 0x7f735c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.814+0000 7f736126c700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 0x7f735c143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 -- 172.21.15.26:0/2802656842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 msgr2=0x7f735c143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:45.831 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 0x7f735c143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 -- 172.21.15.26:0/2802656842 >> v2:172.21.15.120:3300/0 conn(0x7f735c0add30 msgr2=0x7f735c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:45.832 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 --2- 172.21.15.26:0/2802656842 >> v2:172.21.15.120:3300/0 conn(0x7f735c0add30 0x7f735c0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 -- 172.21.15.26:0/2802656842 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0b1110 con 0x7f735c060950 2022-07-20T03:36:45.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f736126c700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 0x7f735c143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:36:45.833 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735bfff700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c060950 0x7f735c060dc0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f7350002f70 tx=0x7f7350006eb0).ready entity=mon.0 client_cookie=a597d2418e7d027b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735b7fe700 1 -- 172.21.15.26:0/2802656842 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f735000f970 con 0x7f735c060950 2022-07-20T03:36:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735b7fe700 1 -- 172.21.15.26:0/2802656842 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350000e90 con 0x7f735c060950 2022-07-20T03:36:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f735b7fe700 1 -- 172.21.15.26:0/2802656842 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f735000fd00 con 0x7f735c060950 2022-07-20T03:36:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 -- 172.21.15.26:0/2802656842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c060950 msgr2=0x7f735c060dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c060950 0x7f735c060dc0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f7350002f70 tx=0x7f7350006eb0).stop 2022-07-20T03:36:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 -- 172.21.15.26:0/2802656842 shutdown_connections 2022-07-20T03:36:45.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c0612b0 0x7f735c143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.837 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 --2- 172.21.15.26:0/2802656842 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c060950 0x7f735c060dc0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 --2- 172.21.15.26:0/2802656842 >> v2:172.21.15.120:3300/0 conn(0x7f735c0add30 0x7f735c0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 -- 172.21.15.26:0/2802656842 >> 172.21.15.26:0/2802656842 conn(0x7f735c0b3050 msgr2=0x7f735c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 -- 172.21.15.26:0/2802656842 shutdown_connections 2022-07-20T03:36:45.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.818+0000 7f7361a6d700 1 -- 172.21.15.26:0/2802656842 wait complete. 2022-07-20T03:36:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 Processor -- start 2022-07-20T03:36:45.848 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- start start 2022-07-20T03:36:45.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c060950 0x7f735c064590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.849 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f735c0add30 0x7f735c13b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.850 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c13d2c0 con 0x7f735c0612b0 2022-07-20T03:36:45.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c14e810 con 0x7f735c0add30 2022-07-20T03:36:45.851 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c14ea80 con 0x7f735c060950 2022-07-20T03:36:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45074/0 (socket says 172.21.15.26:45074) 2022-07-20T03:36:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 -- 172.21.15.26:0/1744373217 learned_addr learned my addr 172.21.15.26:0/1744373217 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:45.853 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f736126c700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f735c0add30 0x7f735c13b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.854 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7360a6b700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c060950 0x7f735c064590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 -- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c060950 msgr2=0x7f735c064590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c060950 0x7f735c064590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.855 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 -- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f735c0add30 msgr2=0x7f735c13b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f735c0add30 0x7f735c13b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 -- 172.21.15.26:0/1744373217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735000b1e0 con 0x7f735c0612b0 2022-07-20T03:36:45.856 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f735bfff700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f735c14e9a0 tx=0x7f7350017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.857 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7350020070 con 0x7f735c0612b0 2022-07-20T03:36:45.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350010d60 con 0x7f735c0612b0 2022-07-20T03:36:45.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c14ed20 con 0x7f735c0612b0 2022-07-20T03:36:45.858 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f735001a670 con 0x7f735c0612b0 2022-07-20T03:36:45.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.822+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c0640f0 con 0x7f735c0612b0 2022-07-20T03:36:45.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c0642b0 con 0x7f735c0612b0 2022-07-20T03:36:45.859 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73500068b0 con 0x7f735c0612b0 2022-07-20T03:36:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7359ffb700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 0x7f733c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f735006ede0 con 0x7f735c0612b0 2022-07-20T03:36:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7360a6b700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 0x7f733c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:45.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.826+0000 7f7360a6b700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 0x7f733c055e10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f735400b000 tx=0x7f73540091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:45.861 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.830+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7350045e80 con 0x7f735c0612b0 2022-07-20T03:36:45.958 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.958+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f735c0b13b0 con 0x7f733c053950 2022-07-20T03:36:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.958+0000 7f7359ffb700 1 -- 172.21.15.26:0/1744373217 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f735c0b13b0 con 0x7f733c053950 2022-07-20T03:36:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 msgr2=0x7f733c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 0x7f733c055e10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f735400b000 tx=0x7f73540091d0).stop 2022-07-20T03:36:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 msgr2=0x7f735c0af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f735c14e9a0 tx=0x7f7350017000).stop 2022-07-20T03:36:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 shutdown_connections 2022-07-20T03:36:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f735c0612b0 0x7f735c0af790 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f733c053950 0x7f733c055e10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f735c0add30 0x7f735c13b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 --2- 172.21.15.26:0/1744373217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f735c060950 0x7f735c064590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 >> 172.21.15.26:0/1744373217 conn(0x7f735c0b3050 msgr2=0x7f735c0b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 shutdown_connections 2022-07-20T03:36:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:45.962+0000 7f7361a6d700 1 -- 172.21.15.26:0/1744373217 wait complete. 2022-07-20T03:36:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 5 to deep-scrub 2022-07-20T03:36:45.978 INFO:tasks.ceph:Scrubbing osd.6 2022-07-20T03:36:45.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:46.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 Processor -- start 2022-07-20T03:36:46.108 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 -- start start 2022-07-20T03:36:46.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f605c061a20 0x7f605c1432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.109 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 0x7f605c0b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c061e00 con 0x7f605c1468c0 2022-07-20T03:36:46.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c147040 con 0x7f605c061a20 2022-07-20T03:36:46.110 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.086+0000 7f6061f2b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1472e0 con 0x7f605c12c750 2022-07-20T03:36:46.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f605c061a20 0x7f605c1432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 0x7f605c0b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.111 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f605c061a20 0x7f605c1432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.26:36542/0 (socket says 172.21.15.26:36542) 2022-07-20T03:36:46.112 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 0x7f605c0b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.26:38370/0 (socket says 172.21.15.26:38370) 2022-07-20T03:36:46.113 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 -- 172.21.15.26:0/3021757901 learned_addr learned my addr 172.21.15.26:0/3021757901 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.121 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605affd700 1 --1- 172.21.15.26:0/3021757901 >> v1:172.21.15.120:6789/0 conn(0x7f605c1468c0 0x7f605c12c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.26:58576/0 (socket says 172.21.15.26:58576) 2022-07-20T03:36:46.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 -- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 msgr2=0x7f605c0b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:46.123 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 --2- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 0x7f605c0b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.124 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 -- 172.21.15.26:0/3021757901 >> v1:172.21.15.120:6789/0 conn(0x7f605c1468c0 legacy=0x7f605c12c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.125 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 -- 172.21.15.26:0/3021757901 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c147580 con 0x7f605c061a20 2022-07-20T03:36:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605b7fe700 1 --2- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c061a20 0x7f605c1432e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f604c002f70 tx=0x7f604c006eb0).ready entity=mon.1 client_cookie=2ad287ff9c73bd94 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605a7fc700 1 -- 172.21.15.26:0/3021757901 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604c00d570 con 0x7f605c061a20 2022-07-20T03:36:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605a7fc700 1 -- 172.21.15.26:0/3021757901 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c000e90 con 0x7f605c061a20 2022-07-20T03:36:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f605a7fc700 1 -- 172.21.15.26:0/3021757901 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604c00d900 con 0x7f605c061a20 2022-07-20T03:36:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 -- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c061a20 msgr2=0x7f605c1432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 --2- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c061a20 0x7f605c1432e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f604c002f70 tx=0x7f604c006eb0).stop 2022-07-20T03:36:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 -- 172.21.15.26:0/3021757901 shutdown_connections 2022-07-20T03:36:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 --2- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c12c750 0x7f605c0b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 --2- 172.21.15.26:0/3021757901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c061a20 0x7f605c1432e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 -- 172.21.15.26:0/3021757901 >> 172.21.15.26:0/3021757901 conn(0x7f605c05faf0 msgr2=0x7f605c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 -- 172.21.15.26:0/3021757901 shutdown_connections 2022-07-20T03:36:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.090+0000 7f6061f2b700 1 -- 172.21.15.26:0/3021757901 wait complete. 2022-07-20T03:36:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 Processor -- start 2022-07-20T03:36:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- start start 2022-07-20T03:36:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c061a20 0x7f605c156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c12c750 0x7f605c1296c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 0x7f605c152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1524c0 con 0x7f605c1468c0 2022-07-20T03:36:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b4480 con 0x7f605c12c750 2022-07-20T03:36:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b46e0 con 0x7f605c061a20 2022-07-20T03:36:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 0x7f605c152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605b7fe700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c061a20 0x7f605c156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 0x7f605c152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45092/0 (socket says 172.21.15.26:45092) 2022-07-20T03:36:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 -- 172.21.15.26:0/2823499646 learned_addr learned my addr 172.21.15.26:0/2823499646 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605affd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c12c750 0x7f605c1296c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 -- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c061a20 msgr2=0x7f605c156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605c061a20 0x7f605c156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 -- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c12c750 msgr2=0x7f605c1296c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f605c12c750 0x7f605c1296c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c00b1e0 con 0x7f605c1468c0 2022-07-20T03:36:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f605bfff700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 0x7f605c152b50 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f6054006e00 tx=0x7f60540091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f605400c640 con 0x7f605c1468c0 2022-07-20T03:36:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605400c7a0 con 0x7f605c1468c0 2022-07-20T03:36:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c0b4980 con 0x7f605c1468c0 2022-07-20T03:36:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f605400cb00 con 0x7f605c1468c0 2022-07-20T03:36:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.094+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c0b50d0 con 0x7f605c1468c0 2022-07-20T03:36:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f605c1528b0 con 0x7f605c1468c0 2022-07-20T03:36:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6054022070 con 0x7f605c1468c0 2022-07-20T03:36:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f603c053b30 0x7f603c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f605406c300 con 0x7f605c1468c0 2022-07-20T03:36:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f605b7fe700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f603c053b30 0x7f603c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f603c05a7c0 0x7f603c05cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f603c05d240 con 0x7f603c05a7c0 2022-07-20T03:36:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f605406e990 con 0x7f605c1468c0 2022-07-20T03:36:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f605affd700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f603c05a7c0 0x7f603c05cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f605b7fe700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f603c053b30 0x7f603c055ff0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f604c01c000 tx=0x7f604c015000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.098+0000 7f605affd700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f603c05a7c0 0x7f603c05cc60 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.102+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== osd.6 v2:172.21.15.120:6808/23449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f603c05d240 con 0x7f603c05a7c0 2022-07-20T03:36:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 --> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f605c153360 con 0x7f603c05a7c0 2022-07-20T03:36:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6058ff9700 1 -- 172.21.15.26:0/2823499646 <== osd.6 v2:172.21.15.120:6808/23449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f605c153360 con 0x7f603c05a7c0 2022-07-20T03:36:46.157 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:46.158 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:46.159 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f603c05a7c0 msgr2=0x7f603c05cc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.120:6808/23449,v1:172.21.15.120:6809/23449] conn(0x7f603c05a7c0 0x7f603c05cc60 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f603c053b30 msgr2=0x7f603c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f603c053b30 0x7f603c055ff0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f604c01c000 tx=0x7f604c015000).stop 2022-07-20T03:36:46.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 msgr2=0x7f605c152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 --2- 172.21.15.26:0/2823499646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f605c1468c0 0x7f605c152b50 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f6054006e00 tx=0x7f60540091d0).stop 2022-07-20T03:36:46.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f605bfff700 1 -- 172.21.15.26:0/2823499646 reap_dead start 2022-07-20T03:36:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 shutdown_connections 2022-07-20T03:36:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 >> 172.21.15.26:0/2823499646 conn(0x7f605c05faf0 msgr2=0x7f605c12bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 shutdown_connections 2022-07-20T03:36:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.118+0000 7f6061f2b700 1 -- 172.21.15.26:0/2823499646 wait complete. 2022-07-20T03:36:46.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2022-07-20T03:36:46.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 Processor -- start 2022-07-20T03:36:46.305 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 -- start start 2022-07-20T03:36:46.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f14240b0070 0x7f1424060210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14240aa340 con 0x7f14240b2c70 2022-07-20T03:36:46.306 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1424062590 con 0x7f1424141560 2022-07-20T03:36:46.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142912c700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1424067270 con 0x7f14240b0070 2022-07-20T03:36:46.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f141bfff700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f14240b2c70 0x7f1424060700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44378/0 (socket says 172.21.15.26:44378) 2022-07-20T03:36:46.307 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f141bfff700 1 -- 172.21.15.26:0/4197127773 learned_addr learned my addr 172.21.15.26:0/4197127773 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:46.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> v2:172.21.15.120:3300/0 conn(0x7f14240b0070 0x7f1424060210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.308 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515447382 0 0) 0x7f14240aa340 con 0x7f14240b2c70 2022-07-20T03:36:46.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1404003260 con 0x7f14240b2c70 2022-07-20T03:36:46.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295684498 0 0) 0x7f1424062590 con 0x7f1424141560 2022-07-20T03:36:46.309 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14240aa340 con 0x7f1424141560 2022-07-20T03:36:46.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246983775 0 0) 0x7f1404003260 con 0x7f14240b2c70 2022-07-20T03:36:46.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 >> v2:172.21.15.120:3300/0 conn(0x7f14240b0070 msgr2=0x7f1424060210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:46.310 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 --2- 172.21.15.26:0/4197127773 >> v2:172.21.15.120:3300/0 conn(0x7f14240b0070 0x7f1424060210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 >> v1:172.21.15.120:6789/0 conn(0x7f1424141560 legacy=0x7f14241419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240b0450 con 0x7f14240b2c70 2022-07-20T03:36:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f140c001660 con 0x7f14240b2c70 2022-07-20T03:36:46.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> v2:172.21.15.120:3300/0 conn(0x7f14240b0070 0x7f1424060210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:46.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 >> v1:172.21.15.26:6789/0 conn(0x7f14240b2c70 legacy=0x7f1424060700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.312 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1404003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1404004500 0x7f1404004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1404004e70 0x7f140400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.313 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424067270 con 0x7f1404003960 2022-07-20T03:36:46.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140400bd50 con 0x7f1404004500 2022-07-20T03:36:46.314 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1404003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140400bf10 con 0x7f1404004e70 2022-07-20T03:36:46.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f141bfff700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1404004500 0x7f1404004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.315 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142359e700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1404004e70 0x7f140400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 -- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1404004e70 msgr2=0x7f140400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1404004e70 0x7f140400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 -- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1404004500 msgr2=0x7f1404004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:46.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1404004500 0x7f1404004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f142359e700 1 -- 172.21.15.26:0/4197127773 reap_dead start 2022-07-20T03:36:46.317 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.290+0000 7f1422d9d700 1 -- 172.21.15.26:0/4197127773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1404003260 con 0x7f1404003960 2022-07-20T03:36:46.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f1422d9d700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1404003d60 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f141400c000 tx=0x7f1414006df0).ready entity=mon.0 client_cookie=7a0edc5e6311d443 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141400e830 con 0x7f1404003960 2022-07-20T03:36:46.318 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400e990 con 0x7f1404003960 2022-07-20T03:36:46.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142259c700 1 -- 172.21.15.26:0/4197127773 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141400ecf0 con 0x7f1404003960 2022-07-20T03:36:46.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 msgr2=0x7f1404003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.319 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1404003d60 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f141400c000 tx=0x7f1414006df0).stop 2022-07-20T03:36:46.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- 172.21.15.26:0/4197127773 shutdown_connections 2022-07-20T03:36:46.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 --2- 172.21.15.26:0/4197127773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1404003d60 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.320 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- 172.21.15.26:0/4197127773 >> 172.21.15.26:0/4197127773 conn(0x7f14240b66e0 msgr2=0x7f14240b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.323 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- 172.21.15.26:0/4197127773 shutdown_connections 2022-07-20T03:36:46.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- 172.21.15.26:0/4197127773 wait complete. 2022-07-20T03:36:46.324 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 Processor -- start 2022-07-20T03:36:46.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- start start 2022-07-20T03:36:46.326 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.327 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1424144330 0x7f14241447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1424146d10 0x7f1424147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.328 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424066230 con 0x7f1404003960 2022-07-20T03:36:46.329 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424065fd0 con 0x7f1424144330 2022-07-20T03:36:46.330 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142912c700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1424150e40 con 0x7f1424146d10 2022-07-20T03:36:46.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f1422d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.331 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f1422d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45110/0 (socket says 172.21.15.26:45110) 2022-07-20T03:36:46.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f1422d9d700 1 -- 172.21.15.26:0/357652036 learned_addr learned my addr 172.21.15.26:0/357652036 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.332 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f142359e700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1424146d10 0x7f1424147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.333 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.294+0000 7f141bfff700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1424144330 0x7f14241447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 -- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1424146d10 msgr2=0x7f1424147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.334 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1424146d10 0x7f1424147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 -- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1424144330 msgr2=0x7f14241447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1424144330 0x7f14241447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 -- 172.21.15.26:0/357652036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14241510e0 con 0x7f1404003960 2022-07-20T03:36:46.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1422d9d700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f141400c000 tx=0x7f1414019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141401c9d0 con 0x7f1404003960 2022-07-20T03:36:46.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414000e90 con 0x7f1404003960 2022-07-20T03:36:46.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141401cd20 con 0x7f1404003960 2022-07-20T03:36:46.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1424147680 con 0x7f1404003960 2022-07-20T03:36:46.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1424065b10 con 0x7f1404003960 2022-07-20T03:36:46.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14240574e0 con 0x7f1404003960 2022-07-20T03:36:46.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.298+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f14140243f0 con 0x7f1404003960 2022-07-20T03:36:46.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.302+0000 7f1420d99700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 0x7f1408055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.302+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f141406d440 con 0x7f1404003960 2022-07-20T03:36:46.344 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.302+0000 7f141bfff700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 0x7f1408055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.345 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.302+0000 7f141bfff700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 0x7f1408055fc0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f140c005000 tx=0x7f140c001d00).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.306+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1414047eb0 con 0x7f1404003960 2022-07-20T03:36:46.440 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f14240683d0 con 0x7f1408053b00 2022-07-20T03:36:46.441 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f1420d99700 1 -- 172.21.15.26:0/357652036 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f14240683d0 con 0x7f1408053b00 2022-07-20T03:36:46.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 msgr2=0x7f1408055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.442 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 0x7f1408055fc0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f140c005000 tx=0x7f140c001d00).stop 2022-07-20T03:36:46.443 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 msgr2=0x7f1424065480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.444 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f141400c000 tx=0x7f1414019000).stop 2022-07-20T03:36:46.445 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 shutdown_connections 2022-07-20T03:36:46.446 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f1408053b00 0x7f1408055fc0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.447 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1424146d10 0x7f1424147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.448 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1424144330 0x7f14241447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.449 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 --2- 172.21.15.26:0/357652036 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1404003960 0x7f1424065480 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.450 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 >> 172.21.15.26:0/357652036 conn(0x7f14240b66e0 msgr2=0x7f14240b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.452 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 shutdown_connections 2022-07-20T03:36:46.453 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.434+0000 7f142912c700 1 -- 172.21.15.26:0/357652036 wait complete. 2022-07-20T03:36:46.455 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 6 to deep-scrub 2022-07-20T03:36:46.458 INFO:tasks.ceph:Scrubbing osd.7 2022-07-20T03:36:46.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2022-07-20T03:36:46.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.570+0000 7f47a6ff1700 1 Processor -- start 2022-07-20T03:36:46.575 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.570+0000 7f47a6ff1700 1 -- start start 2022-07-20T03:36:46.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47a00670c0 0x7f47a00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47a00b3380 0x7f47a014b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.576 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a0064880 con 0x7f47a0064450 2022-07-20T03:36:46.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00b85f0 con 0x7f47a00b3380 2022-07-20T03:36:46.577 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00b80b0 con 0x7f47a00670c0 2022-07-20T03:36:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47a00b3380 0x7f47a014b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f47a00b3380 0x7f47a014b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45120/0 (socket says 172.21.15.26:45120) 2022-07-20T03:36:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 -- 172.21.15.26:0/3675121978 learned_addr learned my addr 172.21.15.26:0/3675121978 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a5fef700 1 --2- 172.21.15.26:0/3675121978 >> v2:172.21.15.120:3300/0 conn(0x7f47a00670c0 0x7f47a00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 -- 172.21.15.26:0/3675121978 >> v2:172.21.15.120:3300/0 conn(0x7f47a00670c0 msgr2=0x7f47a00b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 --2- 172.21.15.26:0/3675121978 >> v2:172.21.15.120:3300/0 conn(0x7f47a00670c0 0x7f47a00b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 -- 172.21.15.26:0/3675121978 >> v1:172.21.15.26:6789/0 conn(0x7f47a0064450 legacy=0x7f47a0064f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 -- 172.21.15.26:0/3675121978 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00b8350 con 0x7f47a00b3380 2022-07-20T03:36:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a67f0700 1 --2- 172.21.15.26:0/3675121978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a00b3380 0x7f47a014b840 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f4790002f70 tx=0x7f4790006eb0).ready entity=mon.0 client_cookie=fedd08b8c75f080a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a4fed700 1 -- 172.21.15.26:0/3675121978 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f479000d5c0 con 0x7f47a00b3380 2022-07-20T03:36:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a4fed700 1 -- 172.21.15.26:0/3675121978 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790000e90 con 0x7f47a00b3380 2022-07-20T03:36:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a4fed700 1 -- 172.21.15.26:0/3675121978 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f479000d950 con 0x7f47a00b3380 2022-07-20T03:36:46.594 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- 172.21.15.26:0/3675121978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a00b3380 msgr2=0x7f47a014b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/3675121978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a00b3380 0x7f47a014b840 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f4790002f70 tx=0x7f4790006eb0).stop 2022-07-20T03:36:46.595 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- 172.21.15.26:0/3675121978 shutdown_connections 2022-07-20T03:36:46.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/3675121978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a00b3380 0x7f47a014b840 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.596 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/3675121978 >> v2:172.21.15.120:3300/0 conn(0x7f47a00670c0 0x7f47a00b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- 172.21.15.26:0/3675121978 >> 172.21.15.26:0/3675121978 conn(0x7f47a005f840 msgr2=0x7f47a005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- 172.21.15.26:0/3675121978 shutdown_connections 2022-07-20T03:36:46.597 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.574+0000 7f47a6ff1700 1 -- 172.21.15.26:0/3675121978 wait complete. 2022-07-20T03:36:46.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 Processor -- start 2022-07-20T03:36:46.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- start start 2022-07-20T03:36:46.598 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47a00670c0 0x7f47a0149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47a013b5e0 0x7f47a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.599 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 0x7f47a0148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b1e50 con 0x7f47a013f280 2022-07-20T03:36:46.600 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a013afe0 con 0x7f47a013b5e0 2022-07-20T03:36:46.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a013ba80 con 0x7f47a00670c0 2022-07-20T03:36:46.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 0x7f47a0148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.601 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 0x7f47a0148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45128/0 (socket says 172.21.15.26:45128) 2022-07-20T03:36:46.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 -- 172.21.15.26:0/2664464562 learned_addr learned my addr 172.21.15.26:0/2664464562 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.602 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a5fef700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47a00670c0 0x7f47a0149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a57ee700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47a013b5e0 0x7f47a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 -- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47a00670c0 msgr2=0x7f47a0149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.603 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47a00670c0 0x7f47a0149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.604 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 -- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47a013b5e0 msgr2=0x7f47a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47a013b5e0 0x7f47a013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.605 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479000b1e0 con 0x7f47a013f280 2022-07-20T03:36:46.607 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a67f0700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 0x7f47a0148890 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7f4790002f70 tx=0x7f4790015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.608 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47900116e0 con 0x7f47a013f280 2022-07-20T03:36:46.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790011840 con 0x7f47a013f280 2022-07-20T03:36:46.611 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a01402c0 con 0x7f47a013f280 2022-07-20T03:36:46.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4790011ba0 con 0x7f47a013f280 2022-07-20T03:36:46.612 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.578+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0062730 con 0x7f47a013f280 2022-07-20T03:36:46.613 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f479001a040 con 0x7f47a013f280 2022-07-20T03:36:46.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47a0140760 con 0x7f47a013f280 2022-07-20T03:36:46.614 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4780053aa0 0x7f4780055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.615 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f479006ccb0 con 0x7f47a013f280 2022-07-20T03:36:46.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f47a5fef700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4780053aa0 0x7f4780055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.616 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f478005a730 0x7f478005cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.617 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f478005d1b0 con 0x7f478005a730 2022-07-20T03:36:46.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f479006f230 con 0x7f47a013f280 2022-07-20T03:36:46.618 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f47a5fef700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4780053aa0 0x7f4780055f60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f479800b000 tx=0x7f47980091d0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.619 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f47a57ee700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f478005a730 0x7f478005cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f47a57ee700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f478005a730 0x7f478005cbd0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.620 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.582+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== osd.7 v2:172.21.15.120:6824/23460 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f478005d1b0 con 0x7f478005a730 2022-07-20T03:36:46.622 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.598+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 --> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f47a013bc40 con 0x7f478005a730 2022-07-20T03:36:46.623 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f479effd700 1 -- 172.21.15.26:0/2664464562 <== osd.7 v2:172.21.15.120:6824/23460 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f47a013bc40 con 0x7f478005a730 2022-07-20T03:36:46.624 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2022-07-20T03:36:46.624 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T03:36:46.625 INFO:teuthology.orchestra.run.smithi026.stdout:} 2022-07-20T03:36:46.630 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f478005a730 msgr2=0x7f478005cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.120:6824/23460,v1:172.21.15.120:6825/23460] conn(0x7f478005a730 0x7f478005cbd0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.631 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4780053aa0 msgr2=0x7f4780055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.632 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f4780053aa0 0x7f4780055f60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f479800b000 tx=0x7f47980091d0).stop 2022-07-20T03:36:46.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 msgr2=0x7f47a0148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.633 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 --2- 172.21.15.26:0/2664464562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f47a013f280 0x7f47a0148890 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7f4790002f70 tx=0x7f4790015000).stop 2022-07-20T03:36:46.634 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a67f0700 1 -- 172.21.15.26:0/2664464562 reap_dead start 2022-07-20T03:36:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 shutdown_connections 2022-07-20T03:36:46.635 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 >> 172.21.15.26:0/2664464562 conn(0x7f47a005f840 msgr2=0x7f47a005ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.636 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 shutdown_connections 2022-07-20T03:36:46.637 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.602+0000 7f47a6ff1700 1 -- 172.21.15.26:0/2664464562 wait complete. 2022-07-20T03:36:46.638 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2022-07-20T03:36:46.758 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 Processor -- start 2022-07-20T03:36:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 -- start start 2022-07-20T03:36:46.759 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7350147be0 0x7f73500aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73500b62d0 con 0x7f73500b5ef0 2022-07-20T03:36:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735014aac0 con 0x7f73500a7fc0 2022-07-20T03:36:46.760 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f7354e6e700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735014ad60 con 0x7f7350147be0 2022-07-20T03:36:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734f7fe700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f73500b5ef0 0x7f73501474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44414/0 (socket says 172.21.15.26:44414) 2022-07-20T03:36:46.761 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734f7fe700 1 -- 172.21.15.26:0/754010841 learned_addr learned my addr 172.21.15.26:0/754010841 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734effd700 1 --2- 172.21.15.26:0/754010841 >> v2:172.21.15.120:3300/0 conn(0x7f7350147be0 0x7f73500aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.762 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734e7fc700 1 -- 172.21.15.26:0/754010841 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403283181 0 0) 0x7f73500b62d0 con 0x7f73500b5ef0 2022-07-20T03:36:46.764 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734e7fc700 1 -- 172.21.15.26:0/754010841 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f733c003260 con 0x7f73500b5ef0 2022-07-20T03:36:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734effd700 1 -- 172.21.15.26:0/754010841 >> v1:172.21.15.120:6789/0 conn(0x7f73500a7fc0 legacy=0x7f73500af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734effd700 1 -- 172.21.15.26:0/754010841 >> v1:172.21.15.26:6789/0 conn(0x7f73500b5ef0 legacy=0x7f73501474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.765 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734effd700 1 -- 172.21.15.26:0/754010841 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735014b000 con 0x7f7350147be0 2022-07-20T03:36:46.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734effd700 1 --2- 172.21.15.26:0/754010841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7350147be0 0x7f73500aee40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f7340002f70 tx=0x7f7340006df0).ready entity=mon.1 client_cookie=9c6d585b2bd4c8f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.766 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734e7fc700 1 -- 172.21.15.26:0/754010841 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f734000d610 con 0x7f7350147be0 2022-07-20T03:36:46.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734e7fc700 1 -- 172.21.15.26:0/754010841 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7340000e90 con 0x7f7350147be0 2022-07-20T03:36:46.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.754+0000 7f734e7fc700 1 -- 172.21.15.26:0/754010841 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f734000d960 con 0x7f7350147be0 2022-07-20T03:36:46.767 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- 172.21.15.26:0/754010841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7350147be0 msgr2=0x7f73500aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 --2- 172.21.15.26:0/754010841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7350147be0 0x7f73500aee40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f7340002f70 tx=0x7f7340006df0).stop 2022-07-20T03:36:46.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- 172.21.15.26:0/754010841 shutdown_connections 2022-07-20T03:36:46.768 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 --2- 172.21.15.26:0/754010841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7350147be0 0x7f73500aee40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- 172.21.15.26:0/754010841 >> 172.21.15.26:0/754010841 conn(0x7f7350060310 msgr2=0x7f73500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- 172.21.15.26:0/754010841 shutdown_connections 2022-07-20T03:36:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- 172.21.15.26:0/754010841 wait complete. 2022-07-20T03:36:46.777 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 Processor -- start 2022-07-20T03:36:46.778 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- start start 2022-07-20T03:36:46.779 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 0x7f73500abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73500b5ef0 0x7f7350156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.780 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7350064ec0 con 0x7f7350147be0 2022-07-20T03:36:46.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73500b4610 con 0x7f73500a7fc0 2022-07-20T03:36:46.781 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f7354e6e700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73500b48b0 con 0x7f73500b5ef0 2022-07-20T03:36:46.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f734ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f734ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45144/0 (socket says 172.21.15.26:45144) 2022-07-20T03:36:46.782 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f734ffff700 1 -- 172.21.15.26:0/3499289787 learned_addr learned my addr 172.21.15.26:0/3499289787 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:46.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f734effd700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73500b5ef0 0x7f7350156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.783 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.758+0000 7f734f7fe700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 0x7f73500abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 -- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73500b5ef0 msgr2=0x7f7350156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.784 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73500b5ef0 0x7f7350156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 -- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 msgr2=0x7f73500abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.785 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 0x7f73500abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 -- 172.21.15.26:0/3499289787 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734000b180 con 0x7f7350147be0 2022-07-20T03:36:46.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734f7fe700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 0x7f73500abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:36:46.786 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734ffff700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f735006b790 tx=0x7f7348007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.787 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f734800e4a0 con 0x7f7350147be0 2022-07-20T03:36:46.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734800e600 con 0x7f7350147be0 2022-07-20T03:36:46.788 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73500626f0 con 0x7f7350147be0 2022-07-20T03:36:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f734800e960 con 0x7f7350147be0 2022-07-20T03:36:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7350062eb0 con 0x7f7350147be0 2022-07-20T03:36:46.789 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735014b2a0 con 0x7f7350147be0 2022-07-20T03:36:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7348019030 con 0x7f7350147be0 2022-07-20T03:36:46.790 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 0x7f7330055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:46.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f734806b4f0 con 0x7f7350147be0 2022-07-20T03:36:46.791 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.762+0000 7f734f7fe700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 0x7f7330055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:46.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.766+0000 7f734f7fe700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 0x7f7330055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f734400b000 tx=0x7f7344009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:46.792 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.770+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73480423a0 con 0x7f7350147be0 2022-07-20T03:36:46.895 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.890+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f73500574e0 con 0x7f7330053b30 2022-07-20T03:36:46.896 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f734cff9700 1 -- 172.21.15.26:0/3499289787 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f73500574e0 con 0x7f7330053b30 2022-07-20T03:36:46.905 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 msgr2=0x7f7330055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.906 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 0x7f7330055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f734400b000 tx=0x7f7344009040).stop 2022-07-20T03:36:46.907 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 msgr2=0x7f7350156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:46.908 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f735006b790 tx=0x7f7348007f40).stop 2022-07-20T03:36:46.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 shutdown_connections 2022-07-20T03:36:46.909 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7350147be0 0x7f7350156b10 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.910 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f7330053b30 0x7f7330055ff0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.911 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73500b5ef0 0x7f7350156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 --2- 172.21.15.26:0/3499289787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73500a7fc0 0x7f73500abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:46.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 >> 172.21.15.26:0/3499289787 conn(0x7f7350060310 msgr2=0x7f73500b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:46.912 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 shutdown_connections 2022-07-20T03:36:46.913 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:46.894+0000 7f7354e6e700 1 -- 172.21.15.26:0/3499289787 wait complete. 2022-07-20T03:36:46.913 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 7 to deep-scrub 2022-07-20T03:36:46.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T03:36:47.014 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 Processor -- start 2022-07-20T03:36:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 -- start start 2022-07-20T03:36:47.018 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b34147be0 0x7f2b340aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b340af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b340b62d0 con 0x7f2b340b5ef0 2022-07-20T03:36:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3414aac0 con 0x7f2b34147be0 2022-07-20T03:36:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b399c7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3414ad60 con 0x7f2b340a7fc0 2022-07-20T03:36:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7f2b34147be0 0x7f2b340aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b389c5700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2b340b5ef0 0x7f2b341474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44428/0 (socket says 172.21.15.26:44428) 2022-07-20T03:36:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b389c5700 1 -- 172.21.15.26:0/1887056828 learned_addr learned my addr 172.21.15.26:0/1887056828 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:36:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b391c6700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b340af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 -- 172.21.15.26:0/1887056828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 msgr2=0x7f2b340af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:36:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b340af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 -- 172.21.15.26:0/1887056828 >> v1:172.21.15.26:6789/0 conn(0x7f2b340b5ef0 legacy=0x7f2b341474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 -- 172.21.15.26:0/1887056828 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3414b000 con 0x7f2b34147be0 2022-07-20T03:36:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b33fff700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b340aee40 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24006df0).ready entity=mon.0 client_cookie=75f625070e1b0a44 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b337fe700 1 -- 172.21.15.26:0/1887056828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2400d570 con 0x7f2b34147be0 2022-07-20T03:36:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b337fe700 1 -- 172.21.15.26:0/1887056828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b24000e90 con 0x7f2b34147be0 2022-07-20T03:36:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.014+0000 7f2b337fe700 1 -- 172.21.15.26:0/1887056828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2400d8c0 con 0x7f2b34147be0 2022-07-20T03:36:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- 172.21.15.26:0/1887056828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 msgr2=0x7f2b340aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.027 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b340aee40 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24006df0).stop 2022-07-20T03:36:47.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- 172.21.15.26:0/1887056828 shutdown_connections 2022-07-20T03:36:47.028 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b340af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.029 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- 172.21.15.26:0/1887056828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b340aee40 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- 172.21.15.26:0/1887056828 >> 172.21.15.26:0/1887056828 conn(0x7f2b34060310 msgr2=0x7f2b340b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- 172.21.15.26:0/1887056828 shutdown_connections 2022-07-20T03:36:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- 172.21.15.26:0/1887056828 wait complete. 2022-07-20T03:36:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 Processor -- start 2022-07-20T03:36:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- start start 2022-07-20T03:36:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b34147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 0x7f2b340aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b340ab4d0 con 0x7f2b34147be0 2022-07-20T03:36:47.038 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b340ab770 con 0x7f2b340b5ef0 2022-07-20T03:36:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b399c7700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b340aba10 con 0x7f2b340a7fc0 2022-07-20T03:36:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b391c6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.039 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b389c5700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b34147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b33fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 0x7f2b340aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b391c6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45158/0 (socket says 172.21.15.26:45158) 2022-07-20T03:36:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b391c6700 1 -- 172.21.15.26:0/4172137435 learned_addr learned my addr 172.21.15.26:0/4172137435 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:36:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b33fff700 1 -- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 msgr2=0x7f2b34147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.041 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.018+0000 7f2b33fff700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b34147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b33fff700 1 -- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 msgr2=0x7f2b34159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b33fff700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.042 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b33fff700 1 -- 172.21.15.26:0/4172137435 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2400b180 con 0x7f2b340b5ef0 2022-07-20T03:36:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b391c6700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:36:47.043 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b33fff700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 0x7f2b340aa450 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b24013da0 con 0x7f2b340b5ef0 2022-07-20T03:36:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2401e400 con 0x7f2b340b5ef0 2022-07-20T03:36:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2401e760 con 0x7f2b340b5ef0 2022-07-20T03:36:47.045 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b340abcb0 con 0x7f2b340b5ef0 2022-07-20T03:36:47.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b340aa940 con 0x7f2b340b5ef0 2022-07-20T03:36:47.046 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3414b2a0 con 0x7f2b340b5ef0 2022-07-20T03:36:47.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b2401c070 con 0x7f2b340b5ef0 2022-07-20T03:36:47.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 0x7f2b14055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:36:47.047 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2b2406cc40 con 0x7f2b340b5ef0 2022-07-20T03:36:47.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.022+0000 7f2b389c5700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 0x7f2b14055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:36:47.048 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.026+0000 7f2b389c5700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 0x7f2b14055f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f2b2800d000 tx=0x7f2b28009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:36:47.049 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.034+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b24043bd0 con 0x7f2b340b5ef0 2022-07-20T03:36:47.160 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.158+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b340574e0 con 0x7f2b14053a70 2022-07-20T03:36:47.162 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b31ffb700 1 -- 172.21.15.26:0/4172137435 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67831 (secure 0 0 0) 0x7f2b340574e0 con 0x7f2b14053a70 2022-07-20T03:36:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T03:36:47.165 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 msgr2=0x7f2b14055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 0x7f2b14055f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f2b2800d000 tx=0x7f2b28009040).stop 2022-07-20T03:36:47.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 msgr2=0x7f2b340aa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:36:47.166 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 0x7f2b340aa450 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f2b24002f70 tx=0x7f2b24014000).stop 2022-07-20T03:36:47.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 shutdown_connections 2022-07-20T03:36:47.167 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b34147be0 0x7f2b34159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f2b14053a70 0x7f2b14055f30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b340b5ef0 0x7f2b340aa450 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.168 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 --2- 172.21.15.26:0/4172137435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b340a7fc0 0x7f2b34147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:36:47.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 >> 172.21.15.26:0/4172137435 conn(0x7f2b34060310 msgr2=0x7f2b340af040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:36:47.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 shutdown_connections 2022-07-20T03:36:47.169 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:36:47.162+0000 7f2b399c7700 1 -- 172.21.15.26:0/4172137435 wait complete. 2022-07-20T03:36:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2022-07-20T03:36:47.176 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2302,"stamp":"2022-07-20T03:36:46.580766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8405088,"kb_used_data":16416,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569632,"statfs":{"total":773094113280,"available":764487303168,"internally_reserved":8589934592,"allocated":16809984,"data_stored":8211272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003393"},"pg_stats":[{"pgid":"1.0","version":"14'2","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264259+0000","last_change":"2022-07-20T02:25:25.353187+0000","last_active":"2022-07-20T02:36:09.264259+0000","last_peered":"2022-07-20T02:36:09.264259+0000","last_clean":"2022-07-20T02:36:09.264259+0000","last_became_active":"2022-07-20T02:25:25.351806+0000","last_became_peered":"2022-07-20T02:25:25.351806+0000","last_unstale":"2022-07-20T02:36:09.264259+0000","last_undegraded":"2022-07-20T02:36:09.264259+0000","last_fullsized":"2022-07-20T02:36:09.264259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:19.583498+0000","last_clean_scrub_stamp":"2022-07-20T02:25:19.583498+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.262881+0000","last_change":"2022-07-20T02:25:29.458996+0000","last_active":"2022-07-20T02:36:09.262881+0000","last_peered":"2022-07-20T02:36:09.262881+0000","last_clean":"2022-07-20T02:36:09.262881+0000","last_became_active":"2022-07-20T02:25:29.458580+0000","last_became_peered":"2022-07-20T02:25:29.458580+0000","last_unstale":"2022-07-20T02:36:09.262881+0000","last_undegraded":"2022-07-20T02:36:09.262881+0000","last_fullsized":"2022-07-20T02:36:09.262881+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.263966+0000","last_change":"2022-07-20T02:25:29.455281+0000","last_active":"2022-07-20T02:36:09.263966+0000","last_peered":"2022-07-20T02:36:09.263966+0000","last_clean":"2022-07-20T02:36:09.263966+0000","last_became_active":"2022-07-20T02:25:29.454947+0000","last_became_peered":"2022-07-20T02:25:29.454947+0000","last_unstale":"2022-07-20T02:36:09.263966+0000","last_undegraded":"2022-07-20T02:36:09.263966+0000","last_fullsized":"2022-07-20T02:36:09.263966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":142,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.263414+0000","last_change":"2022-07-20T02:25:29.454317+0000","last_active":"2022-07-20T02:36:09.263414+0000","last_peered":"2022-07-20T02:36:09.263414+0000","last_clean":"2022-07-20T02:36:09.263414+0000","last_became_active":"2022-07-20T02:25:29.453725+0000","last_became_peered":"2022-07-20T02:25:29.453725+0000","last_unstale":"2022-07-20T02:36:09.263414+0000","last_undegraded":"2022-07-20T02:36:09.263414+0000","last_fullsized":"2022-07-20T02:36:09.263414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.262821+0000","last_change":"2022-07-20T02:25:29.457758+0000","last_active":"2022-07-20T02:36:09.262821+0000","last_peered":"2022-07-20T02:36:09.262821+0000","last_clean":"2022-07-20T02:36:09.262821+0000","last_became_active":"2022-07-20T02:25:29.457326+0000","last_became_peered":"2022-07-20T02:25:29.457326+0000","last_unstale":"2022-07-20T02:36:09.262821+0000","last_undegraded":"2022-07-20T02:36:09.262821+0000","last_fullsized":"2022-07-20T02:36:09.262821+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":144,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:44.342500+0000","last_change":"2022-07-20T03:36:44.342614+0000","last_active":"2022-07-20T03:36:44.342500+0000","last_peered":"2022-07-20T03:36:44.342500+0000","last_clean":"2022-07-20T03:36:44.342500+0000","last_became_active":"2022-07-20T02:25:29.901376+0000","last_became_peered":"2022-07-20T02:25:29.901376+0000","last_unstale":"2022-07-20T03:36:44.342500+0000","last_undegraded":"2022-07-20T03:36:44.342500+0000","last_fullsized":"2022-07-20T03:36:44.342500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:44.342499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:44.342499+0000","last_clean_scrub_stamp":"2022-07-20T03:36:44.342499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264800+0000","last_change":"2022-07-20T02:25:29.909282+0000","last_active":"2022-07-20T02:36:09.264800+0000","last_peered":"2022-07-20T02:36:09.264800+0000","last_clean":"2022-07-20T02:36:09.264800+0000","last_became_active":"2022-07-20T02:25:29.908425+0000","last_became_peered":"2022-07-20T02:25:29.908425+0000","last_unstale":"2022-07-20T02:36:09.264800+0000","last_undegraded":"2022-07-20T02:36:09.264800+0000","last_fullsized":"2022-07-20T02:36:09.264800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:09.264964+0000","last_change":"2022-07-20T02:25:29.452052+0000","last_active":"2022-07-20T02:36:09.264964+0000","last_peered":"2022-07-20T02:36:09.264964+0000","last_clean":"2022-07-20T02:36:09.264964+0000","last_became_active":"2022-07-20T02:25:29.451566+0000","last_became_peered":"2022-07-20T02:25:29.451566+0000","last_unstale":"2022-07-20T02:36:09.264964+0000","last_undegraded":"2022-07-20T02:36:09.264964+0000","last_fullsized":"2022-07-20T02:36:09.264964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-20T02:36:08.652855+0000","last_change":"2022-07-20T02:25:29.458394+0000","last_active":"2022-07-20T02:36:08.652855+0000","last_peered":"2022-07-20T02:36:08.652855+0000","last_clean":"2022-07-20T02:36:08.652855+0000","last_became_active":"2022-07-20T02:25:29.458000+0000","last_became_peered":"2022-07-20T02:25:29.458000+0000","last_unstale":"2022-07-20T02:36:08.652855+0000","last_undegraded":"2022-07-20T02:36:08.652855+0000","last_fullsized":"2022-07-20T02:36:08.652855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T02:25:28.425908+0000","last_clean_scrub_stamp":"2022-07-20T02:25:28.425908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575706,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.83199999999999996,"15min":0.85599999999999998},"min":{"1min":0.57899999999999996,"5min":0.47099999999999997,"15min":0.43099999999999999},"max":{"1min":1.0209999999999999,"5min":1.089,"15min":1.1739999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.67000000000000004,"5min":0.56899999999999995,"15min":0.52400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.2430000000000001},"last":0.83399999999999996}]},{"osd":1,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.78200000000000003,"15min":0.78400000000000003},"min":{"1min":0.57599999999999996,"5min":0.54500000000000004,"15min":0.52000000000000002},"max":{"1min":1.097,"5min":1.208,"15min":1.2749999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1830000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.874,"15min":0.873},"min":{"1min":0.63700000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.052,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91200000000000003,"15min":0.93000000000000005},"min":{"1min":0.73199999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.125,"5min":1.135,"15min":1.24},"last":0.91900000000000004}]},{"osd":3,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.82199999999999995,"15min":0.82799999999999996},"min":{"1min":0.70999999999999996,"5min":0.63700000000000001,"15min":0.58799999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.1399999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.87,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.54700000000000004},"max":{"1min":1.1759999999999999,"5min":1.244,"15min":1.244},"last":0.68300000000000005}]},{"osd":4,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72799999999999998,"15min":0.73299999999999998},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.50700000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.163},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.0549999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.88700000000000001,"15min":0.875},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47299999999999998},"max":{"1min":1.133,"5min":1.1719999999999999,"15min":1.355},"last":0.62},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.96199999999999997,"15min":0.94899999999999995},"min":{"1min":0.70899999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1279999999999999,"5min":1.2629999999999999,"15min":1.266},"last":1.03}]},{"osd":6,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92500000000000004,"15min":0.90200000000000002},"min":{"1min":0.63500000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.208,"15min":1.228},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.93999999999999995,"15min":0.91200000000000003},"min":{"1min":0.66500000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.165,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.72199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":1.163,"15min":1.1739999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51200000000000001},"max":{"1min":1.0509999999999999,"5min":1.119,"15min":1.165},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84699999999999998,"15min":0.82399999999999995},"min":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.036,"5min":1.149,"15min":1.2609999999999999},"last":0.754},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.84199999999999997,"15min":0.85799999999999998},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60099999999999998},"max":{"1min":1.111,"5min":1.1200000000000001,"15min":1.1970000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81000000000000005,"15min":0.80700000000000005},"min":{"1min":0.624,"5min":0.59199999999999997,"15min":0.58299999999999996},"max":{"1min":1.004,"5min":1.085,"15min":1.272},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.86499999999999999,"15min":0.86699999999999999},"min":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.0269999999999999,"5min":1.121,"15min":1.2989999999999999},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.874,"15min":0.86599999999999999},"min":{"1min":0.74299999999999999,"5min":0.626,"15min":0.54400000000000004},"max":{"1min":1.097,"5min":1.1639999999999999,"15min":1.2090000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.62,"15min":0.61799999999999999},"max":{"1min":0.98099999999999998,"5min":1.069,"15min":1.179},"last":0.89500000000000002}]},{"osd":4,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76400000000000001,"15min":0.77400000000000002},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51500000000000001},"max":{"1min":1.121,"5min":1.1839999999999999,"15min":1.2609999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84999999999999998,"15min":0.83699999999999997},"min":{"1min":0.60099999999999998,"5min":0.56399999999999995,"15min":0.38700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.871}]},{"osd":5,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81399999999999995,"15min":0.82099999999999995},"min":{"1min":0.60799999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.038,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.56100000000000005},"max":{"1min":1.077,"5min":1.327,"15min":1.3360000000000001},"last":0.85199999999999998}]},{"osd":7,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93500000000000005,"15min":0.94399999999999995},"min":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.436},"max":{"1min":1.403,"5min":1.403,"15min":1.4379999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.80900000000000005,"5min":0.63800000000000001,"15min":0.56000000000000005},"max":{"1min":1.083,"5min":1.2050000000000001,"15min":1.5389999999999999},"last":0.99399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.81799999999999995,"15min":0.80900000000000005},"min":{"1min":0.59299999999999997,"5min":0.57299999999999995,"15min":0.441},"max":{"1min":1.125,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.61499999999999999,"15min":0.59899999999999998},"max":{"1min":1.1200000000000001,"5min":1.1220000000000001,"15min":1.1779999999999999},"last":0.88400000000000001}]},{"osd":1,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.83499999999999996},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.748},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.84999999999999998,"15min":0.84599999999999997},"min":{"1min":0.76800000000000002,"5min":0.625,"15min":0.60399999999999998},"max":{"1min":0.99399999999999999,"5min":1.073,"15min":1.149},"last":1.081}]},{"osd":2,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85699999999999998,"15min":0.86899999999999999},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.621},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86599999999999999,"15min":0.877},"min":{"1min":0.65700000000000003,"5min":0.61399999999999999,"15min":0.61199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.2210000000000001},"last":0.92700000000000005}]},{"osd":3,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"min":{"1min":0.70199999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.0109999999999999,"5min":1.232,"15min":1.232},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91300000000000003,"15min":0.91000000000000003},"min":{"1min":0.78700000000000003,"5min":0.65900000000000003,"15min":0.63800000000000001},"max":{"1min":1.111,"5min":1.3,"15min":1.3},"last":1.012}]},{"osd":4,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.94299999999999995,"15min":0.92000000000000004},"min":{"1min":0.68200000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1950000000000001,"5min":1.242,"15min":1.268},"last":1.03}]},{"osd":6,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92900000000000005,"15min":0.90800000000000003},"min":{"1min":0.65300000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2549999999999999,"5min":1.357,"15min":1.357},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.67900000000000005,"5min":0.63200000000000001,"15min":0.59399999999999997},"max":{"1min":1.21,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.155}]},{"osd":7,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.995,"15min":0.98999999999999999},"min":{"1min":0.82899999999999996,"5min":0.66600000000000004,"15min":0.63800000000000001},"max":{"1min":1.216,"5min":1.2929999999999999,"15min":1.4350000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.042,"15min":1.0229999999999999},"min":{"1min":0.82499999999999996,"5min":0.70299999999999996,"15min":0.60099999999999998},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.131}]}]},{"osd":4,"up_from":13,"seq":55834575706,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.53000000000000003},"max":{"1min":1.081,"5min":1.101,"15min":1.149},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.60699999999999998,"5min":0.59199999999999997,"15min":0.56599999999999995},"max":{"1min":1.0980000000000001,"5min":1.1339999999999999,"15min":1.2150000000000001},"last":0.621}]},{"osd":1,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.83299999999999996},"min":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.59799999999999998},"max":{"1min":1.0620000000000001,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.878,"15min":0.86299999999999999},"min":{"1min":0.63900000000000001,"5min":0.61399999999999999,"15min":0.59399999999999997},"max":{"1min":1.0760000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.82299999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.90400000000000003,"15min":0.88400000000000001},"min":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.58899999999999997},"max":{"1min":1.1779999999999999,"5min":1.1919999999999999,"15min":1.2390000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89200000000000002,"15min":0.90000000000000002},"min":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.59799999999999998},"max":{"1min":1.171,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.84399999999999997,"15min":0.85099999999999998},"min":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.60599999999999998},"max":{"1min":1.198,"5min":1.198,"15min":1.248},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92900000000000005,"15min":0.92400000000000004},"min":{"1min":0.68000000000000005,"5min":0.67800000000000005,"15min":0.65500000000000003},"max":{"1min":1.181,"5min":1.236,"15min":1.25},"last":0.94099999999999995}]},{"osd":5,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.86699999999999999,"15min":0.85499999999999998},"min":{"1min":0.58799999999999997,"5min":0.56699999999999995,"15min":0.54400000000000004},"max":{"1min":1.2030000000000001,"5min":1.268,"15min":1.268},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.58899999999999997,"15min":0.55300000000000005},"max":{"1min":1.272,"5min":1.2729999999999999,"15min":1.413},"last":0.98099999999999998}]},{"osd":6,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.89800000000000002,"15min":0.91300000000000003},"min":{"1min":0.68100000000000005,"5min":0.623,"15min":0.60399999999999998},"max":{"1min":1.125,"5min":1.2390000000000001,"15min":1.2769999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.042,"5min":0.98299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.64400000000000002},"max":{"1min":1.2250000000000001,"5min":1.339,"15min":1.339},"last":0.68300000000000005}]},{"osd":7,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.79300000000000004,"5min":0.65800000000000003,"15min":0.55600000000000005},"max":{"1min":1.3149999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.107,"5min":1.069,"15min":1.044},"min":{"1min":0.90300000000000002,"5min":0.70599999999999996,"15min":0.67300000000000004},"max":{"1min":1.444,"5min":1.4510000000000001,"15min":1.482},"last":1.0800000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61199999999999999,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":0.98699999999999999,"5min":0.995,"15min":1.0620000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.55100000000000005,"5min":0.50700000000000001,"15min":0.46000000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.1299999999999999},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":0.79600000000000004,"5min":1.1180000000000001,"15min":1.236},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.99299999999999999,"5min":1.0820000000000001,"15min":1.2150000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82699999999999996,"15min":0.83999999999999997},"min":{"1min":0.59999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0680000000000001,"5min":1.171,"15min":1.2330000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.89200000000000002,"15min":0.89900000000000002},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.55500000000000005},"max":{"1min":1.143,"5min":1.143,"15min":1.206},"last":1.0509999999999999}]},{"osd":4,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.57899999999999996},"max":{"1min":1.0980000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.93799999999999994,"15min":0.95399999999999996},"min":{"1min":0.748,"5min":0.68899999999999995,"15min":0.63200000000000001},"max":{"1min":1.1870000000000001,"5min":1.579,"15min":1.579},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.95999999999999996,"15min":0.98499999999999999},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.68600000000000005},"max":{"1min":1.198,"5min":1.4590000000000001,"15min":1.4690000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.016,"15min":1.036},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.69799999999999995},"max":{"1min":1.3979999999999999,"5min":1.601,"15min":1.6379999999999999},"last":0.90300000000000002}]},{"osd":6,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.014,"15min":1.012},"min":{"1min":0.77900000000000003,"5min":0.73899999999999999,"15min":0.69499999999999995},"max":{"1min":1.3620000000000001,"5min":1.534,"15min":1.534},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.052,"15min":1.0629999999999999},"min":{"1min":0.82399999999999995,"5min":0.73599999999999999,"15min":0.72799999999999998},"max":{"1min":1.2390000000000001,"5min":1.661,"15min":1.661},"last":0.91500000000000004}]},{"osd":7,"last update":"Wed Jul 20 03:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.077,"15min":1.105},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.74299999999999999},"max":{"1min":1.4510000000000001,"5min":1.5209999999999999,"15min":1.681},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0980000000000001,"15min":1.1000000000000001},"min":{"1min":0.88200000000000001,"5min":0.879,"15min":0.76000000000000001},"max":{"1min":1.294,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.93600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.433},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.1080000000000001},"last":0.495},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.71499999999999997,"15min":0.72699999999999998},"min":{"1min":0.52400000000000002,"5min":0.48099999999999998,"15min":0.44800000000000001},"max":{"1min":1.121,"5min":1.1479999999999999,"15min":1.2270000000000001},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.46200000000000002},"max":{"1min":0.76900000000000002,"5min":1.1359999999999999,"15min":1.238},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.50900000000000001},"max":{"1min":0.873,"5min":1.151,"15min":1.194},"last":0.52100000000000002}]},{"osd":3,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.69699999999999995,"5min":0.66200000000000003,"15min":0.628},"max":{"1min":1.1379999999999999,"5min":1.1910000000000001,"15min":1.2470000000000001},"last":0.754},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78500000000000003,"15min":0.79200000000000004},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52400000000000002},"max":{"1min":0.90400000000000003,"5min":1.2529999999999999,"15min":1.351},"last":0.69399999999999995}]},{"osd":4,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88200000000000001},"min":{"1min":0.72999999999999998,"5min":0.65100000000000002,"15min":0.60499999999999998},"max":{"1min":1.0920000000000001,"5min":1.373,"15min":1.373},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.95599999999999996},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.63},"max":{"1min":1.2929999999999999,"5min":1.385,"15min":1.4179999999999999},"last":1.073}]},{"osd":5,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96699999999999997,"15min":0.97399999999999998},"min":{"1min":0.79100000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2729999999999999,"5min":1.546,"15min":1.546},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.996,"15min":0.997},"min":{"1min":0.80600000000000005,"5min":0.70099999999999996,"15min":0.68300000000000005},"max":{"1min":1.181,"5min":1.4830000000000001,"15min":1.645},"last":0.93999999999999995}]},{"osd":6,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.0429999999999999},"min":{"1min":0.75700000000000001,"5min":0.75,"15min":0.66700000000000004},"max":{"1min":1.232,"5min":1.514,"15min":1.595},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0880000000000001,"15min":1.073},"min":{"1min":0.80500000000000005,"5min":0.77300000000000002,"15min":0.69699999999999995},"max":{"1min":1.3049999999999999,"5min":1.611,"15min":1.611},"last":1.034}]},{"osd":7,"last update":"Wed Jul 20 03:35:44 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1140000000000001,"15min":1.1020000000000001},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.73599999999999999},"max":{"1min":1.339,"5min":1.4990000000000001,"15min":2.0960000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.121,"15min":1.1259999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.359,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.161}]}]},{"osd":1,"up_from":13,"seq":55834575706,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.046},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75,"15min":0.76100000000000001},"min":{"1min":0.63800000000000001,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":1.004,"15min":1.181},"last":0.626}]},{"osd":2,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69099999999999995,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.496},"max":{"1min":0.91900000000000004,"5min":1.085,"15min":1.147},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73099999999999998,"15min":0.70799999999999996},"min":{"1min":0.55000000000000004,"5min":0.52300000000000002,"15min":0.47399999999999998},"max":{"1min":0.96099999999999997,"5min":1.139,"15min":1.139},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78400000000000003,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70799999999999996,"15min":0.72499999999999998},"min":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"max":{"1min":0.93799999999999994,"5min":1.123,"15min":1.171},"last":0.83499999999999996}]},{"osd":4,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93400000000000005,"15min":0.93700000000000006},"min":{"1min":0.76000000000000001,"5min":0.70999999999999996,"15min":0.65900000000000003},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.92800000000000005,"15min":0.91800000000000004},"min":{"1min":0.746,"5min":0.71899999999999997,"15min":0.66900000000000004},"max":{"1min":1.1679999999999999,"5min":1.2190000000000001,"15min":1.3169999999999999},"last":0.94199999999999995}]},{"osd":5,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.95199999999999996,"15min":0.96699999999999997},"min":{"1min":0.84399999999999997,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.395},"last":1.054},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.044,"15min":1.05},"min":{"1min":0.95299999999999996,"5min":0.84499999999999997,"15min":0.64200000000000002},"max":{"1min":1.1919999999999999,"5min":1.3799999999999999,"15min":1.486},"last":1.083}]},{"osd":6,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.012,"15min":1.0389999999999999},"min":{"1min":0.79000000000000004,"5min":0.69099999999999995,"15min":0.65800000000000003},"max":{"1min":1.2190000000000001,"5min":1.444,"15min":1.589},"last":1.153},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.018,"15min":1.012},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69299999999999995},"max":{"1min":1.2989999999999999,"5min":1.653,"15min":1.653},"last":1.0309999999999999}]},{"osd":7,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.018,"15min":1.0429999999999999},"min":{"1min":0.83199999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.248,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.111,"15min":1.123},"min":{"1min":0.94899999999999995,"5min":0.84499999999999997,"15min":0.71099999999999997},"max":{"1min":1.2589999999999999,"5min":1.411,"15min":1.544},"last":1.2170000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575706,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.51500000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.80800000000000005,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.52800000000000002},"max":{"1min":0.91800000000000004,"5min":1.0820000000000001,"15min":1.1779999999999999},"last":0.90500000000000003}]},{"osd":2,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64900000000000002,"15min":0.63700000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.48699999999999999},"max":{"1min":0.88500000000000001,"5min":0.96999999999999997,"15min":1.089},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.53200000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.92700000000000005,"5min":1.161,"15min":1.1639999999999999},"last":0.67200000000000004}]},{"osd":3,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.55200000000000005,"5min":0.53100000000000003,"15min":0.504},"max":{"1min":1.085,"5min":1.1559999999999999,"15min":1.1719999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77300000000000002,"15min":0.76200000000000001},"min":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":1.0549999999999999,"5min":1.177,"15min":1.177},"last":0.59599999999999997}]},{"osd":4,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.93799999999999994,"15min":0.93500000000000005},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.96199999999999997,"15min":0.95999999999999996},"min":{"1min":0.749,"5min":0.71099999999999997,"15min":0.69199999999999995},"max":{"1min":1.29,"5min":1.29,"15min":1.3400000000000001},"last":1.0369999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96699999999999997,"15min":0.98499999999999999},"min":{"1min":0.78700000000000003,"5min":0.69499999999999995,"15min":0.66800000000000004},"max":{"1min":1.2390000000000001,"5min":1.3220000000000001,"15min":1.3560000000000001},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97299999999999998,"15min":0.96199999999999997},"min":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.65800000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.52},"last":1.018}]},{"osd":6,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0269999999999999,"15min":1.0129999999999999},"min":{"1min":0.80700000000000005,"5min":0.72799999999999998,"15min":0.624},"max":{"1min":1.329,"5min":1.3420000000000001,"15min":1.4690000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.075,"5min":1.0369999999999999,"15min":1.056},"min":{"1min":0.93799999999999994,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.615},"last":1.0640000000000001}]},{"osd":7,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.069,"15min":1.077},"min":{"1min":0.95299999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.268,"5min":1.401,"15min":1.7050000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.091,"15min":1.1020000000000001},"min":{"1min":0.89800000000000002,"5min":0.82399999999999995,"15min":0.78600000000000003},"max":{"1min":1.379,"5min":1.419,"15min":1.6399999999999999},"last":0.95999999999999996}]}]}]}} 2022-07-20T03:36:47.192 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-07-20T02:25:19.583498+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=19, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.193 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.194 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.197 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.198 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.202 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.204 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.205 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-07-20T02:25:28.425908+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=2, tm_min=25, tm_sec=28, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=36, tm_sec=42, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T03:36:47.205 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-07-20T03:37:05.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:05.418 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:05.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:05.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:07.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T03:37:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 Processor -- start 2022-07-20T03:37:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- start start 2022-07-20T03:37:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd2200a7fc0 0x7fd2200af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2200b62d0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- --> v2:172.21.15.26:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22014aac0 con 0x7fd2200a7fc0 2022-07-20T03:37:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22014ad60 con 0x7fd220147be0 2022-07-20T03:37:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd2200a7fc0 0x7fd2200af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22596e700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd2200b5ef0 0x7fd2201474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44442/0 (socket says 172.21.15.26:44442) 2022-07-20T03:37:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 --2- >> v2:172.21.15.26:3300/0 conn(0x7fd2200a7fc0 0x7fd2200af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45166/0 (socket says 172.21.15.26:45166) 2022-07-20T03:37:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22596e700 1 -- 172.21.15.26:0/3985805917 learned_addr learned my addr 172.21.15.26:0/3985805917 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:37:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22516d700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 -- 172.21.15.26:0/3985805917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 msgr2=0x7fd2200aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:37:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 -- 172.21.15.26:0/3985805917 >> v1:172.21.15.26:6789/0 conn(0x7fd2200b5ef0 legacy=0x7fd2201474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22496c700 1 -- 172.21.15.26:0/3985805917 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509169269 0 0) 0x7fd2200b62d0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 -- 172.21.15.26:0/3985805917 --> v2:172.21.15.26:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22014b000 con 0x7fd2200a7fc0 2022-07-20T03:37:07.348 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22616f700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200a7fc0 0x7fd2200af330 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7fd218002f70 tx=0x7fd218006df0).ready entity=mon.0 client_cookie=931440f5611ac9e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22496c700 1 -- 172.21.15.26:0/3985805917 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21800d570 con 0x7fd2200a7fc0 2022-07-20T03:37:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22496c700 1 -- 172.21.15.26:0/3985805917 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218000e90 con 0x7fd2200a7fc0 2022-07-20T03:37:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd22496c700 1 -- 172.21.15.26:0/3985805917 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21800d8c0 con 0x7fd2200a7fc0 2022-07-20T03:37:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- 172.21.15.26:0/3985805917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200a7fc0 msgr2=0x7fd2200af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200a7fc0 0x7fd2200af330 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7fd218002f70 tx=0x7fd218006df0).stop 2022-07-20T03:37:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- 172.21.15.26:0/3985805917 shutdown_connections 2022-07-20T03:37:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200a7fc0 0x7fd2200af330 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 --2- 172.21.15.26:0/3985805917 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.322+0000 7fd226970700 1 -- 172.21.15.26:0/3985805917 >> 172.21.15.26:0/3985805917 conn(0x7fd220060310 msgr2=0x7fd2200b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:37:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- 172.21.15.26:0/3985805917 shutdown_connections 2022-07-20T03:37:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- 172.21.15.26:0/3985805917 wait complete. 2022-07-20T03:37:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 Processor -- start 2022-07-20T03:37:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- start start 2022-07-20T03:37:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2200a7fc0 0x7fd220062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.357 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd220064800 con 0x7fd2200b5ef0 2022-07-20T03:37:07.358 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd220064aa0 con 0x7fd2200a7fc0 2022-07-20T03:37:07.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd226970700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd220064d40 con 0x7fd220147be0 2022-07-20T03:37:07.360 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd22516d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd22516d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45170/0 (socket says 172.21.15.26:45170) 2022-07-20T03:37:07.361 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd22516d700 1 -- 172.21.15.26:0/3886937503 learned_addr learned my addr 172.21.15.26:0/3886937503 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:37:07.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd22596e700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2200a7fc0 0x7fd220062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.326+0000 7fd22616f700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.362 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 -- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 msgr2=0x7fd2200aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 -- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2200a7fc0 msgr2=0x7fd220062a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2200a7fc0 0x7fd220062a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 -- 172.21.15.26:0/3886937503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21800b180 con 0x7fd2200b5ef0 2022-07-20T03:37:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22516d700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7fd21c00c000 tx=0x7fd21c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21c012600 con 0x7fd2200b5ef0 2022-07-20T03:37:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c012760 con 0x7fd2200b5ef0 2022-07-20T03:37:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2200ab5e0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd21c012ac0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2200abf70 con 0x7fd2200b5ef0 2022-07-20T03:37:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22014b2a0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd21c022070 con 0x7fd2200b5ef0 2022-07-20T03:37:07.369 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 0x7fd200055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd21c06c3c0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.371 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.330+0000 7fd22596e700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 0x7fd200055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.334+0000 7fd22596e700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 0x7fd200055fc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fd21000d000 tx=0x7fd210009040).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.338+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd21c0433c0 con 0x7fd2200b5ef0 2022-07-20T03:37:07.469 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.462+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 --> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2200574e0 con 0x7fd200053b00 2022-07-20T03:37:07.473 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd20effd700 1 -- 172.21.15.26:0/3886937503 <== mgr.4104 v2:172.21.15.26:6800/22865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67672 (secure 0 0 0) 0x7fd2200574e0 con 0x7fd200053b00 2022-07-20T03:37:07.480 INFO:teuthology.orchestra.run.smithi026.stdout: 2022-07-20T03:37:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 msgr2=0x7fd200055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 0x7fd200055fc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fd21000d000 tx=0x7fd210009040).stop 2022-07-20T03:37:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 msgr2=0x7fd2201565c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7fd21c00c000 tx=0x7fd21c0091d0).stop 2022-07-20T03:37:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 shutdown_connections 2022-07-20T03:37:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2200b5ef0 0x7fd2201565c0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7fd200053b00 0x7fd200055fc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd220147be0 0x7fd2200aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.522 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 --2- 172.21.15.26:0/3886937503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2200a7fc0 0x7fd220062a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 >> 172.21.15.26:0/3886937503 conn(0x7fd220060310 msgr2=0x7fd2200ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:37:07.523 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 shutdown_connections 2022-07-20T03:37:07.524 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.470+0000 7fd226970700 1 -- 172.21.15.26:0/3886937503 wait complete. 2022-07-20T03:37:07.524 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2022-07-20T03:37:07.525 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2022-07-20T03:37:06.586632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9464,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8405088,"kb_used_data":16416,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569632,"statfs":{"total":773094113280,"available":764487303168,"internally_reserved":8589934592,"allocated":16809984,"data_stored":8211272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003646"},"pg_stats":[{"pgid":"1.0","version":"165'4","reported_seq":149,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:49.596655+0000","last_change":"2022-07-20T03:36:49.596833+0000","last_active":"2022-07-20T03:36:49.596655+0000","last_peered":"2022-07-20T03:36:49.596655+0000","last_clean":"2022-07-20T03:36:49.596655+0000","last_became_active":"2022-07-20T02:25:25.351806+0000","last_became_peered":"2022-07-20T02:25:25.351806+0000","last_unstale":"2022-07-20T03:36:49.596655+0000","last_undegraded":"2022-07-20T03:36:49.596655+0000","last_fullsized":"2022-07-20T03:36:49.596655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'4","last_scrub_stamp":"2022-07-20T03:36:49.596654+0000","last_deep_scrub":"165'4","last_deep_scrub_stamp":"2022-07-20T03:36:49.596654+0000","last_clean_scrub_stamp":"2022-07-20T03:36:49.596654+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9464,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":148,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:48.930059+0000","last_change":"2022-07-20T03:36:48.930252+0000","last_active":"2022-07-20T03:36:48.930059+0000","last_peered":"2022-07-20T03:36:48.930059+0000","last_clean":"2022-07-20T03:36:48.930059+0000","last_became_active":"2022-07-20T02:25:29.458580+0000","last_became_peered":"2022-07-20T02:25:29.458580+0000","last_unstale":"2022-07-20T03:36:48.930059+0000","last_undegraded":"2022-07-20T03:36:48.930059+0000","last_fullsized":"2022-07-20T03:36:48.930059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:48.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:48.930059+0000","last_clean_scrub_stamp":"2022-07-20T03:36:48.930059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:50.595792+0000","last_change":"2022-07-20T03:36:50.595984+0000","last_active":"2022-07-20T03:36:50.595792+0000","last_peered":"2022-07-20T03:36:50.595792+0000","last_clean":"2022-07-20T03:36:50.595792+0000","last_became_active":"2022-07-20T02:25:29.454947+0000","last_became_peered":"2022-07-20T02:25:29.454947+0000","last_unstale":"2022-07-20T03:36:50.595792+0000","last_undegraded":"2022-07-20T03:36:50.595792+0000","last_fullsized":"2022-07-20T03:36:50.595792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:50.595791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:50.595791+0000","last_clean_scrub_stamp":"2022-07-20T03:36:50.595791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":145,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:46.202122+0000","last_change":"2022-07-20T03:36:46.202202+0000","last_active":"2022-07-20T03:36:46.202122+0000","last_peered":"2022-07-20T03:36:46.202122+0000","last_clean":"2022-07-20T03:36:46.202122+0000","last_became_active":"2022-07-20T02:25:29.453725+0000","last_became_peered":"2022-07-20T02:25:29.453725+0000","last_unstale":"2022-07-20T03:36:46.202122+0000","last_undegraded":"2022-07-20T03:36:46.202122+0000","last_fullsized":"2022-07-20T03:36:46.202122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:46.202122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:46.202122+0000","last_clean_scrub_stamp":"2022-07-20T03:36:46.202122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":148,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:46.945160+0000","last_change":"2022-07-20T03:36:46.945234+0000","last_active":"2022-07-20T03:36:46.945160+0000","last_peered":"2022-07-20T03:36:46.945160+0000","last_clean":"2022-07-20T03:36:46.945160+0000","last_became_active":"2022-07-20T02:25:29.457326+0000","last_became_peered":"2022-07-20T02:25:29.457326+0000","last_unstale":"2022-07-20T03:36:46.945160+0000","last_undegraded":"2022-07-20T03:36:46.945160+0000","last_fullsized":"2022-07-20T03:36:46.945160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:46.945159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:46.945159+0000","last_clean_scrub_stamp":"2022-07-20T03:36:46.945159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":144,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:44.342500+0000","last_change":"2022-07-20T03:36:44.342614+0000","last_active":"2022-07-20T03:36:44.342500+0000","last_peered":"2022-07-20T03:36:44.342500+0000","last_clean":"2022-07-20T03:36:44.342500+0000","last_became_active":"2022-07-20T02:25:29.901376+0000","last_became_peered":"2022-07-20T02:25:29.901376+0000","last_unstale":"2022-07-20T03:36:44.342500+0000","last_undegraded":"2022-07-20T03:36:44.342500+0000","last_fullsized":"2022-07-20T03:36:44.342500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:44.342499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:44.342499+0000","last_clean_scrub_stamp":"2022-07-20T03:36:44.342499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:56.637419+0000","last_change":"2022-07-20T03:36:56.637731+0000","last_active":"2022-07-20T03:36:56.637419+0000","last_peered":"2022-07-20T03:36:56.637419+0000","last_clean":"2022-07-20T03:36:56.637419+0000","last_became_active":"2022-07-20T02:25:29.908425+0000","last_became_peered":"2022-07-20T02:25:29.908425+0000","last_unstale":"2022-07-20T03:36:56.637419+0000","last_undegraded":"2022-07-20T03:36:56.637419+0000","last_fullsized":"2022-07-20T03:36:56.637419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:56.637418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:56.637418+0000","last_clean_scrub_stamp":"2022-07-20T03:36:56.637418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":144,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:46.344838+0000","last_change":"2022-07-20T03:36:46.344926+0000","last_active":"2022-07-20T03:36:46.344838+0000","last_peered":"2022-07-20T03:36:46.344838+0000","last_clean":"2022-07-20T03:36:46.344838+0000","last_became_active":"2022-07-20T02:25:29.451566+0000","last_became_peered":"2022-07-20T02:25:29.451566+0000","last_unstale":"2022-07-20T03:36:46.344838+0000","last_undegraded":"2022-07-20T03:36:46.344838+0000","last_fullsized":"2022-07-20T03:36:46.344838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:46.344838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:46.344838+0000","last_clean_scrub_stamp":"2022-07-20T03:36:46.344838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2022-07-20T03:36:46.538454+0000","last_change":"2022-07-20T03:36:46.538529+0000","last_active":"2022-07-20T03:36:46.538454+0000","last_peered":"2022-07-20T03:36:46.538454+0000","last_clean":"2022-07-20T03:36:46.538454+0000","last_became_active":"2022-07-20T02:25:29.458000+0000","last_became_peered":"2022-07-20T02:25:29.458000+0000","last_unstale":"2022-07-20T03:36:46.538454+0000","last_undegraded":"2022-07-20T03:36:46.538454+0000","last_fullsized":"2022-07-20T03:36:46.538454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:36:46.538454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:36:46.538454+0000","last_clean_scrub_stamp":"2022-07-20T03:36:46.538454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9464,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575710,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.83199999999999996,"15min":0.85599999999999998},"min":{"1min":0.57899999999999996,"5min":0.47099999999999997,"15min":0.43099999999999999},"max":{"1min":1.0209999999999999,"5min":1.089,"15min":1.1739999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.67000000000000004,"5min":0.56899999999999995,"15min":0.52400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.2430000000000001},"last":0.92000000000000004}]},{"osd":1,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.78200000000000003,"15min":0.78400000000000003},"min":{"1min":0.57599999999999996,"5min":0.54500000000000004,"15min":0.52000000000000002},"max":{"1min":1.097,"5min":1.208,"15min":1.2749999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.56899999999999995},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1830000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.874,"15min":0.873},"min":{"1min":0.63700000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.052,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91200000000000003,"15min":0.93000000000000005},"min":{"1min":0.73199999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.125,"5min":1.135,"15min":1.24},"last":1.1439999999999999}]},{"osd":3,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.82199999999999995,"15min":0.82799999999999996},"min":{"1min":0.70999999999999996,"5min":0.63700000000000001,"15min":0.58799999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.1399999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.87,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.54700000000000004},"max":{"1min":1.1759999999999999,"5min":1.244,"15min":1.244},"last":0.97599999999999998}]},{"osd":4,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72799999999999998,"15min":0.73299999999999998},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.50700000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.163},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.0409999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.88700000000000001,"15min":0.875},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.47299999999999998},"max":{"1min":1.133,"5min":1.1719999999999999,"15min":1.355},"last":1.097},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.96199999999999997,"15min":0.94899999999999995},"min":{"1min":0.70899999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1279999999999999,"5min":1.2629999999999999,"15min":1.266},"last":1.0760000000000001}]},{"osd":6,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92500000000000004,"15min":0.90200000000000002},"min":{"1min":0.63500000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.208,"15min":1.228},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.93999999999999995,"15min":0.91200000000000003},"min":{"1min":0.66500000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.165,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.89500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575710,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":1.163,"15min":1.1739999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51200000000000001},"max":{"1min":1.0509999999999999,"5min":1.119,"15min":1.165},"last":0.73499999999999999}]},{"osd":1,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84699999999999998,"15min":0.82399999999999995},"min":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.036,"5min":1.149,"15min":1.2609999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.84199999999999997,"15min":0.85799999999999998},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60099999999999998},"max":{"1min":1.111,"5min":1.1200000000000001,"15min":1.1970000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81000000000000005,"15min":0.80700000000000005},"min":{"1min":0.624,"5min":0.59199999999999997,"15min":0.58299999999999996},"max":{"1min":1.004,"5min":1.085,"15min":1.272},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.86499999999999999,"15min":0.86699999999999999},"min":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.0269999999999999,"5min":1.121,"15min":1.2989999999999999},"last":0.99099999999999999}]},{"osd":3,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.874,"15min":0.86599999999999999},"min":{"1min":0.74299999999999999,"5min":0.626,"15min":0.54400000000000004},"max":{"1min":1.097,"5min":1.1639999999999999,"15min":1.2090000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.62,"15min":0.61799999999999999},"max":{"1min":0.98099999999999998,"5min":1.069,"15min":1.179},"last":0.879}]},{"osd":4,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76400000000000001,"15min":0.77400000000000002},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51500000000000001},"max":{"1min":1.121,"5min":1.1839999999999999,"15min":1.2609999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84999999999999998,"15min":0.83699999999999997},"min":{"1min":0.60099999999999998,"5min":0.56399999999999995,"15min":0.38700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81399999999999995,"15min":0.82099999999999995},"min":{"1min":0.60799999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.038,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.56100000000000005},"max":{"1min":1.077,"5min":1.327,"15min":1.3360000000000001},"last":1.0549999999999999}]},{"osd":7,"last update":"Wed Jul 20 03:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93500000000000005,"15min":0.94399999999999995},"min":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.436},"max":{"1min":1.403,"5min":1.403,"15min":1.4379999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.80900000000000005,"5min":0.63800000000000001,"15min":0.56000000000000005},"max":{"1min":1.083,"5min":1.2050000000000001,"15min":1.5389999999999999},"last":1.032}]}]},{"osd":5,"up_from":13,"seq":55834575710,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.81799999999999995,"15min":0.80900000000000005},"min":{"1min":0.59299999999999997,"5min":0.57299999999999995,"15min":0.441},"max":{"1min":1.125,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.61499999999999999,"15min":0.59899999999999998},"max":{"1min":1.1200000000000001,"5min":1.1220000000000001,"15min":1.1779999999999999},"last":0.98699999999999999}]},{"osd":1,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.83499999999999996},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.57499999999999996},"max":{"1min":1.04,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.84999999999999998,"15min":0.84599999999999997},"min":{"1min":0.76800000000000002,"5min":0.625,"15min":0.60399999999999998},"max":{"1min":0.99399999999999999,"5min":1.073,"15min":1.149},"last":0.68899999999999995}]},{"osd":2,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85699999999999998,"15min":0.86899999999999999},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.621},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86599999999999999,"15min":0.877},"min":{"1min":0.65700000000000003,"5min":0.61399999999999999,"15min":0.61199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.2210000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86799999999999999,"15min":0.88900000000000001},"min":{"1min":0.70199999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.0109999999999999,"5min":1.232,"15min":1.232},"last":1.149},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91300000000000003,"15min":0.91000000000000003},"min":{"1min":0.78700000000000003,"5min":0.65900000000000003,"15min":0.63800000000000001},"max":{"1min":1.111,"5min":1.3,"15min":1.3},"last":0.84299999999999997}]},{"osd":4,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.94299999999999995,"15min":0.92000000000000004},"min":{"1min":0.68200000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1950000000000001,"5min":1.242,"15min":1.268},"last":1.0960000000000001}]},{"osd":6,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92900000000000005,"15min":0.90800000000000003},"min":{"1min":0.65300000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2549999999999999,"5min":1.357,"15min":1.357},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.67900000000000005,"5min":0.63200000000000001,"15min":0.59399999999999997},"max":{"1min":1.21,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.032}]},{"osd":7,"last update":"Wed Jul 20 03:36:36 2022","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.995,"15min":0.98999999999999999},"min":{"1min":0.82899999999999996,"5min":0.66600000000000004,"15min":0.63800000000000001},"max":{"1min":1.216,"5min":1.2929999999999999,"15min":1.4350000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.042,"15min":1.0229999999999999},"min":{"1min":0.82499999999999996,"5min":0.70299999999999996,"15min":0.60099999999999998},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.96399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.53000000000000003},"max":{"1min":1.081,"5min":1.101,"15min":1.149},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.60699999999999998,"5min":0.59199999999999997,"15min":0.56599999999999995},"max":{"1min":1.0980000000000001,"5min":1.1339999999999999,"15min":1.2150000000000001},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.83299999999999996},"min":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.59799999999999998},"max":{"1min":1.0620000000000001,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.878,"15min":0.86299999999999999},"min":{"1min":0.63900000000000001,"5min":0.61399999999999999,"15min":0.59399999999999997},"max":{"1min":1.0760000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.83699999999999997}]},{"osd":2,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.90400000000000003,"15min":0.88400000000000001},"min":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.58899999999999997},"max":{"1min":1.1779999999999999,"5min":1.1919999999999999,"15min":1.2390000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89200000000000002,"15min":0.90000000000000002},"min":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.59799999999999998},"max":{"1min":1.171,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.996}]},{"osd":3,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.84399999999999997,"15min":0.85099999999999998},"min":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.60599999999999998},"max":{"1min":1.198,"5min":1.198,"15min":1.248},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92900000000000005,"15min":0.92400000000000004},"min":{"1min":0.68000000000000005,"5min":0.67800000000000005,"15min":0.65500000000000003},"max":{"1min":1.181,"5min":1.236,"15min":1.25},"last":0.97599999999999998}]},{"osd":5,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.86699999999999999,"15min":0.85499999999999998},"min":{"1min":0.58799999999999997,"5min":0.56699999999999995,"15min":0.54400000000000004},"max":{"1min":1.2030000000000001,"5min":1.268,"15min":1.268},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.65300000000000002,"5min":0.58899999999999997,"15min":0.55300000000000005},"max":{"1min":1.272,"5min":1.2729999999999999,"15min":1.413},"last":0.64500000000000002}]},{"osd":6,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.89800000000000002,"15min":0.91300000000000003},"min":{"1min":0.68100000000000005,"5min":0.623,"15min":0.60399999999999998},"max":{"1min":1.125,"5min":1.2390000000000001,"15min":1.2769999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.98299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.64400000000000002},"max":{"1min":1.2250000000000001,"5min":1.339,"15min":1.339},"last":0.89100000000000001}]},{"osd":7,"last update":"Wed Jul 20 03:36:23 2022","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.79300000000000004,"5min":0.65800000000000003,"15min":0.55600000000000005},"max":{"1min":1.3149999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.749},{"interface":"front","average":{"1min":1.107,"5min":1.069,"15min":1.044},"min":{"1min":0.90300000000000002,"5min":0.70599999999999996,"15min":0.67300000000000004},"max":{"1min":1.444,"5min":1.4510000000000001,"15min":1.482},"last":1.0409999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61199999999999999,"15min":0.629},"min":{"1min":0.48099999999999998,"5min":0.45900000000000002,"15min":0.45800000000000002},"max":{"1min":0.91900000000000004,"5min":0.995,"15min":1.0620000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.68300000000000005},"min":{"1min":0.51800000000000002,"5min":0.50700000000000001,"15min":0.46000000000000002},"max":{"1min":1.016,"5min":1.0620000000000001,"15min":1.1299999999999999},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65400000000000003,"15min":0.66700000000000004},"min":{"1min":0.502,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.139,"5min":1.139,"15min":1.236},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.76200000000000001},"min":{"1min":0.56799999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0329999999999999,"5min":1.054,"15min":1.2150000000000001},"last":0.747}]},{"osd":2,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.83399999999999996,"15min":0.84299999999999997},"min":{"1min":0.60299999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1080000000000001,"5min":1.1140000000000001,"15min":1.2330000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89300000000000002,"15min":0.89700000000000002},"min":{"1min":0.72099999999999997,"5min":0.63800000000000001,"15min":0.55500000000000005},"max":{"1min":1.113,"5min":1.143,"15min":1.206},"last":0.89100000000000001}]},{"osd":4,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.876,"15min":0.89500000000000002},"min":{"1min":0.70699999999999996,"5min":0.70199999999999996,"15min":0.57899999999999996},"max":{"1min":1.107,"5min":1.4179999999999999,"15min":1.6359999999999999},"last":1.012},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93500000000000005,"15min":0.95599999999999996},"min":{"1min":0.71899999999999997,"5min":0.68899999999999995,"15min":0.63200000000000001},"max":{"1min":1.292,"5min":1.337,"15min":1.579},"last":0.93899999999999995}]},{"osd":5,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.97999999999999998},"min":{"1min":0.73499999999999999,"5min":0.69099999999999995,"15min":0.68600000000000005},"max":{"1min":1.367,"5min":1.367,"15min":1.4690000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1,"15min":1.0329999999999999},"min":{"1min":0.75600000000000001,"5min":0.70099999999999996,"15min":0.69799999999999995},"max":{"1min":1.4299999999999999,"5min":1.601,"15min":1.6379999999999999},"last":0.95799999999999996}]},{"osd":6,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.998,"15min":1.0069999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.69499999999999995},"max":{"1min":1.2430000000000001,"5min":1.534,"15min":1.534},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.046,"15min":1.0640000000000001},"min":{"1min":0.89700000000000002,"5min":0.749,"15min":0.72799999999999998},"max":{"1min":1.284,"5min":1.5069999999999999,"15min":1.661},"last":1.036}]},{"osd":7,"last update":"Wed Jul 20 03:36:58 2022","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1060000000000001},"min":{"1min":0.91500000000000004,"5min":0.79000000000000004,"15min":0.74299999999999999},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.681},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.087,"15min":1.097},"min":{"1min":0.89500000000000002,"5min":0.879,"15min":0.76000000000000001},"max":{"1min":1.264,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.069}]}]},{"osd":2,"up_from":13,"seq":55834575710,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.45400000000000001,"15min":0.433},"max":{"1min":0.86799999999999999,"5min":1.0529999999999999,"15min":1.1080000000000001},"last":0.627},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.72499999999999998},"min":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.872,"5min":1.121,"15min":1.2270000000000001},"last":0.89500000000000002}]},{"osd":1,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64700000000000002,"15min":0.65200000000000002},"min":{"1min":0.51200000000000001,"5min":0.48499999999999999,"15min":0.46200000000000002},"max":{"1min":0.85999999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.749},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71999999999999997,"15min":0.73799999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.50900000000000001},"max":{"1min":0.86099999999999999,"5min":1.151,"15min":1.194},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.84399999999999997,"15min":0.84799999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.004,"5min":1.1910000000000001,"15min":1.2470000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.76700000000000002,"15min":0.78800000000000003},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.52400000000000002},"max":{"1min":1.0369999999999999,"5min":1.1419999999999999,"15min":1.351},"last":0.81399999999999995}]},{"osd":4,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.88700000000000001,"15min":0.88300000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.60499999999999998},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.028,"5min":0.96199999999999997,"15min":0.95899999999999996},"min":{"1min":0.83599999999999997,"5min":0.73299999999999998,"15min":0.63},"max":{"1min":1.294,"5min":1.385,"15min":1.4179999999999999},"last":1.2829999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.77700000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.27,"5min":1.3640000000000001,"15min":1.546},"last":1.246},{"interface":"front","average":{"1min":1.016,"5min":0.99099999999999999,"15min":0.998},"min":{"1min":0.71899999999999997,"5min":0.70099999999999996,"15min":0.68300000000000005},"max":{"1min":1.3100000000000001,"5min":1.4830000000000001,"15min":1.645},"last":1.069}]},{"osd":6,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.0529999999999999,"15min":1.052},"min":{"1min":0.86099999999999999,"5min":0.75,"15min":0.66700000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0660000000000001,"15min":1.073},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.482,"5min":1.611,"15min":1.611},"last":1.347}]},{"osd":7,"last update":"Wed Jul 20 03:36:46 2022","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.103,"15min":1.107},"min":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.78200000000000003},"max":{"1min":1.381,"5min":1.4990000000000001,"15min":2.0960000000000001},"last":1.202},{"interface":"front","average":{"1min":1.105,"5min":1.1140000000000001,"15min":1.127},"min":{"1min":0.95699999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.4119999999999999,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.302}]}]},{"osd":1,"up_from":13,"seq":55834575710,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.046},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75,"15min":0.76100000000000001},"min":{"1min":0.63800000000000001,"5min":0.59099999999999997,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":1.004,"15min":1.181},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69099999999999995,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.496},"max":{"1min":0.91900000000000004,"5min":1.085,"15min":1.147},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73099999999999998,"15min":0.70799999999999996},"min":{"1min":0.55000000000000004,"5min":0.52300000000000002,"15min":0.47399999999999998},"max":{"1min":0.96099999999999997,"5min":1.139,"15min":1.139},"last":0.63700000000000001}]},{"osd":3,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78400000000000003,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70799999999999996,"15min":0.72499999999999998},"min":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"max":{"1min":0.93799999999999994,"5min":1.123,"15min":1.171},"last":0.78100000000000003}]},{"osd":4,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93400000000000005,"15min":0.93700000000000006},"min":{"1min":0.76000000000000001,"5min":0.70999999999999996,"15min":0.65900000000000003},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3},"last":1.05},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.92800000000000005,"15min":0.91800000000000004},"min":{"1min":0.746,"5min":0.71899999999999997,"15min":0.66900000000000004},"max":{"1min":1.1679999999999999,"5min":1.2190000000000001,"15min":1.3169999999999999},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.95199999999999996,"15min":0.96699999999999997},"min":{"1min":0.84399999999999997,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.395},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.044,"15min":1.05},"min":{"1min":0.95299999999999996,"5min":0.84499999999999997,"15min":0.64200000000000002},"max":{"1min":1.1919999999999999,"5min":1.3799999999999999,"15min":1.486},"last":1.1100000000000001}]},{"osd":6,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.012,"15min":1.0389999999999999},"min":{"1min":0.79000000000000004,"5min":0.69099999999999995,"15min":0.65800000000000003},"max":{"1min":1.2190000000000001,"5min":1.444,"15min":1.589},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.018,"15min":1.012},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69299999999999995},"max":{"1min":1.2989999999999999,"5min":1.653,"15min":1.653},"last":1.0680000000000001}]},{"osd":7,"last update":"Wed Jul 20 03:36:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.018,"15min":1.0429999999999999},"min":{"1min":0.83199999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.248,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.036},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.111,"15min":1.123},"min":{"1min":0.94899999999999995,"5min":0.84499999999999997,"15min":0.71099999999999997},"max":{"1min":1.2589999999999999,"5min":1.411,"15min":1.544},"last":1.264}]}]},{"osd":0,"up_from":13,"seq":55834575710,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050636,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321204,"statfs":{"total":96636764160,"available":95560912896,"internally_reserved":1073741824,"allocated":2101248,"data_stored":1026409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.51500000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.80800000000000005,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.52800000000000002},"max":{"1min":0.91800000000000004,"5min":1.0820000000000001,"15min":1.1779999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64900000000000002,"15min":0.63700000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.48699999999999999},"max":{"1min":0.88500000000000001,"5min":0.96999999999999997,"15min":1.089},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.53200000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.92700000000000005,"5min":1.161,"15min":1.1639999999999999},"last":0.69099999999999995}]},{"osd":3,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.55200000000000005,"5min":0.53100000000000003,"15min":0.504},"max":{"1min":1.085,"5min":1.1559999999999999,"15min":1.1719999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77300000000000002,"15min":0.76200000000000001},"min":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":1.0549999999999999,"5min":1.177,"15min":1.177},"last":0.621}]},{"osd":4,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.93799999999999994,"15min":0.93500000000000005},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.96199999999999997,"15min":0.95999999999999996},"min":{"1min":0.749,"5min":0.71099999999999997,"15min":0.69199999999999995},"max":{"1min":1.29,"5min":1.29,"15min":1.3400000000000001},"last":0.80800000000000005}]},{"osd":5,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96699999999999997,"15min":0.98499999999999999},"min":{"1min":0.78700000000000003,"5min":0.69499999999999995,"15min":0.66800000000000004},"max":{"1min":1.2390000000000001,"5min":1.3220000000000001,"15min":1.3560000000000001},"last":0.747},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97299999999999998,"15min":0.96199999999999997},"min":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.65800000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.52},"last":0.78200000000000003}]},{"osd":6,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0269999999999999,"15min":1.0129999999999999},"min":{"1min":0.80700000000000005,"5min":0.72799999999999998,"15min":0.624},"max":{"1min":1.329,"5min":1.3420000000000001,"15min":1.4690000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.075,"5min":1.0369999999999999,"15min":1.056},"min":{"1min":0.93799999999999994,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.615},"last":1.022}]},{"osd":7,"last update":"Wed Jul 20 03:36:01 2022","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.069,"15min":1.077},"min":{"1min":0.95299999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.268,"5min":1.401,"15min":1.7050000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.091,"15min":1.1020000000000001},"min":{"1min":0.89800000000000002,"5min":0.82399999999999995,"15min":0.78600000000000003},"max":{"1min":1.379,"5min":1.419,"15min":1.6399999999999999},"last":0.876}]}]}]}} 2022-07-20T03:37:07.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-07-20T03:37:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.674+0000 7f9c1bfff700 1 Processor -- start 2022-07-20T03:37:07.676 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.674+0000 7f9c1bfff700 1 -- start start 2022-07-20T03:37:07.690 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.674+0000 7f9c1bfff700 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9c1c060f00 0x7f9c1c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.674+0000 7f9c1bfff700 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c1474e0 con 0x7f9c1c147000 2022-07-20T03:37:07.692 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c0b4e40 con 0x7f9c1c05cfd0 2022-07-20T03:37:07.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c05d3b0 con 0x7f9c1c060f00 2022-07-20T03:37:07.693 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1a7fc700 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9c1c147000 0x7f9c1c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44456/0 (socket says 172.21.15.26:44456) 2022-07-20T03:37:07.694 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1a7fc700 1 -- 172.21.15.26:0/2340095519 learned_addr learned my addr 172.21.15.26:0/2340095519 (peer_addr_for_me v1:172.21.15.26:0/0) 2022-07-20T03:37:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/2340095519 >> v2:172.21.15.120:3300/0 conn(0x7f9c1c060f00 0x7f9c1c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854324184 0 0) 0x7f9c1c1474e0 con 0x7f9c1c147000 2022-07-20T03:37:07.706 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc003260 con 0x7f9c1c147000 2022-07-20T03:37:07.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662839707 0 0) 0x7f9c1c0b4e40 con 0x7f9c1c05cfd0 2022-07-20T03:37:07.707 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c1474e0 con 0x7f9c1c05cfd0 2022-07-20T03:37:07.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365240845 0 0) 0x7f9bfc003260 con 0x7f9c1c147000 2022-07-20T03:37:07.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 >> v2:172.21.15.120:3300/0 conn(0x7f9c1c060f00 msgr2=0x7f9c1c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:37:07.708 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 --2- 172.21.15.26:0/2340095519 >> v2:172.21.15.120:3300/0 conn(0x7f9c1c060f00 0x7f9c1c143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 >> v1:172.21.15.120:6789/0 conn(0x7f9c1c05cfd0 legacy=0x7f9c1c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.709 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c1431e0 con 0x7f9c1c147000 2022-07-20T03:37:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 207990462 0 0) 0x7f9c04001660 con 0x7f9c1c147000 2022-07-20T03:37:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 >> v1:172.21.15.26:6789/0 conn(0x7f9c1c147000 legacy=0x7f9c1c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bfc003930 0x7f9bfc003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bfc004500 0x7f9bfc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 0x7f9bfc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.711 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c05d3b0 con 0x7f9bfc004500 2022-07-20T03:37:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00bd20 con 0x7f9bfc003930 2022-07-20T03:37:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00bee0 con 0x7f9bfc004e70 2022-07-20T03:37:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1affd700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bfc003930 0x7f9bfc003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.713 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1a7fc700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bfc004500 0x7f9bfc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 0x7f9bfc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bfc003930 msgr2=0x7f9bfc003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:37:07.714 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bfc003930 0x7f9bfc003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/2340095519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bfc004500 msgr2=0x7f9bfc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:37:07.715 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bfc004500 0x7f9bfc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/2340095519 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc003260 con 0x7f9bfc004e70 2022-07-20T03:37:07.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1affd700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bfc003930 0x7f9bfc003d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:37:07.716 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1a7fc700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9bfc004500 0x7f9bfc004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:37:07.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/2340095519 reap_dead start 2022-07-20T03:37:07.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 0x7f9bfc00b750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f9c1400c000 tx=0x7f9c14006df0).ready entity=mon.2 client_cookie=9fcae9368600fe5a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.717 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c1400be60 con 0x7f9bfc004e70 2022-07-20T03:37:07.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1400e640 con 0x7f9bfc004e70 2022-07-20T03:37:07.718 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c19ffb700 1 -- 172.21.15.26:0/2340095519 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c1400e9a0 con 0x7f9bfc004e70 2022-07-20T03:37:07.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 msgr2=0x7f9bfc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 0x7f9bfc00b750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f9c1400c000 tx=0x7f9c14006df0).stop 2022-07-20T03:37:07.719 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- 172.21.15.26:0/2340095519 shutdown_connections 2022-07-20T03:37:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/2340095519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bfc004e70 0x7f9bfc00b750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- 172.21.15.26:0/2340095519 >> 172.21.15.26:0/2340095519 conn(0x7f9c1c0620c0 msgr2=0x7f9c1c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:37:07.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- 172.21.15.26:0/2340095519 shutdown_connections 2022-07-20T03:37:07.725 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.678+0000 7f9c1bfff700 1 -- 172.21.15.26:0/2340095519 wait complete. 2022-07-20T03:37:07.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 Processor -- start 2022-07-20T03:37:07.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- start start 2022-07-20T03:37:07.726 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c1c146920 0x7f9c1c152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.727 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c1c141fb0 0x7f9c1c152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.728 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c1455f0 con 0x7f9c1c152f20 2022-07-20T03:37:07.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c146d00 con 0x7f9c1c146920 2022-07-20T03:37:07.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c142e30 con 0x7f9c1c141fb0 2022-07-20T03:37:07.729 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45188/0 (socket says 172.21.15.26:45188) 2022-07-20T03:37:07.730 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/3177169906 learned_addr learned my addr 172.21.15.26:0/3177169906 (peer_addr_for_me v2:172.21.15.26:0/0) 2022-07-20T03:37:07.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1a7fc700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c1c141fb0 0x7f9c1c152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.731 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1affd700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c1c146920 0x7f9c1c152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c1c141fb0 msgr2=0x7f9c1c152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.732 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c1c141fb0 0x7f9c1c152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c1c146920 msgr2=0x7f9c1c152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c1c146920 0x7f9c1c152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.733 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 -- 172.21.15.26:0/3177169906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c141490 con 0x7f9c1c152f20 2022-07-20T03:37:07.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1b7fe700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f9c1400c000 tx=0x7f9c14018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.734 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c14022070 con 0x7f9c1c152f20 2022-07-20T03:37:07.735 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14000e90 con 0x7f9c1c152f20 2022-07-20T03:37:07.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c144e30 con 0x7f9c1c152f20 2022-07-20T03:37:07.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c14012d60 con 0x7f9c1c152f20 2022-07-20T03:37:07.736 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.682+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c1416f0 con 0x7f9c1c152f20 2022-07-20T03:37:07.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c143480 con 0x7f9c1c152f20 2022-07-20T03:37:07.737 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9c1401d420 con 0x7f9c1c152f20 2022-07-20T03:37:07.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c13fff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 0x7f9c00055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:37:07.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f9c1406d130 con 0x7f9c1c152f20 2022-07-20T03:37:07.739 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c1affd700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 0x7f9c00055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:37:07.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.686+0000 7f9c1affd700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 0x7f9c00055ff0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f9c0c003000 tx=0x7f9c0c007d80).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:37:07.740 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.690+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c14044170 con 0x7f9c1c152f20 2022-07-20T03:37:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.822+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f9c1c0574e0 con 0x7f9c1c152f20 2022-07-20T03:37:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.830+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9c14044170 con 0x7f9c1c152f20 2022-07-20T03:37:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.830+0000 7f9c13fff700 1 -- 172.21.15.26:0/3177169906 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) v1 ==== 123+0+0 (secure 0 0 0) 0x7f9c1400be60 con 0x7f9c1c152f20 2022-07-20T03:37:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 msgr2=0x7f9c00055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 0x7f9c00055ff0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f9c0c003000 tx=0x7f9c0c007d80).stop 2022-07-20T03:37:07.840 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 msgr2=0x7f9c1c155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:37:07.863 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f9c1400c000 tx=0x7f9c14018000).stop 2022-07-20T03:37:07.864 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 shutdown_connections 2022-07-20T03:37:07.865 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9c1c152f20 0x7f9c1c155960 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.874 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.26:6800/22865,v1:172.21.15.26:6801/22865] conn(0x7f9c00053b30 0x7f9c00055ff0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c1c141fb0 0x7f9c1c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.875 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 --2- 172.21.15.26:0/3177169906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c1c146920 0x7f9c1c152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:37:07.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 >> 172.21.15.26:0/3177169906 conn(0x7f9bfc004e70 msgr2=0x7f9c1c13a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:37:07.876 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 shutdown_connections 2022-07-20T03:37:07.877 INFO:teuthology.orchestra.run.smithi026.stderr:2022-07-20T03:37:07.834+0000 7f9c1bfff700 1 -- 172.21.15.26:0/3177169906 wait complete. 2022-07-20T03:37:07.879 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph.py", line 1858, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:37:07.901 INFO:teuthology.misc:Shutting down mds daemons... 2022-07-20T03:37:07.903 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:37:07.904 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:07.924 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:37:07.926 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-20T03:37:07.926 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:07.992 INFO:tasks.ceph.mds.c:Stopped 2022-07-20T03:37:07.993 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:37:07.994 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.016 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:37:08.019 INFO:teuthology.misc:Shutting down osd daemons... 2022-07-20T03:37:08.020 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-07-20T03:37:08.020 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.089 INFO:tasks.ceph.osd.0:Stopped 2022-07-20T03:37:08.090 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-07-20T03:37:08.091 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.172 INFO:tasks.ceph.osd.1:Stopped 2022-07-20T03:37:08.173 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-07-20T03:37:08.175 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.247 INFO:tasks.ceph.osd.2:Stopped 2022-07-20T03:37:08.248 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2022-07-20T03:37:08.250 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.351 INFO:tasks.ceph.osd.3:Stopped 2022-07-20T03:37:08.352 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2022-07-20T03:37:08.353 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.433 INFO:tasks.ceph.osd.4:Stopped 2022-07-20T03:37:08.434 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-07-20T03:37:08.442 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.497 INFO:tasks.ceph.osd.5:Stopped 2022-07-20T03:37:08.499 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2022-07-20T03:37:08.502 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.571 INFO:tasks.ceph.osd.6:Stopped 2022-07-20T03:37:08.578 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2022-07-20T03:37:08.578 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.652 INFO:tasks.ceph.osd.7:Stopped 2022-07-20T03:37:08.654 INFO:teuthology.misc:Shutting down mgr daemons... 2022-07-20T03:37:08.656 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2022-07-20T03:37:08.657 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.684 INFO:tasks.ceph.mgr.y:Stopped 2022-07-20T03:37:08.686 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2022-07-20T03:37:08.698 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.715 INFO:tasks.ceph.mgr.x:Stopped 2022-07-20T03:37:08.717 INFO:teuthology.misc:Shutting down mon daemons... 2022-07-20T03:37:08.718 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-07-20T03:37:08.720 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.756 INFO:tasks.ceph.mon.a:Stopped 2022-07-20T03:37:08.758 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2022-07-20T03:37:08.760 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.778 INFO:tasks.ceph.mon.b:Stopped 2022-07-20T03:37:08.780 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2022-07-20T03:37:08.781 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:37:08.847 INFO:tasks.ceph.mon.c:Stopped 2022-07-20T03:37:08.849 INFO:tasks.ceph:Checking cluster log for badness... 2022-07-20T03:37:08.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'has not responded to cap revoke by MDS for over' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 'responding to mclientcaps' | head -n 1 2022-07-20T03:37:08.879 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T03:37:08.880 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-07-20T03:37:09.200 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T03:37:09.202 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-07-20T03:37:09.643 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T03:37:09.644 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-07-20T03:37:09.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T03:37:09.883 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2022-07-20T03:37:10.067 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T03:37:10.068 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2022-07-20T03:37:10.227 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T03:37:10.231 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2022-07-20T03:37:10.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T03:37:10.478 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2022-07-20T03:37:10.660 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T03:37:10.665 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2022-07-20T03:37:10.853 INFO:tasks.ceph:Archiving mon data... 2022-07-20T03:37:10.860 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/data/mon.a.tgz 2022-07-20T03:37:10.861 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2022-07-20T03:37:10.869 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.deXcim46r8 2022-07-20T03:37:10.872 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.deXcim46r8 2022-07-20T03:37:11.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.deXcim46r8 2022-07-20T03:37:11.190 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.deXcim46r8 is 537KB 2022-07-20T03:37:11.317 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.deXcim46r8 2022-07-20T03:37:11.339 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/data/mon.b.tgz 2022-07-20T03:37:11.341 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T03:37:11.402 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.k9tX8r53E3 2022-07-20T03:37:11.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.k9tX8r53E3 2022-07-20T03:37:11.695 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.k9tX8r53E3 2022-07-20T03:37:11.798 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.k9tX8r53E3 is 702KB 2022-07-20T03:37:11.887 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.k9tX8r53E3 2022-07-20T03:37:11.896 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/data/mon.c.tgz 2022-07-20T03:37:11.900 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-07-20T03:37:11.955 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.dbX7ezBEct 2022-07-20T03:37:11.957 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dbX7ezBEct 2022-07-20T03:37:12.201 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.dbX7ezBEct 2022-07-20T03:37:12.224 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.dbX7ezBEct is 702KB 2022-07-20T03:37:12.323 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.dbX7ezBEct 2022-07-20T03:37:12.333 INFO:tasks.ceph:Cleaning ceph cluster... 2022-07-20T03:37:12.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T03:37:12.342 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T03:37:12.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T03:37:12.396 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T03:37:12.415 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-07-20T03:37:12.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T03:37:12.432 DEBUG:teuthology.orchestra.run.smithi120:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T03:37:12.448 DEBUG:teuthology.orchestra.run.smithi159:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T03:37:12.454 DEBUG:teuthology.orchestra.run.smithi176:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T03:37:12.470 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T03:37:12.481 INFO:teuthology.orchestra.run.smithi120.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T03:37:12.531 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T03:37:12.543 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T03:37:12.554 INFO:tasks.ceph:Archiving crash dumps... 2022-07-20T03:37:12.567 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi026/crash 2022-07-20T03:37:12.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T03:37:12.600 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi120/crash 2022-07-20T03:37:12.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T03:37:12.630 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi159/crash 2022-07-20T03:37:12.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T03:37:12.664 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi176/crash 2022-07-20T03:37:12.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T03:37:12.708 INFO:tasks.ceph:Shutting down logrotate 2022-07-20T03:37:12.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.777 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.849 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:12.946 INFO:tasks.ceph:Compressing logs... 2022-07-20T03:37:12.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:12.953 DEBUG:teuthology.orchestra.run.smithi120:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:12.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:12.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:29.235 INFO:tasks.ceph:Archiving logs... 2022-07-20T03:37:29.237 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi026/log 2022-07-20T03:37:29.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T03:37:36.404 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi120/log 2022-07-20T03:37:36.407 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T03:37:39.216 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi159/log 2022-07-20T03:37:39.221 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T03:37:39.701 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi176/log 2022-07-20T03:37:39.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T03:37:39.756 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-07-20T03:37:39.769 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:37:39.773 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-07-20T03:37:39.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T03:37:39.777 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T03:37:39.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T03:37:39.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T03:37:39.918 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-07-20T03:37:39.957 INFO:teuthology.task.clock:Checking final clock skew... 2022-07-20T03:37:39.959 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:37:39.974 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:37:39.977 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:37:39.981 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:37:39.991 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:37:39.992 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2022-07-20T03:37:39.992 INFO:teuthology.orchestra.run.smithi026.stdout:+hv01.front.sepi 216.240.36.24 3 u 35 128 377 0.151 1.888 0.420 2022-07-20T03:37:39.993 INFO:teuthology.orchestra.run.smithi026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 128 377 0.199 -4.264 0.653 2022-07-20T03:37:39.993 INFO:teuthology.orchestra.run.smithi026.stdout:+hv03.front.sepi 216.229.4.69 3 u 31 128 377 0.169 -3.343 0.723 2022-07-20T03:37:39.993 INFO:teuthology.orchestra.run.smithi026.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T03:37:40.012 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:37:40.013 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2022-07-20T03:37:40.014 INFO:teuthology.orchestra.run.smithi176.stdout:+hv01.front.sepi 216.240.36.24 3 u 33 128 377 0.173 2.341 0.127 2022-07-20T03:37:40.014 INFO:teuthology.orchestra.run.smithi176.stdout:*hv02.front.sepi 163.237.218.18 2 u 113 128 377 0.190 -3.458 0.174 2022-07-20T03:37:40.015 INFO:teuthology.orchestra.run.smithi176.stdout:+hv03.front.sepi 216.229.4.69 3 u 93 128 377 0.147 -3.510 0.181 2022-07-20T03:37:40.015 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T03:37:40.016 INFO:teuthology.orchestra.run.smithi159.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:37:40.017 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================== 2022-07-20T03:37:40.017 INFO:teuthology.orchestra.run.smithi159.stdout:+hv01.front.sepi 216.240.36.24 3 u 24 128 377 0.153 1.693 0.560 2022-07-20T03:37:40.018 INFO:teuthology.orchestra.run.smithi159.stdout:*hv02.front.sepi 163.237.218.18 2 u 33 128 377 0.159 -4.752 0.758 2022-07-20T03:37:40.018 INFO:teuthology.orchestra.run.smithi159.stdout:+hv03.front.sepi 216.229.4.69 3 u 125 128 377 0.153 -4.212 0.652 2022-07-20T03:37:40.020 INFO:teuthology.orchestra.run.smithi159.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T03:37:40.020 INFO:teuthology.orchestra.run.smithi120.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:37:40.029 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================== 2022-07-20T03:37:40.029 INFO:teuthology.orchestra.run.smithi120.stdout:+hv01.front.sepi 216.240.36.24 3 u 41 256 377 0.171 4.860 0.335 2022-07-20T03:37:40.030 INFO:teuthology.orchestra.run.smithi120.stdout:*hv02.front.sepi 163.237.218.18 2 u 43 256 377 0.182 -1.067 0.417 2022-07-20T03:37:40.031 INFO:teuthology.orchestra.run.smithi120.stdout:+hv03.front.sepi 216.229.4.69 3 u 183 256 377 0.207 -0.655 0.314 2022-07-20T03:37:40.032 INFO:teuthology.orchestra.run.smithi120.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T03:37:40.033 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-07-20T03:37:40.060 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-07-20T03:37:40.061 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-07-20T03:37:40.081 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-07-20T03:37:40.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-07-20T03:37:40.124 INFO:teuthology.task.internal:Duration was 4981.019365 seconds 2022-07-20T03:37:40.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-07-20T03:37:40.148 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-07-20T03:37:40.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T03:37:40.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T03:37:40.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T03:37:40.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T03:37:40.254 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-07-20T03:37:40.255 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2022-07-20T03:37:40.256 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T03:37:40.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2022-07-20T03:37:40.277 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T03:37:40.294 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2022-07-20T03:37:40.295 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T03:37:40.315 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2022-07-20T03:37:40.317 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T03:37:40.339 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-07-20T03:37:40.341 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:40.368 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:40.421 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:40.430 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T03:37:40.468 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-07-20T03:37:40.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T03:37:40.476 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T03:37:40.512 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T03:37:40.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T03:37:40.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-07-20T03:37:40.732 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-07-20T03:37:40.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T03:37:40.758 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T03:37:40.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T03:37:40.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T03:37:40.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-07-20T03:37:40.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.851 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.855 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.868 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2022-07-20T03:37:40.869 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2022-07-20T03:37:40.871 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2022-07-20T03:37:40.891 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2022-07-20T03:37:40.903 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:37:40.950 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:37:40.960 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:37:40.970 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T03:37:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:37:40.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-07-20T03:37:41.004 INFO:teuthology.task.internal:Transferring archived files... 2022-07-20T03:37:41.009 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi026 2022-07-20T03:37:41.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T03:37:41.111 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi120 2022-07-20T03:37:41.113 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T03:37:41.166 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi159 2022-07-20T03:37:41.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T03:37:41.219 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939288/remote/smithi176 2022-07-20T03:37:41.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T03:37:41.281 INFO:teuthology.task.internal:Removing archive directory... 2022-07-20T03:37:41.283 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T03:37:41.297 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T03:37:41.301 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T03:37:41.305 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T03:37:41.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-07-20T03:37:41.370 INFO:teuthology.task.internal:Not uploading archives. 2022-07-20T03:37:41.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-07-20T03:37:41.395 INFO:teuthology.task.internal:Tidying up after the test... 2022-07-20T03:37:41.398 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T03:37:41.417 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T03:37:41.429 INFO:teuthology.orchestra.run.smithi026.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 03:37 /home/ubuntu/cephtest 2022-07-20T03:37:41.430 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T03:37:41.435 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T03:37:41.439 INFO:teuthology.orchestra.run.smithi120.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 03:37 /home/ubuntu/cephtest 2022-07-20T03:37:41.444 INFO:teuthology.orchestra.run.smithi176.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 03:37 /home/ubuntu/cephtest 2022-07-20T14:14:04.896 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-20T14:14:04.935 DEBUG:teuthology.task.console_log:Killing console logger for smithi026 2022-07-20T14:14:04.936 DEBUG:teuthology.task.console_log:Killing console logger for smithi120 2022-07-20T14:14:04.937 DEBUG:teuthology.task.console_log:Killing console logger for smithi159 2022-07-20T14:14:04.938 DEBUG:teuthology.task.console_log:Killing console logger for smithi176 2022-07-20T14:14:04.938 DEBUG:teuthology.exit:Finished running handlers